Context Rot: When More Context Makes Enterprise AI Less Reliable

The problem starts with a very reasonable assumption

When teams begin connecting an AI agent to enterprise systems, the first instinct is usually straightforward. Give the agent access to more data, more history, more policies, more contracts, more transactions, and more risk information. The expectation is that richer context should naturally produce a better answer.

That expectation makes sense. It is also where the problem begins.

In production, an LLM can remain completely within its context window and still make a weak decision. The issue is often not capacity. The issue is competition inside the context. A critical fact can be present, but it can become less visible when it is surrounded by a much larger volume of positive, repeated, or simply more recent information.

This is the practical situation I refer to as Context Rot.

• The model has the information it needs, but not all information receives equal attention.
• Repeated historical evidence can dominate a smaller but more important current signal.
• A distractor does not have to be irrelevant. It can be completely valid information that is simply less important to the decision at hand.
• The final answer can therefore degrade even when the prompt is technically valid and within the context-window limit.

A business use case makes the problem easier to see

Imagine a procurement team using an AI agent to support a request for 500 laptops. The agent is connected to the same systems the procurement team already uses. It can read supplier master data, active contracts, historical purchase orders, pricing, preferred supplier information, policy rules, risk data, and compliance updates.

From a business point of view, this looks like a strong setup. The agent has access to the full picture.

Supplier A quickly looks like the obvious choice. It has competitive pricing, a strong delivery record, preferred-supplier status, and a long history of successful transactions. There may be dozens or hundreds of records supporting that view.

Then there is one new fact: Supplier A has been placed on procurement hold because of a compliance issue.

That single fact should change the decision completely. Yet if all of the retrieved information is passed directly to the LLM, the agent can still recommend Supplier A because the historical evidence is much louder than the one compliance signal.

• Nothing is missing from the context.
• The model is not exceeding its context window.
• The compliance information is present and valid.
• The failure happens because one critical fact is competing with a much larger body of supporting evidence.

What actually went wrong

The failure is not really a procurement problem. It is a context design problem.

The LLM was asked to do too much in a single step. It had to read the raw enterprise context, decide which evidence mattered, resolve conflicting signals, apply policy logic, compare suppliers, and make the final recommendation.

Those are different responsibilities, yet they were all pushed into one model call.

Why the obvious fix may also fail

One common answer is to reduce the context using a sliding window. That can be useful in conversational applications, but it is risky for enterprise decisioning.

If the compliance hold was retrieved earlier and newer pricing, contract, or performance data keeps arriving, the sliding window may eventually remove the exact fact that should never be forgotten.

• Recent does not always mean important.
• Old information is not automatically stale.
• Decision-critical information should be retained because of its meaning, not because of its position in the prompt.

The better approach is to separate collection from judgment

Instead of asking the LLM to reason directly over the entire raw context, I would first create a Context Engineering layer.

The first responsibility of that layer is not to choose the supplier. Its job is to collect, compact, structure, and preserve the facts that matter.

In practice, this means converting a noisy context into a smaller candidate-and-evidence view.

• Retain every viable supplier candidate.
• Summarize repeated evidence instead of passing every historical record.
• Keep critical policy, risk, and compliance signals explicit.
• Attach source references so the final decision can still be explained and audited.
• Do not drop a candidate simply because another candidate looks stronger at an earlier stage.

Then separate hard controls from LLM reasoning

There is another important design decision. Some rules should never depend on the model's judgment.

If the procurement policy says that a supplier with an active compliance hold cannot receive a Purchase Order, that rule should be deterministic. The LLM can explain the trade-offs between eligible suppliers, but it should not decide whether a compliance block can be ignored.

• Context compaction preserves evidence.
• Deterministic controls enforce mandatory business rules.
• The LLM reasons only over the candidates that remain eligible.
• The final recommendation is therefore both more reliable and easier to explain.

What a practical production solution can look like

A production implementation does not have to start with a large and expensive AI data platform. For this use case, I would keep the architecture lean and reuse enterprise systems as the systems of record.

The design objective is simple: retrieve only what is needed, compact what is retrieved, apply hard controls outside the LLM, and use the model only where reasoning creates value.

The AWS pattern, in business and technical terms

• Enterprise systems remain authoritative. SAP, Oracle, Coupa, HICX, and policy repositories continue to own supplier, contract, transaction, and compliance data.
• Amazon API Gateway and AWS Lambda provide lightweight integration with those systems. This avoids unnecessary replication of transactional data.
• Amazon S3 stores documents such as policies, contracts, and guidelines that genuinely need AI retrieval.
• Amazon Bedrock Knowledge Bases can be used for managed retrieval of unstructured content when required.
• Amazon Bedrock performs context compaction and returns a structured JSON representation containing candidates, evidence, compliance facts, and source references.
• AWS Lambda performs deterministic policy checks such as supplier-block rules, approval thresholds, and mandatory compliance checks.
• Amazon Bedrock performs final reasoning only on eligible candidates. Smaller models can handle routine decisions, with larger models reserved for complex exceptions.
• API Gateway and Lambda write the result back to procurement or workflow applications. EventBridge or SNS can be added only when event-driven processing or notifications are required.
• CloudWatch and CloudTrail provide the operational and audit trail needed for production support and governance.

Why this approach is practical from a TCO perspective

The architecture is intentionally lean. It avoids introducing infrastructure simply because it is available.

• No dedicated application database is required on day one if the source systems already own the data.
• No always-on vector cluster is required for the initial use case.
• Serverless components reduce idle infrastructure cost.
• Managed Bedrock services reduce the operational burden of hosting and scaling model infrastructure.
• Model routing allows the solution to use a lower-cost model for routine requests and escalate only when the case justifies it.
• Additional services can be introduced later if scale, latency, advanced filtering, or relationship analysis genuinely requires them.

A simple structured context for the same procurement decision

After compaction, the final decision step does not need hundreds of raw records. It can work from a compact, auditable view like this:

What this changes for an Enterprise AI Architect

The conversation changes once we stop treating the prompt as the complete solution.

The important questions become both business and technical:

• ·What information is truly required for this decision?
• Which system is authoritative for each fact?
• Which information can be summarized safely?
• Which facts must never disappear during compaction?
• Which rules must remain deterministic?
• Where should LLM reasoning begin and where should it stop?
• What evidence needs to be preserved for audit, support, and explainability?
• Which cases should be escalated to a human reviewer?

The bigger lesson

Context Rot is a useful reminder that more data, more integrations, and a larger context window do not automatically create a better Enterprise AI system.

The quality of the decision depends on how the architecture shapes the context before the model reasons over it.

That, for me, is the difference between connecting an LLM to enterprise data and designing an AI system that can operate reliably in production.