Context Engineering for AI Agents: Why a Bigger Window Won't Save You

Your agent works beautifully on turn three and starts hallucinating on turn thirty. The model did not get worse. Its context did.

That gap between a demo and a production agent has a name now: context engineering. It is the discipline of deciding what goes into the model's working memory at every single step, and it has quietly become one of the most valuable skills in AI engineering — more valuable than clever prompt wording, and more valuable than access to a bigger window.

This post covers what context engineering actually is, why long context degrades, the four-strategy playbook you can implement this week, and the code to enforce a token budget.

Prompt engineering vs. context engineering

Prompt engineering is writing one good instruction. Context engineering is designing the whole information supply chain that surrounds that instruction across many turns.

The split is real enough that the industry now treats them as separate roles, the way web design split into UI and UX.

A useful test: if your fix is a better sentence, that is prompt engineering. If your fix is a better pipeline, that is context engineering.

Diagram 1 — What is actually in your context window

Most engineers picture "the prompt." The model sees something much messier.

Engineers spend hours polishing a 400-token system prompt while a single unfiltered API response dumps 12,000 tokens of JSON into the window.

Context rot: the research behind the panic

In July 2025, Chroma Research published Context Rot: How Increasing Input Tokens Impacts LLM Performance. They tested 18 frontier models across deliberately simple tasks — needle-in-a-haystack retrieval, conversational QA on LongMemEval, and even just repeating a series of words back.

The finding that reframed agent architecture: models do not process their context uniformly. Performance degrades as input grows, well before the window is full, and it degrades non-uniformly depending on how similar the needle is to the question, how many distractors are present, and how the surrounding text is structured.

Two failure modes people conflate are worth separating:

• Context overflow is binary. You hit the token limit, something gets truncated, or you get an error.
• Context rot is gradual. Nothing errors. Quality just quietly drops.

Advertised window does not equal usable window. Treat context as an expensive, rationed resource even when the provider tells you it is nearly free.

Diagram 2 — The write / select / compress / isolate pipeline

Lance Martin at LangChain formalised the four moves that make up context engineering. Every technique you will read about is one of these four.

Write — anything the agent will need later goes to durable storage, not the transcript.

Select — retrieve wide, then rerank narrow. A pipeline that pulls 50 candidates and reranks to 5 beats one that dumps all 50 into the prompt.

Compress — summarise, do not accumulate. Old turns become one paragraph; a 40 KB API response becomes the three fields you needed.

Isolate — give a sub-agent its own clean window, let it do the messy work, and return only the conclusion to the parent.

Code: a token budget you can actually enforce

The single highest-leverage change most teams can make is turning "keep the context small" from a vibe into a number. Allocate a budget per component, and enforce it before every call.

from dataclasses import dataclass, field
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def ntok(text: str) -> int:
    return len(enc.encode(text))


@dataclass
class ContextBudget:
    """Hard caps per component. Tune these against your evals, not your intuition."""
    total: int = 32_000
    system: int = 1_200
    tools: int = 2_000
    memory: int = 1_500
    retrieved: int = 8_000
    history: int = 6_000
    tool_output: int = 4_000
    reserve_for_output: int = 4_000

    def spend(self) -> int:
        return (self.system + self.tools + self.memory + self.retrieved
                + self.history + self.tool_output + self.reserve_for_output)

    def validate(self) -> None:
        assert self.spend() <= self.total, (
            f"Budget over-allocated: {self.spend()} > {self.total}"
        )


@dataclass
class ContextAssembler:
    budget: ContextBudget = field(default_factory=ContextBudget)

    def fit(self, text: str, cap: int, strategy: str = "tail") -> str:
        """Trim a component to its cap. 'tail' keeps the most recent tokens."""
        toks = enc.encode(text)
        if len(toks) <= cap:
            return text
        kept = toks[-cap:] if strategy == "tail" else toks[:cap]
        return enc.decode(kept)

    def build(self, *, system, tools, memory, chunks, history, tool_out) -> str:
        b = self.budget
        b.validate()

        parts = [
            ("SYSTEM",     self.fit(system,   b.system,      "head")),
            ("TOOLS",      self.fit(tools,    b.tools,       "head")),
            ("MEMORY",     self.fit(memory,   b.memory,      "tail")),
            ("KNOWLEDGE",  self._pack(chunks, b.retrieved)),
            ("HISTORY",    self.fit(history,  b.history,     "tail")),
            ("OBSERVED",   self.fit(tool_out, b.tool_output, "head")),
        ]
        return "\n\n".join(f"<{tag}>\n{body}\n</{tag}>" for tag, body in parts)

    @staticmethod
    def _pack(chunks: list[str], cap: int) -> str:
        """Add reranked chunks in order until the cap is hit. Never partially truncate a chunk."""
        out, used = [], 0
        for c in chunks:
            cost = ntok(c) + 2
            if used + cost > cap:
                break
            out.append(c)
            used += cost
        return "\n---\n".join(out)

Two details matter more than they look:

1. total is not the model's maximum. Set it to the window size where your evals still hold up.
2. _pack refuses to half-include a chunk. A truncated chunk is a distractor with a confident tone.

Compaction: the checkpoint that keeps long sessions alive

SUMMARISE = """Compress this conversation into <= 200 words.
Keep: decisions made, constraints stated, unresolved questions, tool results still in play.
Drop: pleasantries, superseded attempts, verbatim tool output.
Write as terse notes, not prose."""

async def maybe_compact(history: list[dict], client, threshold: int = 6_000):
    joined = "\n".join(m["content"] for m in history)
    if ntok(joined) < threshold:
        return history

    head, tail = history[:-6], history[-6:]
    summary = await client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=400,
        messages=[{"role": "user",
                   "content": f"{SUMMARISE}\n\n<conversation>\n"
                              f"{chr(10).join(m['content'] for m in head)}\n</conversation>"}],
    )
    note = summary.content[0].text
    return [{"role": "user", "content": f"[EARLIER SESSION NOTES]\n{note}"}] + tail

Keeping the last few turns verbatim is deliberate. Summaries lose the exact phrasing the model needs for immediate follow-ups; older turns only need their conclusions.

Diagram 3 — One agent turn, with context management in the loop

Agentic context engineering: letting the agent manage its own budget

The frontier of this in 2026 is handing the curation job to the agent itself. A research line from Stanford, SambaNova and UC Berkeley on Agentic Context Engineering splits the work across three roles:

Instead of one static system prompt, the context becomes a living document the agent edits. The model did not get smarter. Its context did.

The context engineering checklist

• There is a number for the token budget, and it is enforced in code
• Raw tool output is never passed straight into the window
• Retrieval reranks; you do not ship top-k similarity search alone
• Conversation history compacts at a threshold, keeping recent turns verbatim
• Only the tools relevant to the current step are loaded
• Durable facts live in a store, not in the transcript
• You log tokens-per-component on every call
• You have an eval that runs at long context, not just short
• Critical instructions sit at the start or end, never in the middle
• Sub-agents have isolated windows and return conclusions, not transcripts

FAQ

Sources

• Hong, Troynikov & Huber — Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma Research (2025)
• Liu et al. — Lost in the Middle, TACL (2024)
• Lance Martin / LangChain — write, select, compress, isolate taxonomy
• Karpathy's June 2025 framing of the term "context engineering"
• Agentic Context Engineering — Stanford, SambaNova, UC Berkeley