Retrieval Still Beats a 1M-Token Window for Your Codebase

A few weeks ago I was debugging an auth bug in a side project — a TypeScript backend, maybe 140 files, comfortably under a million tokens. I had a 1M-token model in front of me, so I did the lazy thing: I shoved the whole src/ tree into one prompt and asked the agent to find where the session token got refreshed and why it sometimes didn't.

It read everything. It cost me a small fortune in input tokens for that one turn. And it confidently pointed me at a refreshToken helper that looked right and was the wrong one — there were two, in different modules, and the one it picked wasn't on the path that fired. I'd given it the whole haystack and it grabbed a plausible needle from the middle of it.

I then closed that session, opened a fresh one with the same model, and let it work the normal way: list the tree, grep for refreshToken, read the three files that matched, follow the imports. It found the real bug in about a quarter of the tokens and a third of the wall-clock time. Same model. Same repo. The difference was entirely how I fed it the code.

The short version: for a real codebase, dumping everything into a huge context window is usually the worse move — not because the window is too small, but because models get measurably less reliable as you fill it, and code is the worst-case input for that failure. Targeted retrieval — grep, symbol indexing, dependency-aware reading — wins on cost, latency, and accuracy for most code tasks. Long context wins in a few specific cases, and I'll give you the rule at the end.

The window number is real; the usable part isn't

First, the part that's true: the big numbers exist. As of June 2026 the three leading coding models — Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro — all ship a 1M-token window, and thirteen-plus hosted frontier models advertise 1M or more (Morph's context-window comparison). Anthropic and OpenAI even dropped the long-context surcharge, so you're billed at the same per-token rate across the whole window now.

The problem is that advertised and usable are different numbers. Independent recall testing in 2026 keeps showing the effective capacity landing well short of the label — one roundup pegs effective context at roughly 50–65% of nominal, and reports MRCR multi-needle scores that fall off a cliff in the back half of the window: Gemini 3.1 Pro holding ~84.9% recall through the 128K–256K band and collapsing to 26.3% in the 512K–1M band (CodingFleet's "context window lie" writeup). Treat the exact percentages as directional — they're vendor-benchmark figures, not a controlled study — but the shape is consistent everywhere: recall sags long before the window is full.

That sag has a name and a research trail, and it matters more for code than for almost anything else.

Why code is the worst-case input for a full window

Two well-replicated effects work against you when you stuff a repo into one prompt.

Lost in the middle. The Liu et al. paper (published in TACL) showed that model accuracy is a U-shaped function of where the relevant information sits: highest when it's at the start or end of the context, and degrading by more than 30% when it's buried in the middle — and this held across six model families on multi-document QA and key-value retrieval. A codebase is nothing but "relevant information in the middle." The function you need is rarely the first or last file you pasted.

Context rot. Chroma's 2025 study tested 18 frontier models — Claude Opus 4 / Sonnet 4 / 3.7 / 3.5 / Haiku 3.5, OpenAI o3 and the GPT-4.1 family, Gemini 2.5 Pro/Flash, and three Qwen3 sizes — and found that performance drops as input length grows even on trivial tasks like finding a string or replicating repeated words (Chroma's "Context Rot" report). The finding that should make every code-agent builder uncomfortable: on the LongMemEval task, every model did worse with the full ~113k-token history than with a focused ~300-token excerpt of the same relevant content. More tokens of the right stuff, presented in bulk, scored lower than a small curated slice.

There's a detail in that report that's almost funny if you write code: models scored worse when the haystack "preserve[d] a logical flow of ideas" than when it was shuffled. A repository is maximally logically structured — imports, call graphs, type definitions, ordered modules. You are handing the model the exact input shape it handles worst.

And code punishes the failure harder than prose does. If a chatbot misses one sentence of a long document, it gives a slightly vaguer answer. If a code agent misses the real refreshToken definition, it writes code that calls a method that doesn't exist or patches the wrong function — and it does so confidently. As one teardown of agent retrieval put it, "the failure mode of RAG in coding contexts is silent and compounding" (MindStudio). A wrong retrieval doesn't error out. It ships.

What "retrieval" means for code (it's mostly not embeddings)

When people hear "retrieval" they picture a vector database. For code, the agents that actually ship — Claude Code, Cursor, Devin — mostly don't reach for one first. They navigate the way an experienced developer does: look at the file tree, grep for a symbol, read the specific files that match, follow the import chain (MindStudio). Claude Code's primary search tool is ripgrep — fast, exact, line-numbered text matching that works on any repo with zero preprocessing.

There's a good reason vectors aren't the default for code, and it's not nostalgia. A useful framing is three layers, escalate only when you need to (Code search for AI agents):

• Lexical (ripgrep): exact-match, milliseconds, gitignore-aware. Your default.
• Structural (ast-grep): AST-pattern matching — "find every async function," "every catch-and-rethrow" — things regex can't express cleanly. Escalate here when the query is about shape, not strings.
• Semantic (embeddings / repo-map): natural-language conceptual queries. A last resort.

Why is semantic the last resort and not the first? Because the queries a code agent actually generates are short and symbol-shaped — auth flow, user service, refreshToken — and that's precisely the format that breaks embedding search. The CoREB benchmark found that short keyword queries "collapse nearly every semantic model tested to near-zero nDCG@10." Embeddings capture meaning similarity; code is a graph of dependencies, imports, and type definitions, and similarity isn't the relation you're querying. Two functions can be semantically near-identical and structurally unrelated; the one you need is the one on the call path, not the one that reads alike.

That doesn't make semantic indexing useless — it makes it a specialist. When it earns its keep, the good implementations don't embed arbitrary text windows. They parse with tree-sitter into real units — functions, classes, interfaces with their docstrings intact — keep those chunks whole, and pair vector similarity with BM25 keyword matching plus call-graph tools for dependency tracing and blast-radius analysis (opencode-codebase-index). That's the whole game for code retrieval: dependency-aware, symbol-boundary chunking with a keyword fallback — not "split the file every 512 tokens and hope."

The cost and latency math nobody runs

Set accuracy aside for a second and just run the bill. Say a medium repo is 600K tokens. Dumping it into one turn means paying for 600K input tokens on every turn of the conversation unless you're caching carefully — and even cached, you carry the latency of the model attending over all of it. A retrieval pass instead sends the tree (a few KB), a grep result (a few hundred tokens), and maybe four files you actually read (~15K tokens). That's not a rounding-error difference; it's one to two orders of magnitude per turn, multiplied by every turn in a debugging session.

Latency tracks the same curve. A model reasoning over 600K tokens is slower to first token and slower overall than the same model reasoning over 18K. In an interactive loop where you're doing twenty turns to chase a bug, that compounds into minutes of dead waiting. Retrieval keeps each turn small, so each turn is fast, so the loop stays tight.

And there's a quieter cost: attention budget. Every irrelevant file in the window is a distractor, and Chroma showed distractors measurably drag accuracy down — one distractor hurt, four compounded it. A full repo in context is thousands of distractors for any given question. You're not just paying more to be slower; you're paying more to be slower and wronger.

When long context actually wins

This isn't "retrieval always." There's a real envelope where dumping it all in is the right call, and it's defined by repo size and task shape, not by the size of the window you're paying for.

Long context wins when:

• The whole repo genuinely fits with headroom — say under ~150K tokens, well inside the high-recall front of the window — and the task needs cross-file reasoning that's hard to retrieve piecemeal. A small repo where everything matters is exactly the case retrieval over-complicates.
• You need whole-file or whole-module reasoning in one shot — a single big file you want fully refactored, a top-to-bottom security read of one module, an architectural review where the model needs to hold the relationships in view at once. Retrieval fragments that; context preserves it.
• It's a one-shot, read-only pass, not an iterative loop. A single "review this PR" or "summarize how this service works" call amortizes the cost over one turn. The economics that kill long context — paying for the full window on every turn — don't bite when there's only one turn.

Retrieval wins everywhere else, which in practice is most of the work: any repo too big to fit comfortably, anything iterative, anything where you need a specific definition rather than a general understanding, and anything where a silently-wrong retrieval would ship broken code.

A rough rule I actually use: if the relevant slice of the task is smaller than the repo, retrieve. If the task is the whole repo and the repo fits in the first ~15% of the window, load it. Most days, the slice is smaller than the repo.

The takeaway

The 1M-token window is a real capability, and it's worth paying down the surcharge-free pricing to use it for the jobs it's good at — one-shot reviews of code that fits, whole-module reasoning, the occasional small repo where everything is relevant. But it is not a substitute for knowing where your code lives. For a working code agent, the boring stack — list the tree, grep the symbol, read the four files that matched, follow the imports, escalate to AST or embeddings only when the keyword search comes up empty — is cheaper, faster, and more accurate than the impressive one. The big number on the spec sheet is an option, not an instruction. Most of the time, don't fill it.