A while back I watched a retrieval pipeline drop the one document that mattered. A pricing memo, buried in a 40-doc set, silently ranked 51st when we returned the top 50. The agent summarized the other 49 and shipped a confident number to a customer. Wrong number. The fix wasn't a bigger context window. It was fixing the ranker.
I keep that story in my pocket whenever someone gets excited about a 12M-token context LLM, because the pitch is always "just put everything in the prompt and the problem goes away." It doesn't. Let me walk through why, and what's actually coming.
First, the honest disclaimer. There's no shipped 12M model as I write this. The frontier sits at 1M tokens — Claude Opus 4.8 and Fable 5 both run a 1M context window, and Gemini's long-context line is in the same neighborhood. Everything below is analysis, not a benchmark I ran. I'd rather reason about the physics than recite a keynote.
Why a 12M-token context LLM needs sub-quadratic attention
Here's the wall. Standard attention scales with the square of the sequence length. Multiply your input by twelve and the attention cost goes up by twelve squared — 144 times the compute and memory for the attention matrix. You don't buy your way past that with more H100s. The only way through is changing the math.
That's the whole sub-quadratic research line. State space models like Mamba, linear and sparse attention variants, RWKV-style recurrences, and the hybrid designs that bolt a few real attention layers onto a mostly-linear backbone. The Mamba paper reports linear scaling in sequence length and roughly 5x higher inference throughput than comparable Transformers, with quality holding up into million-length sequences.
But the same paper is blunt about the catch. The authors note these architectures "have not performed as well as attention on important modalities such as language." That gap is the whole game. Linear attention is cheap because it crushes the past into a fixed-size state, and crushing means forgetting. The only question that matters is whether it forgets the thing you needed. Hybrids are the smart bet here — keep a handful of full-attention layers for precision, run the rest sub-quadratic for bulk. You lose the guarantee that every token sees every other token. At 12M tokens you were never affording that guarantee anyway.
The failure mode nobody prices in: context rot
We can't fully use the 1M windows we already have. That's the part that makes me skeptical of the whole "more context fixes everything" frame.
Chroma's research on context rot tested a wide set of frontier models and found that accuracy degrades as input grows, even when the window is nowhere near full. Add tokens, lose accuracy. It's an attention-dilution problem, not a capacity one.
The classic symptom is "lost in the middle." Liu et al. (2023) showed that models attend well to the start and end of a long input and poorly to whatever's buried in the middle, with accuracy dropping sharply when the relevant fact sits in the wrong spot. Now picture that at 12M tokens, on a sub-quadratic backbone that's already lossy by design. You're stacking architectural forgetting on top of positional dilution.
And retrieval is not reasoning — that's the trap people fall into. A model can win needle-in-a-haystack, finding one planted sentence in 10M tokens, and still completely fail to synthesize a conclusion from forty facts scattered across that same haystack. Finding a fact is a lookup. Reasoning over many facts is a different, harder thing, and harder long-context evals like NoLiMa, which go beyond literal keyword matching, keep exposing the gap. A 12M needle-test win tells you almost nothing about whether the model can think over the haystack.
A realistic timeline for a 12M-token context LLM
I'll commit to a range, with the caveat that I've blown timeline calls before.
12 to 24 months: research-grade and demo-grade 12M context, mostly hybrid architectures, with asterisks. Strong needle recall, shaky cross-document reasoning. The kind of thing that benchmarks well and disappoints inside your agent loop.
24 to 48 months: production access where the reasoning quality at depth is good enough to trust for real work, the prefill latency is tolerable, and the price isn't absurd. That last clause carries most of the weight.
The economics are brutal even if the architecture lands. A 12M prefill is a wall of compute before the model emits a single output token. Seconds to minutes of latency, and a bill to match. Today Opus 4.8 runs $5 per million input tokens. One cold 12M prefill is $60 in input alone, every time, unless caching saves you. Prompt caching stops being a nice-to-have and becomes load-bearing. The problem is caching only helps when your prefix is stable, and in an agent loop it frequently isn't.
How to prep your agent workflow
You don't wait for 12M. You build like it's coming and like it won't save you.
Stop treating the context window as a junk drawer. The instinct will be to dump your whole codebase, every doc, the full chat history, and let the model sort it out. That's how you trigger context rot. The teams who'll win on big windows are the ones who got disciplined on small ones.
A few things worth doing right now. Build retrieval you actually trust, so the model sees the right 50K tokens instead of the wrong 12M — and yes, that means catching the case where the doc that matters ranks 51st. Treat memory as an explicit store the agent reads and writes, files or a database or structured notes, not "scroll up in the transcript." Prune stale tool results and summarize finished sub-tasks so the working set stays lean. Every one of those skills carries forward. None of them goes obsolete when the window gets bigger.
If your agent only works because you can cram everything into one prompt, a 12M window doesn't fix your design. It lets you fail at larger scale and higher cost.
Is context the bottleneck?
No. And that's the take I'll defend.
Raw context length is the spec-sheet number. Easy to market, easy to benchmark with a needle test, easy to mistake for progress. The real bottleneck for the agentic work most of you are doing is retrieval, memory, and agent design. The question was never "can the model hold 12M tokens." It's "can it find, weigh, and reason over the right tokens" — and a bigger haystack makes that strictly harder.
Give me a 200K model with surgical retrieval over a 12M model that rots in the middle. Every day of the week. A 12M-token context LLM, when it ships, will be a genuine tool for a few genuine problems: whole-repo analysis, long-horizon agents that truly need the history. For most of what you build, it'll be a pricier way to make the same mistake my pipeline made — confidently shipping the wrong number because the right one never made it into the model's attention.
Context isn't the hard part. Deciding what goes in it is the whole job.