The Number That Launched a Thousand Benchmarks
One million tokens. The figure has become a marketing centrepiece for a new generation of large language models, plastered across product announcements and LinkedIn posts with the kind of breathless enthusiasm usually reserved for GPU die counts. But what does a million-token context window actually mean for the software you build, the APIs you call, and the quality of the answers you receive? The honest answer is: far less than the headline suggests — and, in a handful of specific situations, genuinely transformative. Understanding which camp your use case falls into is now a meaningful engineering decision.
A Brief History of the Window
Early large language models shipped with context windows of roughly 4,000 tokens — enough to hold a few pages of text, a short conversation, or a modest code file. Developers quickly ran into the hard ceiling: summarise this document, they would ask, and the model would truncate or simply refuse, because the document did not fit. The pressure to extend that limit was immediate and practical. Models moved to 8K, then 16K, then 32K tokens. GPT-4 Turbo pushed to 128K. Gemini 1.5 announced 1 million tokens for selected users, with 2 million tokens on the experimental roadmap. Each jump felt like a breakthrough. In aggregate, they represent roughly a 250-fold increase in context capacity in just a few years — a pace that rivals the rate of change in GPU memory.
The architectural mechanisms behind these gains are not a single invention but a combination of advances. Positional encoding schemes — the mathematical scaffolding that tells a model where each token sits in a sequence — had to be redesigned to handle sequences orders of magnitude longer than those seen during pretraining. Techniques such as rotary positional embeddings and their various extensions allow models to generalise positional awareness further than absolute encodings ever could. Attention itself, the core operation that lets a model relate any token to any other, scales quadratically with sequence length in its naive form, meaning a million-token sequence would require compute that dwarfs the model's parameter count just for a single forward pass. Sparse attention patterns, sliding window approaches, and purpose-built hardware optimisations have all played a role in making long-context inference commercially viable, even if not cheap.
Lost in the Middle: The Dirty Secret of Long Contexts
Here is the thing the product page does not mention: the ability to process a million tokens is not the same as the ability to attend equally well to all of them. Research into how language models actually use long contexts has surfaced a consistent and troubling pattern, often called the lost-in-the-middle problem. When critical information is placed at the very beginning or the very end of a long context, models tend to retrieve and use it reliably. When that same information is buried in the middle of a lengthy sequence, performance degrades — sometimes sharply. The model is not ignoring the middle of the document in the way a distracted reader might skim, but the attention signal it routes toward central tokens is weaker, less certain, and more prone to being overridden by proximity effects at the edges of the window.
The practical implication is significant. If you are feeding a model a 200,000-token codebase and asking it to find a subtle bug, the answer you get depends substantially on where in the context that bug happens to live. This is not a problem that brute-forcing a longer context window solves — it is a property of how attention is learned and distributed during training. Some newer models have explicitly targeted this weakness with training data and objective modifications designed to reward retrieval from arbitrary positions, and there is genuine progress. But the problem has not been eliminated, and treating a million-token window as a uniform search space remains an act of optimism rather than engineering certainty.
The Cost Equation Nobody Puts in the Press Release
Context length and API cost are not independent variables. Most commercial LLM APIs price on a per-token basis, charging separately for tokens sent in the prompt (input tokens) and tokens generated in the response (output tokens). Input tokens are typically cheaper than output tokens, but a million-token prompt is still a million tokens, and at any non-trivial per-token price, the arithmetic becomes uncomfortable fast. Sending a long novel, an entire codebase, or a year of financial records as context in every API call can produce invoices that would make an engineering manager wince — and because the cost is incurred per request, not once up front, it compounds with usage in ways that are easy to underestimate during prototyping.
Prompt caching mechanisms, now offered by several providers, offer partial relief. When the same long prefix is reused across many requests, the provider can cache the key-value representations from the attention layers and avoid recomputing them from scratch, passing some of the savings on to the caller. This makes patterns like "load a large document once and ask many questions about it" substantially more economical. But caching has its own constraints — cache validity windows, cache key granularity, and the requirement that the prefix remain identical — and it does not help when the long context is genuinely different for every request. For many real workloads, the cost of large-context calls remains a genuine architectural constraint, not just a line item to optimise later.
When a Million Tokens Actually Changes What You Can Build
Despite the caveats, there are categories of problems where a very large context window does not merely speed up existing workflows — it enables fundamentally new ones. The clearest example is whole-codebase reasoning. With a 4K or even 32K context, analysing how a change in one module propagates through a large repository requires either chunking (which loses cross-chunk relationships) or retrieval-augmented approaches (which depend on the quality of the embedding and retrieval step). At a million tokens, it becomes at least plausible to load an entire mid-sized codebase into context and ask questions that span files without a retrieval intermediary, provided the lost-in-the-middle problem is manageable for the specific query. Similarly, long-form document analysis — legal contracts, scientific papers with extensive appendices, regulatory filings — benefits from having the full document in context rather than a summarised or chunked version.
Multi-turn conversations that need to remain coherent over very long sessions are another genuine beneficiary. Rather than truncating or summarising earlier turns, a large window can retain the full exchange, preserving nuance and reducing the risk of the model contradicting something it said two hours ago. For agents that must maintain state across many tool calls and observations, a large context window is less a luxury than a requirement.
The pattern that emerges from these use cases is specificity. A million-token context window changes what you can build when your problem has genuine long-range dependencies that cannot be decomposed cleanly, when retrieval-augmented approaches introduce unacceptable precision loss, and when the cost of large-context calls is acceptable relative to the value of the answer. Those conditions are real, but they are not the default case for most applications being built today.
The Practical Engineering Position
For most teams, the sensible approach is to treat large context windows as a capability to reach for when smaller windows genuinely fail — not as the default architecture for every AI feature. Retrieval-augmented generation, chunking strategies, and prompt engineering can still solve the majority of document-heavy tasks at a fraction of the cost, and they sidestep the lost-in-the-middle risk entirely by putting only the relevant material in front of the model. Where those approaches break down — where the signal that matters cannot be isolated in advance, where the relationships are emergent rather than locatable — that is when a million tokens earns its place in the system design. The headline number is real. The uniform quality across that window is not, yet. And the bill at the end of the month is very real indeed.