The context problem. The invisible cage of every human-AI conversation

The Transformer architecture, formalised by Vaswani and others in 2017, operates under a finite attention window. What fits in the window exists for the model. What's left outside, for the model never existed. In 2026 the big labs have scaled the window to one or two million tokens, and the marketing presents it as if the problem were solved. It isn't. The cage is bigger, it hasn't stopped being a cage. Any serious conversation with an AI is fought inside those edges, and the "comprehension" that's promised comes tied to the frame, not beyond it.

The window isn't a commercial decision

There's a comfortable confusion worth cutting off at the start. An LLM's context window isn't a seatbelt the provider put on you to stop you abusing the service. It's a property of the architecture. The Transformer, described in Attention Is All You Need (Vaswani et al., NeurIPS 2017), processes a sequence of tokens by computing, for each one, how much it attends to each of the others. If your input has a thousand tokens, the model internally builds a thousand-by-thousand attention matrix. The operation is at the heart of the mechanism. Without that matrix, the Transformer isn't a Transformer.

When you double the input length, the matrix doesn't double: it quadruples. Ten thousand tokens means a hundred million entries in the matrix. A hundred thousand tokens means ten billion. A million tokens means a trillion. The operation of computing all that attention doesn't scale linearly, it scales quadratically, and the cost is paid twice: in memory, because the matrix has to be stored; and in compute, because each entry has to be calculated. That's why expanding the model's window costs dearly, and each jump of an order of magnitude requires either architectural tricks or a notable energy bill that someone ends up paying.

The window, then, isn't a product choice. It's a compromise between what the architecture can sustain, what the GPU of the moment allows, and what the customer is willing to pay per query. Within that compromise, the labs adjust the dial. But the dial lives within a range imposed by the silicon and by the maths, not by the will to be generous.

The attempts to break the wall

The academic industry has spent eight years pushing against this restriction, and it's worth knowing what's been achieved and what hasn't.

Longformer, proposed by Beltagy, Peters and Cohan in 2020, was one of the first serious attempts. The idea is to replace full attention with sparse attention: each token pays attention only to a local window of nearby tokens, plus a few "global" tokens chosen by hand to sustain long-range coherence. The cost becomes linear. The window rises to tens of thousands. The price paid is that the text's representation loses finesse: not all tokens are connected to all, so certain long-range relations break. It's a technique that works reasonably well on long documents where most of the relevant connections are local, and not so well on tasks that require tying a detail at the start to one at the end.

From attention to state spaces

Mamba, presented by Gu and Dao in arXiv 2312.00752 in 2023, is a deeper jump. It renounces the attention mechanism and goes back to an old lineage, that of state space models (SSM), modernised with a piece called selective scan that lets the state update in a content-dependent way. The complexity is linear in the sequence length. On language benchmarks it competes with comparable Transformers. It works, and well. But it still hasn't been tried at the brute scale to which commercial Transformers have been scaled, nor with the post-training investment these carry accumulated. It's a promising architecture, not a deployed substitute.

RWKV (Peng et al., 2023), a hybrid between recurrent network and attention, plays a similar role. Linear cost in inference, a theoretically infinite window, an active community, open models of modest sizes. And the same practical limitation: the highest state-of-the-art performance figures, the ones that fill press kits and rankings, are still held up by the classic Transformer with its quadratic window and its bill.

What's honestly concluded from this landscape is the following. The wall of quadratic cost has real technical cracks. The cracks still haven't translated into mass products that match the Transformers in everything. And while that doesn't happen, the business is fought inside the quadratic frame and, therefore, inside a window that grows, but not as much as the marketing pretends.

Lost in the middle. What happens inside the window

There's an experimental result worth putting on the wall next to the two-million-token figure. In 2023 Nelson Liu and others published Lost in the Middle: How Language Models Use Long Contexts, which later appeared in TACL 2024. The experiment is modest and that's why it's brutal. They take several commercial models with long windows, feed them a document inside which they hide the piece of information needed to answer a question, and move that piece around within the context: start, middle, end.

The performance graph is U-shaped. When the key information is at the start or the end of the context, the model finds it with reasonable accuracy. When it's buried in the middle, performance collapses. In some models the drop is tens of percentage points. That is, the model "has" the information inside its window, technically it's processing it, and even so it loses it.

This matters a great deal. It means the context window isn't even a uniform container: inside it has privileged zones and blind zones. The advertising promise that a model "can read two million tokens" translates, in practice, into being able to read the first fifty thousand well and the last fifty thousand well, and from there on its attention is serving coffee. Liu and others measured it. The industry, in general, doesn't cite it in the keynotes.

The deletion with no warning

There's a second property of the window the user almost never knows explicitly and which changes how any long conversation reads. When it overflows, the model doesn't warn you.

If your conversation with a commercial assistant exceeds the window's size, most systems do one of three things. Either they cut the oldest messages out at the root, without touching anything in the chat's front end. Or they summarise them with another process and put the summary where the original was. Or they replace them with a generic placeholder. In any of the three cases, what the user sees on their screen is the whole conversation, intact, as if everything were still inside the model's head. What the model actually sees is something else.

It's an important informational asymmetry, and it's not accidental. Warning the user that "this turn I've forgotten what you said in the first forty messages" would break the continuity effect the product rests on. Keeping the warning hidden preserves the illusion that the assistant is still with you, when in reality it's been, for three turns, a different assistant from the one that started the session.

When the model goes weird

This has practical consequences. Anyone who's used an LLM to sustain a complex task across many messages has tripped over the moment the model "goes weird": it contradicts something it itself stated at the start, forgets a constraint it had been given, invents a conclusion incompatible with the data it was handed. The user's intuitive explanation is that the model has made a mistake. The correct explanation, most of the time, is that the part of the conversation where that information was is no longer in the window, and the model is improvising from what's left.

When someone argues that LLMs "hold context well", they're usually describing short conversations within the window's comfort zone. The claim stops being true past a certain threshold, and the threshold isn't marked anywhere visible to the user.

The "comprehension" that's sold and the one that fits

There's a more philosophical argument worth glancing at even briefly. Emily Bender and Alexander Koller, in Climbing towards NLU (ACL 2020), held that a system trained only on linguistic form, with no access to the external referent of what's said, isn't comprehending in the strong sense of the term. Their argument, that of the "octopus" that learns English by listening to phone calls but never touches a coconut, is aimed at LLMs with or without a long window.

The window adds to that discussion an operational nuance, not a philosophical one. Even if you accept by convention calling "comprehension" what LLMs do, that comprehension is tied to what's been fed into the window this turn. If the context document fits, there's comprehension. If it doesn't fit, there isn't. And if it fits but is in the middle zone, there's half-comprehension, with the U shape Liu and others measured.

What's promised when it's said that a model "understands a whole book" is something stronger. It's the suggestion that it has integrated the book into a stable representation that endures beyond the turn, that weighs some sections against others, that holds a hierarchy of what was important and what was incidental. None of that happens. What happens is that the book fits in the window, the model reads, the model answers, and on closing the session the book's representation stops existing. To speak of comprehension when describing that flow is elastic at best, dishonest at worst.

Marcus, in Rebooting AI (Pantheon, 2019), had been insisting on this for a few years when Transformers still worked with two-thousand-token windows. The window has grown three orders of magnitude. The argument has barely aged.

A cage that widens isn't an open field

The usual rhetorical trick consists of presenting each window jump as an advance towards the problem dissolving. Eight thousand, thirty-two thousand, a hundred thousand, a hundred and twenty-eight thousand, two hundred thousand, a million, two million. The line keeps rising, and the advertising argument is that at some point the window will be big enough for the problem to stop being one in practical terms.

What that reading hides is the nature of the problem. The window isn't only a quantity. It's a border. As long as there's a border there's an outside, there's a silent deletion on crossing it, there's an uneven weight of attention inside, there's a dependence on which portion of the world you choose to put in each turn. A cage of two million tokens is bigger than one of eight thousand, but it's still a cage, and all human-AI conversations are still fought inside it.

The honest thing would be to say it that way from the start, and to operate accordingly. That the conversation with an AI is a duel inside a frame the model didn't choose and the user doesn't see. That the frame moves, but doesn't disappear. And that any promise of long-term comprehension, of real continuity between sessions, of sustained thought beyond the turn, is selling something the architecture, for now, doesn't allow.

Definitions

Token. The minimum unit of text the model processes. In English it's roughly half a word. In Spanish, depending on the tokeniser, it tends to be somewhat smaller.

Context window. The maximum number of tokens a model can have in front of it simultaneously in one inference. It determines how much text it can read and keep active at once. Everything exceeding the window stays out of the model's reach in that turn.

Transformer. A neural-network architecture described by Vaswani et al. in 2017, based on the attention mechanism. It dominates the field of language models today and is the architecture underpinning GPT, Claude, Gemini and most open models.

Attention. The operation by which each token in the sequence computes how much it should "look at" each of the others to build its representation. Full attention has a quadratic cost in the sequence length.

Sparse attention. Variants that don't compute the whole attention matrix but only a part: local neighbours, global tokens, predefined patterns. They reduce the cost to nearly linear in exchange for losing coverage. Longformer is the classic example.

State space model (SSM). A family of architectures alternative to the Transformer, which process the sequence by keeping an internal state that updates step by step. Linear cost in length. Mamba is the modern representative.

Lost in the middle. An experimental result showing that LLMs with long windows attend much better to the extremes of their context (start and end) than to the centre. Performance as a function of the position of the key information is U-shaped.

Quadratic / linear. Terms describing how the cost of an operation grows as the input grows. Quadratic: if the input doubles, the cost quadruples. Linear: if the input doubles, the cost doubles.

References

Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017. arXiv 1706.03762. The origin of the Transformer architecture and the attention mechanism on which contemporary LLMs rest.

Beltagy, I., Peters, M. E. & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv 2004.05150. Sparse attention with linear cost, cited as the first serious attempt to break the quadratic barrier.

Gu, A. & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv 2312.00752. An alternative architecture based on state space models, cited as a candidate substitute for the Transformer.

Peng, B. et al. (2023). RWKV: Reinventing RNNs for the Transformer Era. arXiv 2305.13048. A hybrid between recurrent network and attention, cited as another line with linear cost in inference.

Liu, N. F. et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. arXiv 2307.03172 (2023). The source for the data on U-shaped performance inside the window.

Bender, E. & Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. ACL 2020. The conceptual argument on the limits of comprehension in systems trained only on linguistic form.

Marcus, G. (2019). Rebooting AI: Building Artificial Intelligence We Can Trust. Pantheon. An early critique of the use of the term "comprehension" applied to models with no stable representation beyond the turn.

You might also like

• Intelligence without memory. The doctor who forgets your case on closing the door
• Learning without remembering. The word the industry abuses
• "Attention Is All You Need", the paper that changed AI
• Thinking without understanding. If it works the same, what did we think thinking was

Elsewhere

• Attention in transformers, step-by-step (3Blue1Brown): how attention works
• AI: A Guide for Thinking Humans (Melanie Mitchell)