Learning Without Remembering. The Word the Industry Abuses

What the industry calls "in-context learning" — ICL in the acronym it goes by in the papers — is adjustment inside the window, not learning in the strict sense. Close the session and it all vanishes. No human would call reading a manual and forgetting it the moment the door shuts learning. The persistence of what's learned, the bedrock of any operational definition of learning since Tulving in 1972, is something an LLM doesn't have. Calling it learning in a salesperson's mouth is advertising. In a serious researcher's mouth, a concession to everyday language that's worth auditing before inheriting it wholesale.

What it means to learn, slowly

There's a common minimum that philosophy of mind, experimental psychology and neuroscience reached long ago, without great quarrels, to tell learning apart from anything else that resembles it. For there to be learning, at least three things have to hold. First, persistence: what's learned has to remain available when the same situation comes back after a while. Second, transfer: what's learned in one situation has to serve, at least partly, in similar but not identical situations. Third, stable modification of the response: faced with the same input, the system responds differently before and after the formative episode, and the difference holds.

It's not a demanding definition. It squares with common sense. It squares with motor learning, with conceptual learning, with the implicit learning of one's mother tongue. It squares, above all, with the ordinary use of the verb "to learn" in any language. If one morning you can do something you couldn't do yesterday, and tomorrow you can still do it, there was learning. If the thing evaporates when the door shuts, there was something else, and it's worth having a different word for that something else.

Tulving back in 1972 already separated episodic memory from semantic memory with this logic of persistence. Almost everything that has come since in cognition has kept the same demand, refining it but not weakening it. What happens in a session with an LLM doesn't meet it.

Three different mechanisms under the same label

The industry packages three technically very different processes as "learning," and the confusion almost always comes from not separating them.

Fine-tuning modifies the model's weights. It's the closest thing to learning an LLM does. The base model is retrained on a specific corpus, the parameters change, and the new version responds differently to the same inputs. The modification is persistent: a fine-tuned model still responds differently next week. It meets persistence and stable modification. It partly meets transfer. There is one operation that deserves the verb, and this is it.

ICL: adjustment inside the window

In-context learning, by contrast, doesn't touch a single weight of the model. You put a few "X → Y" style examples into the window, and from there the model is able to follow that pattern when you ask it "Z → ?". It looks miraculous. Brown and others presented it as a great novelty in Language Models are Few-Shot Learners (NeurIPS 2020), the paper selling the GPT-3 leap, and the metaphor of "learning from a few examples" stuck so hard that it has outlived the fact that it's technically inaccurate. There is no learning, because once the session closes the weights stay identical. Come back tomorrow without feeding the examples again and the model doesn't remember what it "learned." And for the same reason it doesn't transfer outside the window in which it happened: what ICL does is adjust, within the turn, which pattern carries on.

RAG and external memory systems are the third family, further still from the verb. What they do is index text outside the model and re-inject it into the window when the system deems it relevant. The model doesn't integrate that text into its structure, not when it receives it and not afterward. It reads it, uses it and forgets it when the session closes. The "memory" of a RAG assistant is a dossier that a parallel process hands it before each turn, not something the model learns.

Of the three, only the first deserves the word. The other two do something else, something useful, valuable, but else. The mixing under the common label is what lets you write headlines like "the new model learns continuously" when what the product does is index.

What Min and others measured, and what they found

There's an experiment worth bringing in because it cleanly takes apart the intuitive reading of in-context learning. Min, Lyu, Holtzman and others published in EMNLP 2022 Rethinking the Role of Demonstrations. What Makes In-Context Learning Work?. They took twelve models of different sizes and architectures and compared performance on classification tasks under two conditions. In one, the examples in the prompt had the correct labels. In the other, the examples had labels assigned at random.

Human intuition would say the second condition should wreck performance. Teach me three wrongly solved examples and I should learn badly. Turns out, no. The model's performance barely moves when the labels are random. What really affects performance is the set of possible labels, the statistical distribution of the inputs, and the format of the prompt. That is, the model wasn't learning from the examples in the sense a student would learn. It was using the examples as formal cues: to recognise the kind of task, the range of answers, the expected syntax.

That's adjustment inside the window. It isn't learning. And it's worth reading Min's paper slowly because it has the rare virtue of measuring what the popular metaphor took for granted and showing it false. Marcus, in Rebooting AI (2019), had already anticipated that criticism of using the verb "to learn" for systems without persistent modification; the metaphor was convenient, and the field preferred to inherit it rather than wrestle with the language.

Induction heads. The mechanism under the trick

Olsson and others, in In-context Learning and Induction Heads (arXiv 2209.11895, 2022), contributed from the mechanistic side the other piece of the puzzle. Studying the inner workings of Transformers they identified structures called induction heads, attention heads specialised in one concrete operation: when a pattern has appeared earlier in the context, copy it or continue it. If the model has read "Paris is the capital of France. Berlin is the capital of Germany. Madrid is the capital of," an induction head recognises the pattern and pushes the next prediction toward "Spain."

In-context learning, on this reading, is in good part the emergent consequence of having many well-trained induction heads. It's not that the model learns the concept "capital of" during the prompt. It's that the internal machinery was already set up to chain patterns, and the prompt's examples switch on the right heads so the pattern carries on. The paper is technical, partial and honest about its limits, but the general picture it offers is clear: ICL is sophisticated pattern matching and not conceptual learning.

This doesn't strip value from the phenomenon. It places it. And placing it, it stops being a mystery, stops being a metaphor for human learning, and becomes what it is: an operation that exploits the Transformer's internal structure to sustain tasks within the turn. The word "learning" in ICL is historical, not descriptive, and the technical community itself uses it with growing discomfort.

"It learns from your conversations." Who learns, exactly?

The advertising abuse arrives here. Anyone who has read the terms of service of a commercial assistant will have seen the phrase, in some variant. "The system learns from your interactions to improve." It sounds like personal feedback, like a relationship that matures, like a model that becomes your model over time.

Read carefully what it says and what it doesn't.

The company learns, the model doesn't

What technically happens, in most products, is that the company stores your conversations on its servers and uses them to train the next version of the model, which will be deployed for all users. It's the company that learns, in the literal sense, and the model will learn when the next version is released, not before. Your instance of the assistant, today, tomorrow and next week, is still the same static model with the same weights. What changes between sessions is the system wrapping the model, the storage of your profile, the list of facts some process has decided to keep. Not the model.

The difference between "my assistant learns with me" and "the company trains its next model with my conversations" is the difference between a personal tutor and an anonymous researcher taking notes on everything you say. The first reading is the one the marketing encourages. The second is the one that's happening.

And it's worth putting it that way without tangling it up. Not because it's scandalous to train the next version with user data — the terms say so, usually, and users can switch the option on or off depending on the provider. But because selling one thing with the metaphor of the other is a distortion that affects decisions about use, privacy, delegation.

Catastrophic forgetting, an old crack that hasn't closed

So it doesn't seem like the difficulty of learning is exclusive to contemporary LLMs, it's worth mentioning an old problem deep learning has dragged from its beginnings and that no recent advance has fully solved. It's called catastrophic forgetting. When a neural network trained to do one task is retrained to do another, it tends to degrade dramatically on the first, even when the second has nothing to do with it. The most complete review to date is Wang and others in A Comprehensive Survey of Forgetting in Deep Learning (arXiv 2307.09218, several versions up to 2024), complemented by Lesort and others in Continual Learning and Catastrophic Forgetting (arXiv 2403.05175, 2024).

The operational consequence matters. Even if someone built a system that genuinely learned from each user, modifying weights in real time, that system would run the risk of erasing prior knowledge to accommodate the new. Continual learning is a field open for decades, with partial advances and no algorithm that has solved the question in general. And as long as the field isn't solved, the commercial promise of "a model that learns with you" rests either on periodic fine-tuning controlled by the provider, or on external indexing, or on nothing.

Delegating to something that doesn't learn

If the system doesn't learn in the human sense of the verb, and we operate with it as if it did learn, we are making false assumptions to take decisions that do count.

It's what happens when someone delegates to a commercial assistant the tracking of a months-long project, taking for granted that the system will keep integrating what it learns about their work. What technically happens is that the system, in each session, receives a synthetic dossier made by heuristic processes whose logic the user doesn't see. The corrections the user makes again and again aren't "learned": they're noted, if the system decides they're worth it, and re-injected, if the system decides they apply. The difference from a human collaborator who remembers is not one of degree, it's one of nature.

The same goes for more serious institutional uses. When a public administration adopts an LLM-based system for processing or advice, and the public discourse takes for granted that the system "improves with use," it's taking for granted something that technically doesn't happen in the deployed instance. The deployed system doesn't learn. The provider company does, in its lab, with its own timelines and processes. That difference is enormous when it comes to auditing who improves what with whose data.

Bender, Gebru, McMillan-Major and Shmitchell, in Stochastic Parrots (FAccT 2021), pointed at this with language that already sounded harsh back then and that later reality has refined more than refuted. What gets sold as learning is usually sophisticated repetition with contextual adjustment. They aren't the same. And the question isn't whether the difference is relevant for a casual user asking for pasta recipes: it isn't. The question is whether the difference is relevant for someone who delegates serious work to a system and then signs their name to what comes out.

Definitions

Learning. In its minimal operational definition, a process that produces a persistent change in a system's response, transferable to similar situations and stable over time. Without persistence, there is no learning in the strict sense.

Fine-tuning. Retraining a pretrained model on a specific corpus, modifying its weights. The resulting version is a different model from the original. It's the procedure of an LLM that most resembles learning, though it is neither continuous nor personal.

In-context learning (ICL). The ability of an LLM to solve a task from examples given in the prompt, without modifying weights. It works within the turn and vanishes when the session closes. Technically it is pattern matching, not learning.

RAG (retrieval-augmented generation). A system that indexes external text and injects it into the model's window when relevant to the query. It doesn't modify the model; it complements the context.

Induction head. An attention head inside a Transformer specialised in recognising and continuing repeated patterns in the context. Olsson and others proposed it as the main mechanism underlying in-context learning.

Catastrophic forgetting. The phenomenon by which a neural network trained for one task loses performance on it when trained for a new one. It's the main crack the field of continual learning tries to close, without success yet.

Continual learning. A subfield of machine learning that studies how to train models so they acquire new capabilities without degrading earlier ones. Open since the eighties, with partial advances and no general solution.

References

Tulving, E. (1972). Episodic and Semantic Memory. In Organization of Memory, Academic Press. Operational framework of persistence as a condition of learning, cited to fix the definition against the commercial metaphor.

Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020. arXiv 2005.14165. The paper that popularised the term "in-context learning" by presenting GPT-3.

Min, S. et al. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. EMNLP 2022. arXiv 2202.12837. Empirical evidence that correct labels in the prompt's examples barely affect ICL performance.

Olsson, C. et al. (2022). In-context Learning and Induction Heads. Anthropic Research / arXiv 2209.11895. Identifies induction heads as the mechanism underlying ICL, presenting it as pattern matching and not conceptual learning.

Wang, L. et al. (2024). A Comprehensive Survey of Forgetting in Deep Learning. arXiv 2307.09218. Exhaustive review of the state of catastrophic forgetting in neural networks.

Lesort, T. et al. (2024). Continual Learning and Catastrophic Forgetting. arXiv 2403.05175. A panorama of continual learning, cited to show that the crack is still unclosed.

Bender, E., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. FAccT 2021. DOI: . A foundational critique of using cognitive vocabulary to describe what LLMs do.

Marcus, G. (2019). Rebooting AI. Pantheon. An early critique of using the word "learning" for systems without persistent modification.

You might also like

• Intelligence without memory. The doctor who forgets your case when the door shuts
• The context problem. The invisible cage of every human-AI conversation
• Thinking without understanding. If it works the same, what did we think thinking was
• Knowledge doesn't settle

Elsewhere

• Neural networks (3Blue1Brown): what changes and what doesn't when you train a network
• AI: A Guide for Thinking Humans (Melanie Mitchell)