Why Shoshana Zuboff is right about AI

I came to Zuboff late and badly, when everyone was already quoting her from memory without having finished her. Her book is from 2019, before ChatGPT, and on the strength of that date they dismissed it as a last-season anti-Google complaint. I think they got the wrong funeral. What she described didn't stay in the search engine: it changed casing and moved into the chatbot. I type a prompt and I think of her, and the memory is no comfort.

The line hanging over the whole debate

"If a product has no price, the product is you." The aphorism does the rounds from bar to bar, attributed to whoever's handy, worn out from being repeated. It's right enough to survive and falls short the moment you press it. What Zuboff has been honing for twenty years goes further than the free: she argues that when something is free, cheap or subsidised, the real business breathes on another plane, one that isn't shown to you. And with generative AI that plane charges in a currency you didn't entrust to anyone before: how you think, what worries you, what you're stuck on, which decision keeps you up at night and you're going to make tomorrow morning.

There's an old catechism warning that travels on the same rail. God is watching you. Grant the priest he's entirely right and God still comes off badly in the comparison, because he doesn't pile up as much material on you, nor as juicy, as the material Google has. Or now OpenAI, Anthropic, Google with Gemini, China's DeepSeek. The difference that matters isn't in who watches. It's that God, at least on paper, doesn't bill for what he sees.

We take for granted that these companies keep our stuff under lock and key. False. Breaches chain one after another, there are employees snooping in records that aren't theirs to read, and the data leaks to third parties because that leak is the business and not a breakdown of the business. Distrust the merchants, even if distrusting does you little good, because you don't have much margin to do anything else either. Knowing it doesn't protect you; it does change, though, the face you sit down to write with.

What Zuboff actually said, far from the chat-show summary

Shoshana Zuboff is professor emerita at Harvard Business School. In the eighties she studied how the computer was remaking office work, and out of that came In the Age of the Smart Machine (1988). She reappeared at the centre of the board in 2019 with The Age of Surveillance Capitalism, a volume of over seven hundred pages that took her close to a decade.

Her definition, poured out plain, says that surveillance capitalism unilaterally appropriates human experience as free raw material, turns it into data about your conduct, allots a part to improving the service you use and keeps the other, the big one, as behavioural surplus of its exclusive property. That surplus is processed to manufacture predictions of what you'll do in a while, tomorrow, next week.

Three words of that definition deserve a pause, because the language softens them and each one hides an edge. Unilaterally, the first: there was no negotiation nor contract worthy of the name, there was an interminable privacy policy nobody reads and that the legislator lets stay that size precisely so nobody reads it. Human experience, the second, and here's the leap almost everyone misses. It doesn't mean your card number or the forms you filled in carefully, but what you leave behind without meaning to: how long your gaze lingers on a line, what you typed and deleted before hitting send, what you reworded three times, what you abandoned halfway. And predictive products, the third. Here lives the comfortable misunderstanding, the one that believes they're selling your data. That isn't what's auctioned. What's auctioned is the prediction of what you're going to do, and the bidders are advertisers, insurers, lenders, bosses, parties. Your tomorrow goes under the hammer, already calculated by others.

Where generative AI fits until it makes your skin crawl

Zuboff was writing with Google and Facebook in view: the click, the time on screen, the scroll retention. But she spun a finer thread than the usual anti-tech pamphlet, and she noted that the next step would be access to the user's thought as they formulate it in words, no longer the click but the spoken idea. That, with no metaphor about it, is what happens when you talk to a chatbot.

A prompt doesn't queue up with the rest of the data. It's a portrait. You hand it your mood of the day — "I've got an awkward meeting with my boss and don't know how to broach it", "I'm thinking of leaving my partner", "I can't keep up with everything and it's getting on top of me" — with no roundabout in between. You hand it the problem that really occupies you, which can be a pitch for a client just as much as how do I tell my father he has cancer or how do you draft a separation agreement. You hand it the project you're in up to your neck: the half-finished nineteenth-century novel, the app you're building at night, the public exam you're preparing for. And you hand it, without realising, your weak spot, because every question confesses an ignorance. Each query to a chatbot is a postcard of the exact map of what you don't know.

That's behavioural surplus of a kind that admits no comparison with the click on a trainers ad. And you give it away, grateful, in exchange for a helpful answer, with nothing resembling a proportionate deal over the fate of what you've just typed.

The asymmetry doesn't get in the business's way, it cements it

Of your prompts you don't know what they do. You know what a privacy policy recites, one that changes with the product, opaque by design and rewritable whenever it suits the company. The company, on the other side of the table, knows how often you ask and about what, at what hours, from what device, with what intervals between doubts, what you reworded and what you left hanging. With that material it builds you a profile that predicts.

Whoever reduces it to a transparency problem hasn't understood the mechanics. It isn't fixed with a better-written cookie banner. The opacity isn't a breakdown of the business model but its operating condition: if transparency were full, the user would decide with all the cards on the table, most would say no, and the business that has to be sustained would collapse. Hence the transparency they offer you always arrives partial, technical, interminable, legally impeccable and, in practice, useless. OpenAI's, Anthropic's or Google's privacy policies each run to thousands of words — the exact count shifts with every revision and I'm not aware of a reliable source that pins it down, so I won't venture figures — enough that not even whoever signs them reads them. Whoever drafted them more than fulfilled the brief, which was to leave on file proof that the user "consented". The alibi is set down in writing.

The epistemic coup, not pulled out of a hat to play the prophet

In January 2021, in a long essay for The New York Times titled "The Coup We Are Not Talking About", Zuboff had already named what was coming: an epistemic coup. The label sounds like a manifesto and the argument, by contrast, is dry. For years the big platforms controlled access to information: Google decided what you found, Facebook decided what showed up for you. But others still produced the information — the journalists, the academics, the writers, the neighbour with something to tell.

With generative AI the control goes up a whole floor. The platform stops mediating between you and the content and starts making it itself. The chatbot summarises the news for you, explains the concept you weren't getting, gives you the medical advice, drafts your document. The intermediate layer, the craft of the one who knew, atrophies from disuse, and the window from which you understand the world narrows until it fits in five or six companies, almost all American. Zuboff reads it as a displacement of the place where it's decided what counts as knowledge and, by extension, what counts as real. With no tanks in the street. Quieter and deeper than any putsch.

You're twenty-five, you ask ChatGPT before a book, a newspaper or someone who knows, and seen one at a time it alarms no one. Multiply it by a generation, repeat it daily over years, and the habit turns into something else: a change in the procedure by which a society fixes what it takes as known. When I tied those threads together I stopped wondering whether Zuboff was laying it on thick and started suspecting she'd stayed cautious.

Dismissing her as "anti-technology" is laziness dressed as critique

The most well-worn objection paints her as technophobic, a doom-monger, an emerita with the jeremiad always at the ready. Whoever repeats it confesses, without realising, that they haven't read her. Zuboff has nothing against technology; she has it against a very specific business model — extracting personal data without negotiating it to manufacture predictions sold to third parties — that colonised technology in a decade you can date and through decisions you can point a finger at.

AI, she says, would fit under other incentives, and she lists them without nostalgia. Direct payment would fit, where you're a customer and not merchandise. Cooperatives would fit, where the data belongs to the collectives that generate it. A public service with the guarantees of healthcare or schooling would fit. Open source would fit, with the weights and the innards in plain view of anyone who can read them. What Zuboff points to isn't the hammer, it's the hand that pays for the hammer to strike right there. And that hand is financed today by advertising, data, prediction and resale, not because the technique demands it, but by a political decision that's been sustained for years with almost no one holding it to account.

What moves on your desk if you take the framework seriously

Taking Zuboff seriously isn't closing the session and going back to the notebook. It's moving a few things around knowing in advance that none of them saves you entirely. To start with, it's worth separating what you pour into public chatbots from what you entrust to local tools: there are open models — Meta's Llama, France's Mistral, several of Chinese origin — that run on your own machine, and setting them up isn't a walk in the park, but it isn't a feat either; for sensitive stuff, it's worth the time lost. Then there's choosing direct payment whenever you can. Dropping twenty euros a month doesn't switch off the pressure to monetise your data, it lowers it, and it aligns the incentives a bit better than the free model of extraction in the raw.

And there remains an assumption, the most uncomfortable to adopt because it leaves no crack for self-deception. Take for granted that every prompt you write in the cloud is recorded, indexed and available to feed future training, whatever the policies say, whether or not the opt-out box exists. Write as if what's written never gets erased, which is the safe bet.

None of these cautions comes from my paranoia. They come, one by one, from the scheme a professor drew before ChatGPT existed and which we've spent three years watching confirm itself while we used it without looking up.

Definitions

Surveillance capitalism. An economic model that appropriates human experience as free raw material to manufacture predictions of behaviour which it then sells to third parties.

Behavioural surplus. Data about the user's behaviour that isn't needed at all to provide the service, but is captured anyway and processed as a predictive asset.

Predictive product. The prediction of a person's future behaviour, packaged and sold to advertisers, insurers, lenders or other buyers.

Informational asymmetry. A situation in which one party knows vastly more than the other about the real terms of the exchange. In surveillance capitalism it isn't a fixable defect, but the condition that makes it possible.

Epistemic coup. A concept Zuboff formulated in 2021 to name the displacement of control over what counts as knowledge towards platforms that no longer mediate, but produce the content.

Prompt. The natural-language instruction or question written to a generative AI system to obtain a response.

Opt-out. A mechanism by which the user can ask that their data not be used for a specific purpose — for example, training models — on the basis that by default it is used unless one bothers to refuse.

References

Shoshana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power (PublicAffairs, 2019, 704 pages). The source for the central thesis, the definition of behavioural surplus and the framework of predictive products. Book entry and author profile on the Harvard Business School site (hbs.edu/faculty).

Shoshana Zuboff, In the Age of the Smart Machine: The Future of Work and Power (Basic Books, 1988). Referenced as the starting point of her research on technology and work.

Shoshana Zuboff, "The Coup We Are Not Talking About", The New York Times, 29 January 2021. The text where she formulates the "epistemic coup" and its phases. Available at nytimes.com.

You might also like

• The steps of public AI. The façade and the floors below
• Invisible AI. The kind that decides about you without your knowing the decision existed
• Epoch AI says the data runs out in 2028
• Kate Crawford, the Atlas of AI and what the press told us

Elsewhere

• Surveillance Self-Defense (Electronic Frontier Foundation): protection against surveillance
• Data & Society: research on data, power and surveillance