The future conditioned by old data. The new is decided by you or by it without you noticing

The models that filter resumes at large companies run on hiring data from the last ten years. The ones that assign credit scores look at your history and that of people "like you" from five years ago. The ones that suggest housing prices work with values from two years back. Any decision you make with AI support is mediated by data that no longer represents the present. The new is decided by you, decided by it, decided by something in between — and almost always decided by it without you noticing.

The inertia not announced on the screen

Every dataset has a date. That date is always earlier than the moment the model is used, and almost never appears in the interface the user sees. When a system offers you a recommendation, a score or a filter, the word that would come to mind if you knew what was happening would be memory, not anticipation. The machine remembers. It hands you back the statistical median of the period it was trained on and presents it to you as a diagnosis of the present.

This isn't a defect or an oversight. It's structure. Supervised learning consists, precisely, of learning regularities from a historical set and applying them to future cases on the assumption that those cases belong to the same distribution as the historical set. While the assumption holds, the system works. When reality moves, the system keeps answering with the same fluency as before, now about a world that has already changed. It doesn't warn you.

What sets this lag apart from other technological lags is its operational invisibility. When a paper map is out of date, you notice because the street isn't there. When a model is out of date, you don't notice, because it gives you a plausible street that is in fact the one that was there before they rebuilt it.

Your resume filtered by a decade

The canonical case is the ATS, the automatic applicant tracking system that filters resumes before a human sees them. Amazon built one between 2014 and 2017, trained on resumes received over a decade. The result was documented by Reuters in 2018: the system learned that the good candidates of the last decade had been mostly men, and systematically penalised any mention suggesting a woman in the resume, down to the word "women's" in "women's chess club." Amazon abandoned the project. The interesting detail isn't the gender bias — that already has its own literature. The detail is that the system worked exactly as it should have. It was predicting, with statistical precision, the type of candidate the company itself had hired for ten years. That is, it predicted continuity, which is the only thing a learning-based system knows how to predict.

The problem scales beyond the Amazon case. Fuller and others, in Hidden Workers. Untapped Talent (Harvard Business School / Accenture, 2021), report that 88% of respondents acknowledge their automatic filters discard qualified, high-profile candidates. They estimate around twenty-seven million hidden workers in the United States, profiles the system renders invisible because they don't resemble the profiles the system saw. Carers who spent years out of the labour market. Veterans with atypical records. People with disabilities who requested accommodations. Any resume that deviates enough from the historical median for the filter to read it as an anomaly.

Here the first uncomfortable twist appears. This kind of failure isn't the result of an ideology, nor of a bias in the engineers who wrote the code. It's the expected behaviour of a system whose function is to project the past onto the future. The company that adopts the ATS doesn't get a predictive assistant. It gets a retroactive statistical censor sold as an objective filter.

Credit as a mirror of five years ago

In credit the same mechanism operates with finer-grained datasets. Fuster and others, in Predictably Unequal. The Effects of Machine Learning on Credit Markets (Journal of Finance, 2022), analysed around ten million US mortgages granted between 2009 and 2013 and compared default predictions with classic logit models and with Random Forest and XGBoost tree-type models. The machine learning model predicted better in aggregate. And, at the same time, it widened the dispersion of interest rates: it distributed the cost of risk better by statistical profile, but it made Black and Hispanic borrowers end up, in aggregate, paying more. The flexibility gained in precision was spent making the historical default patterns more legible, and those patterns carried embedded the whole geography of prior financial inequality.

The detail worth retaining isn't moral. It's temporal. A model trained in 2013 keeps operating in 2026 — or rather, its descendant does, reworked with successive datasets but dragging the inertia of the base corpus — and any recent change in the default behaviour of specific neighbourhoods, specific generations or specific economic sectors takes years to filter into the weight those traits carry in the decision. The mobility of the profile sitting in front of the screen travels faster than the statistical weight the model assigns it.

The paradox of success

There's an operational observation the AI business learned ahead of time and almost nobody verbalises outside internal meetings. When an automated system works reasonably well for months, it stops being audited. The human analysts who reviewed the outputs are reassigned. The control samplings are spaced out. Complaint rates fall because the rejected don't know they were rejected by a machine. The system enters a dark zone where getting it right is indistinguishable from not looking.

It's exactly then that the old-data problem starts to bite. The labour market pivoted with the pandemic. Housing prices in mid-sized Spanish cities went out of sync within twelve months. The cohorts of borrowers changed their habits. The musical genre that was marginal in the 2021 corpus is now what half the country listens to. The system, without an auditor, keeps producing the answer it produced. It accumulates invisible error, decision upon decision, with no threshold tripping, because the thresholds were configured with the data of the period where the system got it right.

Here cheap cynicism will say "well, you have to audit more." That phrase confuses the symptom with the apparatus. Auditing more costs time and money and breaks the system's main commercial appeal, which is not needing humans. The companies that would benefit from auditing are, exactly, the ones that won't audit, because they're buying the system precisely so they don't have to.

Historical bias dressed as objectivity

The part the advertising discourse doesn't touch elegantly is this. When a human makes a decision, there's someone who can be held to account. When a model makes it, the institutional response is that the model is objective. Cathy O'Neil, in Weapons of Math Destruction (Crown, 2016), already then brought the concept to its legible form: commercial algorithms encode opinions about the world, not neutral measures of the world. The system's declared objectivity is the political decision of which dataset the trainer chose.

Virginia Eubanks, in Automating Inequality (St. Martin's, 2018), took the argument to concrete cases of the US social services system, where the child-protection scoring model of Allegheny County operated on a history of reports already biased against poor and Black families. The system reproduced the amplified bias and presented it to social workers as a "data-based recommendation." The phrase is interesting: technically true, practically deceptive. Yes, it's based on data. The data is what's biased.

Kate Crawford, in Atlas of AI (Yale University Press, 2021), generalises the pattern. Every model carries an archaeology. There's a period, a place, a team and a sampling bias buried in the corpus, and all of that operates behind every prediction as a fossil layer. AI is not contemporary with whoever uses it. It's contemporary with the period it was trained on, dressed in the interface of the year it's used.

The empirical face of the lag

The ProPublica COMPAS case gave the best-known journalistic example. Angwin and others, in Machine Bias (ProPublica, 2016), documented that the COMPAS recidivism-risk assessment system, used in Broward County courts, assigned Black defendants a high violent-recidivism risk score 77% more often than white defendants, once variables like prior record, age and sex were controlled for. Dressel and Farid replicated part of the analysis in The Accuracy, Fairness, and Limits of Predicting Recidivism (Science Advances, 2018) and added an uncomfortable observation: COMPAS, with its 137 variables, was no more accurate than a linear classifier with two variables or than a random group of people with zero legal training. The system wasn't doing statistical magic. It was running a reasonable projection of historical patterns of arrest, conviction and post-release supervision, all of them loaded with the racial inertia of the very judicial system that produced the data.

The later academic discussion has gone off into bias correction, fairness frameworks, retraining efforts. It's legitimate. What it tends to forget is the structural problem. Even if the system corrected the racial bias of the history, it would still be a system that decides the present with past data. And that, in a domain where human behaviour and legislation change, is an architectural choice sold as precision.

The new, who decides it?

There's a practical question worth asking before accepting the mediation of an automatic system. Who is deciding? If the resume is filtered by an ATS before a human reads it, the machine decides, with the statistical criteria of the previous decade. If the price of the house you put up for sale is suggested by a comparator trained on transactions from two years ago, the machine decides with the price curve of two years ago, in a market that pivots every six months. If the insurance is set by a dynamic tariff that reads your profile against past cohorts, the machine decides with the prior claims history.

The human who appears in the flow appears late, and appears to confirm. This is what the aviation literature called, decades ago, automation bias: when an automatic system suggests an answer, human operators tend to accept it, especially under cognitive load. The recruiter who receives ten resumes pre-filtered by the ATS out of the thousand that arrived rarely reviews the discards. The social worker who sees a high score rarely re-analyses the family context. The firm that sees the pricing model's suggestion rarely lowers it. The human who joins at the end of the process doesn't decide. They approve.

And here the question turns uncomfortable again. Is what we decide new, when we approve the suggestion a system trained on old data has produced for today's case? The question admits no single answer. It admits the observation that, in most everyday cases, the new doesn't appear. What appears is the median of the training period, slightly rescaled, presented as a ruling on the present. What doesn't resemble that median is called an anomaly, and the word anomaly is very useful for systems that need to justify a discard without having to think about it.

Definitions

ATS (Applicant Tracking System). An automatic resume-filtering and -classification system used by companies to reduce the volume of applications reaching a human reviewer. It usually operates on models trained on the company's own hiring history.

Distribution (in ML). The statistical pattern of the data a model operates on. When the data in use "falls out of distribution" relative to the training data, the model's performance degrades without the model announcing it.

Hidden workers. A category defined by Fuller and others (HBS / Accenture, 2021) to describe the qualified candidates automatic filters systematically discard for deviating from the historical median profile. Carers, veterans, people with disabilities, profiles with career interruptions.

Scoring. The assignment of a numerical score to a profile (credit, employment, judicial, insurance) generated by a statistical model trained on past cases. The score is used as an input to automated or semi-automated decisions.

Automation bias. The documented tendency of the human operator to accept an automatic system's suggestion, especially under time pressure, without verifying against independent sources.

References

Angwin, J., Larson, J., Mattu, S. & Kirchner, L. (2016). Machine Bias. ProPublica. The reference investigation into the bias of the COMPAS system in Broward County, the empirical base of the debate on AI and criminal justice.

Crawford, K. (2021). Atlas of AI. Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press. General frame on the material and temporal archaeology of commercial models.

Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, 10 October. Press coverage of the Amazon case, cited in the body of the article.

Dressel, J. & Farid, H. (2018). The Accuracy, Fairness, and Limits of Predicting Recidivism. Science Advances, 4(1), eaao5580. Replication of the ProPublica analysis and demonstration that COMPAS is no more accurate than a simple linear classifier.

Eubanks, V. (2018). Automating Inequality. How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin's Press. Cases from the US social services system, with specific analysis of the Allegheny County child-protection scoring.

Fuller, J., Raman, M., Sage-Gavin, E. & Hines, K. (2021). Hidden Workers. Untapped Talent. Harvard Business School Project on Managing the Future of Work / Accenture. A survey of more than two thousand employers and eight thousand employees on the effect of automatic filters on access to employment.

Fuster, A., Goldsmith-Pinkham, P., Ramadorai, T. & Walther, A. (2022). Predictably Unequal? The Effects of Machine Learning on Credit Markets. The Journal of Finance, 77(1), 5–47. A study of ten million US mortgages; machine learning models increase rate dispersion between groups relative to classic models.

O'Neil, C. (2016). Weapons of Math Destruction. How Big Data Increases Inequality and Threatens Democracy. Crown. Central reference on the use of opaque statistical models in high-impact decisions, a conceptual frame on declared objectivity and biased architecture.

You might also like

• The problem with training on the past
• Invisible biases
• Cathy O'Neil and the mathematical weapons
• The training loop

Elsewhere

• AI Now Institute
• OECD.AI Policy Observatory