The Real AI Risk Isn’t Superintelligence. It’s Sameness.

Long before AI becomes smarter than humanity, it may make people and institutions think more alike.

This month, one of the sharpest reversals of the AI momentum trade in years exposed how many apparently separate portfolios had converged on the same companies, the same bottlenecks and the same assumptions. At the center, according to news reports, was Situational Awareness, a heavily leveraged fund built on the era’s most widely read AI investment thesis, forced to sell its public holdings when the correlation it had embraced turned against it. A shared thesis had become common positioning and leverage turned intellectual convergence into forced selling.

Regulators saw this coming. A month ago, at the European Central Bank’s forum in Sintra, the Bank of England’s deputy governor for financial stability, Sarah Breeden, warned that AI agents responding in similar ways to similar prompts could amplify volatility in stressed markets, and floated market-wide “kill switches” to halt trading if they do. The Bank is now running herding simulations with the Bank for International Settlements and Germany’s Bundesbank.  But this month’s unwind required no autonomous agent. People who had read the same essay, bought the same names and borrowed against the same story did it themselves.

For three decades, I made my living pricing risk, managing uncertainty and watching models fail. The costliest failures were often quiet convergences: moments when many investors, each apparently acting independently, turned out to be making the same bet. Sameness enters through three doors… the stories we share, the systems we share and, increasingly, the thinking we share.

In August 2007, many of the world’s most successful quantitative hedge funds suffered extraordinary losses at nearly the same time. Their strategies had appeared independent in normal markets. But many had drawn on similar signals, held similar positions and managed risk in similar ways. When large portfolios began unwinding, the selling cascaded through everyone built the same way. What had looked like diversification in calm weather became correlation in a storm.

The lesson was not new. In 1987, portfolio-insurance strategies mechanically sold as markets fell, amplifying Black Monday. In 2008, mortgage securities were valued and rated with models built on an unusually benign history of housing prices and defaults, and shared assumptions made institutions appear safer and more diversified, than they were. Those models did not cause the financial crisis by themselves. But shared assumptions made institutions appear safer, and their risks more diversified, than they were. The issue is that when the game itself changes, errors aren’t confined to models. They become institutional: embedded in prices, policies and expectations. 

Nor is this pattern confined to finance. In 2020, with final examinations canceled during the pandemic, England’s qualifications regulator assigned grades using a statistical standardization process built heavily on schools’ historical results. In 39 percent of entries, the calculated grade came in below the teacher’s assessment, and the policy collapsed under public outcry. The mistake was not using mathematics. It was allowing a model of the past to override evidence about particular students in an exceptional present.

Artificial intelligence brings that old risk into a new phase. Today’s systems are extraordinarily useful in stable, well-specified settings, where they find patterns in vast data at remarkable speed. But fluency is not judgement.  A system can produce a plausible, reasonable-looking answer in circumstances it does not recognize as fundamentally different from its past. 

Now consider the risk at scale, as more institutions rely on a relatively concentrated layer of foundation models, cloud providers, common data sources and shared ideas about what counts as a good decision. Their products may differ at the surface while sharing deeper dependencies and blind spots — and the shared premise becomes infrastructure, operating at machine speed.

The Financial Stability Board has warned that widespread use of common AI models and data sources could increase correlation in trading, lending and pricing, and has identified concentration in AI hardware, cloud services and pre-trained models as a potential source of systemic vulnerability. The threat is not simply that an AI system will be wrong. Human beings and institutions can be wrong, too. The threat is that one kind of error can be replicated across decisions at enormous scale and speed, while widespread adoption is mistaken for independent confirmation.

There is a temptation to answer all this by keeping a human in the loop. 

But the humans in the loop are converging too. Doctors now edit machine-drafted clinical notes; analysts edit machine-drafted memos; students edit machine-drafted essays. Done habitually, across a profession, these shortcuts pull millions of separate reasoning processes toward the same statistical center. A monoculture of minds forms alongside the monoculture of models, and it is harder to see, because every document still carries a different byline. And a single designated reviewer — seeing the same dashboard, facing the same pressure to approve — becomes a rubber stamp or another single point of failure. Judgment is useful precisely because it is plural.

A resilient institution needs multiple centers of judgment: people with different information, methods, incentives and authority to challenge the model and one another. It needs people close enough to the facts to see what a centralized system has missed, and secure enough to say that yesterday’s model is answering the wrong question. It must protect their agency when dissent is inconvenient. This is not an argument against artificial intelligence. It is an argument against trading resilience for efficiency. 

A century ago, the economist Frank Knight distinguished measurable risk from true uncertainty: situations in which probabilities can be estimated from those in which they cannot be known well enough to justify calculation. AI is powerful in the former, and it can even help explore the latter. The danger begins when institutions mistake its fluency for evidence that the uncertainty has been resolved — when the underlying rules are changing and yesterday’s data cannot tell us what matters next.

We should treat sameness itself as a risk. Regulators should test common dependencies and correlated behavior, not only whether each institution’s system performs well in isolation. Corporate boards should ask not just, “Does this system work?” but, “How many others will fail with us if it is wrong?” And inside institutions, the discipline is concrete: require independent first-pass judgments on consequential decisions before anyone sees the shared AI synthesis. Insist that machine summaries preserve competing interpretations and dissent rather than reporting only consensus. Then map where supposedly independent controls quietly rest on one model, one dataset or one provider and stress-test what happens when they fail together.

The purpose of human oversight is to preserve a living ecology of independent judgment that no single model, manager or institution can flatten into consensus. Correlated failure is usually invisible in good times. But when every trade follows the same signal — or every loan denial, diagnosis, hiring decision or emergency response rests on the same unexamined premise — the warning will have become the event. The systemic risk of artificial intelligence, in other words, is not the intelligence. It is the artificial consensus.