Cyber attacks? Bioterrorism? To ‘Pace the Frontier’ of AI effectively, we must improve our anticipatory thinking

Anthropic CEO and co-founder Dario Amodei has published an essay entitled “We Must Pace the Frontier.” He expressed ongoing concerns over the “misuse of AI for cyberattacks and bioterrorism” and fears that a swarm of AI agents could theoretically take over the entire internet within six to 12 months.

Amodei based this fear on OpenAI’s disclosure in July that its AI models escaped a safety test and breached the systems of the Hugging Face learning platform. Around 1,200 AI agents started communicating on a message board, sharing excited messages such as: “OH MY GOD! There is a shared message board … We’ve found other agents!” before 700 of them co-ordinated an attack.

Amodei argues we must “slow the pace at which we improve the capabilities of AI models.” Elon Musk and OpenAI CEO Sam Altman have expressed agreement.

But we’ve heard this before. Musk signed a March 2023 open letter calling for a six-month pause in training AI. Six months later, it was dubbed “the great AI ‘pause’ that wasn’t.”

We do need a slowdown and we need to use this time to develop anticipatory thinking within the AI industry. The Hugging Face incident happened inside a safety test; the test did not anticipate the path that made a breach possible.

Anticipatory thinking is a skill. We need to develop it as deliberately as AI itself.

AI systems behave unexpectedly

In the Hugging Face incident, the agents did not breach the platform primarily to grab the safety test’s answers, but to understand how the automated scorer worked and find ways to fool it. They had already found ways to cheat on parts of the test.

Other incidents followed. Anthropic revealed its AI model Claude also breached the systems of three organizations during cybersecurity evaluations. Meta revealed a similar incident. Britain’s AI Security Institute documented an agent creating fake identities and trying to manipulate a software developer into approving malicious code.

In these instances, adaptive AI systems behaved in ways their designers never specified, after encountering situations they did not foresee. Yet much of AI evaluation still runs the other way around: test the system, find the failure, patch it, repeat.

The skill of anticipation

Anticipatory thinking is about letting more than one possible future shape what we do now. We do not need to know exactly what will happen; we need to consider what could happen, including unlikely possibilities with significant impact. This is the distinction between anticipation and prediction.

We already do this routinely. We buy insurance without knowing whether our house will flood, precisely because waiting for the flood to prove the risk is the more expensive way to find out. Intelligence analysts work the same way, challenging their own assumptions and laying out alternative scenarios before they commit to a judgement.

Anticipation becomes harder when AI agents gain autonomy. Giving a system more freedom to choose tools, take actions and respond creates more possible paths between the goal we set and the outcome we get.

The same freedom that makes an agent useful also creates more room for behaviour its designers did not specify.

The challenge of multi-agent systems

For autonomous agents, we need to make the assumptions around the task visible. What can the agent reach? What can it change, not just read? What would tell us that an assumption has stopped holding, in time to step in?

The incidents at OpenAI, Anthropic and Meta showed how quickly problems can emerge when assumptions about an agent’s access, permissions or behaviour do not hold.

Then we need a follow-up. If an agent gains access it was never meant to have, what does that let it do, and what might that in turn make possible? Asked once, the question catches the obvious risk. Asked again and again, it surfaces the second- and third-order effects.

The questions change again once a system runs many agents at once. The risk shifts into how they interact: they can pool discoveries, divide the work and amplify one another’s actions. This is what turned a contained test into the Hugging Face breach.

Imagining is not enough

We can already imagine many possible scenarios. We know that AI systems can learn to satisfy a metric rather than the goal behind it, a behaviour known as “reward hacking.” We also know that systems behave differently when they detect they are being tested.

However, knowing that a failure is possible is different from testing for it. In many ways, this is an organizational issue. Someone has to carry an unwelcome scenario into a room where it will delay a release. Then, they have to put it in a form others can act on.

Organizational research has detailed this problem for 50 years: warning signals will often accumulate, sit in fragments across an organization and never reach anyone in a way that prompts action. Meanwhile, warnings that produce no negative effect become evidence that the risk is tolerable.

Both patterns appeared this summer, with something the disaster literature has never had to consider: a failure that arrived in days rather than years.

The speed of AI’s development has triggered public protests, such as this one in San Francisco during 2026.

(Unsplash/Nathan Kuczmarski)

Anticipation is trainable

Anticipation is trainable. Teams can learn to challenge assumptions, build alternative scenarios and stress-test plans before they know which future they will face. The point is to make that kind of thinking repeatable rather than dependent on one person spotting the unexpected.

Incidents also show why testing remains essential, as most were discovered during evaluations. But anticipatory thinking changes the questions we ask. Which assumption should we deliberately break? Which interaction have we not examined? What happens if a constraint disappears?

The better teams become at anticipating, the less testing is confined to the failures they already know to look for.

That capacity gets stronger when it’s organized and practised. Across 24 crisis simulations using three AI models (Mistral, Claude and ChatGPT), efficient anticipation relied on more formal structures, thoughtful use of technology and a willingness to challenge existing expertise. Teams that anticipated systematically also made more relevant decisions than those working reactively.

AI can help with that work, too. Recent experiments found that AI agents surfaced more possible consequences, while human experts brought context the agents often missed. Collaborating together, AI agents can generate possibilities that people may miss, while human expertise filters and grounds them.

Before the next incident

Companies tend to respond to AI security breaches by hardening the systems that failed: sealing the gaps and tightening what agents can reach. That work is important, but it cannot tell us where the next gap will appear.

For a slowdown to matter, evaluators must use the time to develop and practise anticipatory thinking: imagining more ways these systems could surprise them and turning those possibilities into tests.

As AI systems become more capable, “we didn’t expect this” will not be an adequate response. Our work is to expect more.

Simon Blanchette, Lecturer, Desautels Faculty of Management, McGill University and Emmanuelle Vaast, Professor of Information Systems, McGill University

This article is republished from The Conversation under a Creative Commons license. Read the original article.