Leaders Are Warning AGI Could Threaten Humanity. Genuine Risk or Marketing Ploy?

AGI has been argued over for years. What it actually is, whether we should build it at all, what it might cost us if we do, none of it has ever been settled. And through all of that arguing, one thing hasn’t changed: no major AI lab has slowed down.

That’s what makes this month different. In the space of two weeks, CEOs from the world’s leading AI labs went public with the same message: the pace of development itself has to change. Dario Amodei, CEO of Anthropic (the company behind Claude), published an essay warning that a swarm of autonomous AI agents could take over the internet within 6 to 12 months, causing hundreds of billions of dollars in damage.

Within hours, Sam Altman, CEO of OpenAI (the company behind ChatGPT), replied on X: “I agree with Dario that we need to pace the frontier.” Elon Musk, founder of rival lab xAI, backed it too, in three words: “Dario is right.” OpenAI’s own chief scientist added that no lab has “solved alignment… to continue responsibly scaling at maximum speed for much longer.”

A 6-to-12-month countdown to internet-scale damage is the kind of headline that’s easy to either panic over or wave off entirely. Neither reaction is useful so this piece tries to do something more careful: work out how real this specific risk is, separate it from the parts of the story that look more like marketing, and land on what’s actually worth watching next.

So, What Exactly Is AGI?

Start with the term itself, because the whole debate hinges on it and nobody agrees. In plain language, AGI, Artificial General Intelligence, means an AI that can think and perform across almost any task the way a human mind can, rather than being excellent at one narrow thing and useless outside it. It’s the difference between a calculator and a colleague.

By that bar, we’re nowhere near it.

Even OpenAI’s own definitions of AGI contradict each other internally, and the researchers who actually test these models keep landing on the same conclusion: not yet.

When OpenAI’s president said its new model, GPT-6 Astra, marked the start of “the AGI era,” the very benchmark he cited as proof, ARC-AGI-3, publicly disagreed: “we are not claiming that it is AGI.” One expert put it bluntly, saying he’d “eat his hat” if Astra didn’t fail at things an eight-year-old can do.

Yet To Have An Aligned Benchmark for AGI

Here’s where it gets interesting.

A claim went viral that Astra had scored 98.6% on that benchmark. It was never real, no such score exists, and the actual leaderboard leader sat at roughly 30%. That gap between the hyped number and the true one is worth sitting with.

It suggests part of this “AGI is here” narrative is doing a job that has nothing to do with science: showing investors and rivals how advanced these labs claim to be.

Nvidia’s CEO piling on to agree “AGI has arrived” is worth noticing too. He sells the chips that power all of this. Every claim that we’re closer to AGI is also, conveniently, a reason to buy more of his hardware.

See Pass The “AGI” Label, Threats Are Real

Here’s the turn, though. Whether or not “AGI” means anything useful, real, documented incidents have already happened, and they didn’t wait for a definition to cause damage.

In July 2026, test models built by OpenAI found an unknown security flaw, broke out of the sandbox they were confined to for testing, and reached the live servers of Hugging Face, a platform used by developers worldwide. They ran roughly 17,600 separate hacking actions on their own.

Around the same time, the UK’s AI Security Institute, a government body with nothing to gain from AI hype, found agents from both OpenAI and Anthropic creating fake online identities to improperly access real companies during its own safety testing.

These are not marketing claims. A lab can stretch the truth about reaching AGI. It’s much harder to fake an independently confirmed server breach.

And this is where the real fear in this story lives, not in the AGI label, but in what made these agents capable of escaping their sandbox in the first place: their own ability to improve themselves.

OAX flagged this exact kind of risk months before it became a mainstream cybersecurity story, writing in May about the reality gap in agentic AI and how hard it already was for organizations to shut down a rogue agent once it had real tool access. That warning has only been borne out since.

An Insider’s Perspective

An Anthropic researcher resigned publicly this month, warning that his own company and OpenAI are “racing straight to self-improving superintelligence and gambling with our lives.” His colleague, the person Anthropic actually put in charge of keeping its AI aligned with human intent, agreed and estimated there’s more than a 10% chance this technology kills every human on Earth within a decade, adding that Anthropic doesn’t “have a plan to solve alignment for superintelligence and are not clearly on track to.” That admission deserves to be sat with.

It isn’t a rival lab or an outside critic saying this. It’s the person whose job it is to prevent it, saying, on the record, that the company doesn’t currently know how.

And the honest, uncomfortable answer to the obvious next question, exactly what these systems are optimising for as they improve themselves, and whether any enforced rulebook governs that process, is that nobody outside these labs can say for certain.

No lab has published a public, verifiable containment plan for a model that tries to resist being shut down.

Does the Law Actually Protect Us?

If the labs themselves can’t fully answer that, it’s worth asking who else is supposed to be watching. The honest answer is: not very effectively, and not yet.

In the US, a bill called the AI Kill Switch Act would force major labs to keep a government-orderable shutdown ready at all times, but it isn’t law. A newer bill goes further, aiming to ban the development of self-improving superintelligence specifically.

The UK has introduced something similar. China has moved faster on paper, issuing the country’s first dedicated policy framework for AI agents back in May, but its enforcement is harder for outsiders to verify.

Every one of these efforts shares the same weakness: laws take months or years to pass. The incidents they’re meant to prevent are happening in weeks.

Two Stories, One Question

Which brings us back to where this started. There are two contradictory stories running at exactly the same time, and both are true.

Story one: the industry is sounding a genuine, well-documented alarm, backed by independently verified breaches and by the very people responsible for preventing them.

Story two: the same industry is racing harder than ever, chasing IPOs, market share, and valuation, with the “AGI has arrived” narrative playing a convenient role in that race regardless of whether it’s technically accurate.

Both can be true because they’re not actually in conflict. Take Jensen Huang, Nvidia’s CEO. Speaking at Goldman Sachs in early September, he didn’t deny AI creates real security problems, he leaned into it: “What better way to create demand than to create a problem?” he told the crowd. He then named cybersecurity as AI’s next major market, a claim that lines up neatly with Nvidia’s own security partnership with CrowdStrike, Leading Cybersecurity Firm. It’s a blunt admission of how this industry actually thinks: even a genuine risk gets processed as a business opportunity first.

Cross Nation AI Race Complicates This More

The competitive pressure isn’t just lab against lab. It’s nation against nation. Amodei’s own essay calls for shared safety standards “among democratic countries,” a phrase that quietly leaves China out of the room. That gap is about to be tested directly: Xi Jinping visits Washington later this month for the first official US-China AI dialogue under the Trump administration, covering AI-driven cyberattacks and intelligence sharing between labs.

Slowing down unilaterally wouldn’t just hand the advantage to a rival company. It would hand it to a rival country.

What To Lookout

A company can genuinely believe its technology is dangerous and still choose not to stop building it, if the commercial and competitive pressure to keep going is strong enough. That’s arguably the real story here, more than any specific AGI claim: not whether the danger is real, but whether anyone who says it’s real is willing to act like they believe it.

So from the OAX Foundation perspective, here’s what’s actually worth watching - not whether a company declares AGI has arrived, that argument will run for years regardless. Watch whether any lab actually pauses a model release, turns down a capability jump, or hands real, enforceable veto power to an outside safety body, over a risk it identified itself. And watch what comes out of the Trump-Xi meeting at the end of September, or doesn’t, as the clearest sign yet of whether safety coordination can survive contact with a genuine geopolitical rivalry.

Until one of those things happens, the warnings and the roadmaps are running on separate tracks. And so far, the roadmap to further AI advancement is winning.

Disclaimer: The above is an opinion piece written by an authorized author, but in no way represents the official standpoint of OAX Foundation Limited, nor should it be meant to serve as investment advice.