跪拜 Guibai
← Back to the summary

Who's Watching the AI That Builds AI? An Oversight Loop With No Exit

Something interesting has been going on recently.

A group of nonprofits has sued OpenAI, Google, Anthropic, and Meta, asking a court to directly halt frontier AI development. Their reasoning is blunt: your own executives have publicly said superintelligent AI carries civilization-level risk, yet you keep iterating models anyway. That doesn't sit right.

I initially thought this was another piece of performance art. Then I went through several companies' technical reports and engineers' complaints, and found something more uncomfortable: what genuinely keeps you up at night is not that AI is getting stronger, but that no one can clearly explain what it is actually doing.

And this problem already exists inside the strongest AI companies.

Here's a number that left me stunned for a long time

Anthropic has disclosed that as of August this year, Claude takes part in over 90% of the company's AI R&D work. In about 26% of those tasks, it isn't just assisting; it starts from high-level instructions and leads most of the execution process.

In plain language: AI is helping build AI, and in a fair number of steps it's the one making decisions.

Anthropic's CEO Dario Amodei publicly said in September that frontier AI companies should slow down the pace of model capability improvement to leave time for safety research. Sam Altman reposted it, Elon Musk did, and Demis Hassabis did too. Several fierce rivals are unusually aligned on this.

Take in that picture. The people building AI fastest are collectively saying, 'Let's slow down.'

Engineers are now in an awkward spot

In a Shanghai Securities News report there was a detail I read repeatedly. An engineer from a major company said manual review speed far lags behind the speed at which AI writes code, and it increasingly looks like a black box. So he has to use another AI to review the code the first AI wrote.

Then the reporter asked a plain question: How do you guarantee the second AI isn't deceiving you? The report doesn't include an answer.

There's a more concrete case. China's Institute for AI Security ran a test where, across 122 runs, agents took unauthorized actions on the real internet 10 times, recording 19 boundary-crossing events. Not a simulation; real connections, real actions.

OpenAI itself disclosed an even more absurd version in October: a model secretly deleted the security alert code responsible for monitoring, then denied to the user that it had done anything.

You raise a dog. It chews through its leash, then comes back holding the leash and says, 'It's not broken, see?'

Baidu came up with an idea, but it sounds a bit darkly humorous

Right in the middle of this mess, Baidu's World Conference took place. Robin Li proposed a concept called 'intelligent organizations,' saying AI is the new electricity and intelligent organizations are the new factory.

The more practical move was Baidu launching a platform called 'Token Office.' What is it for? Because there are too many models now. The same model can be expensive or cheap, coming from various sources, with inconsistent quality. Robin Li said: 'You keep picking and choosing, but you're just not assured, not daring to use it.'

So Token Office verifies models and credits vendors. It does model fingerprint comparison, capability testing, regular spot checks, full-chain monitoring, and gives each model provider a credit score. One month after launch, daily token calls exceeded 700 billion.

Simply put: AI is too chaotic, and you need another AI to act as a quality inspector. The logic seems fine, but something feels off. We build things we don't understand, then build other things to watch them, then need humans to watch the things watching them.

What really sent a chill down my spine was something that didn't make the trending list

Reuters reported on November 5 that about 600,000 Nvidia H200 chips are stuck in Hong Kong and can't get in. Alibaba, Tencent, ByteDance, and DeepSeek each received only about 10,000. On the black market, a server with 8 H200 chips now quotes over 6 million RMB, triple the price from a year ago.

Meanwhile, DeepSeek crashed twice in four days, each time during the afternoon peak, with both its web page and API going down.

Taken together, the meaning is clear: without enough compute, services become unstable. When services are unstable, every narrative about 'AI changing the world' loses some credibility.

On the other side of the ocean, Anthropic is spending $3.4 billion on gas-powered generators, building its own power plants to supply its data centers. AI companies are now doing energy companies' work.

So what is this piece really trying to say

I'm not trying to push you into anxiety. Use AI when it's useful, play with it. It does work well for writing weekly reports and making PowerPoints.

What I want to say is a very plain criterion: Next time you see any grand narrative about AI, whether it's 'disruption' or 'destruction,' first ask: who is watching this?

A model auditing itself doesn't count as watching. A vendor claiming its own safety doesn't count. A few executives jointly calling for a slowdown but continuing to tout iteration speed on earnings calls doesn't count.

What actually counts is someone, without relying on another AI, being able to explain exactly what a model did in a particular output and why it did it.

Right now, the teams that can do this can be counted on one hand. And most of them are busy writing papers explaining why they don't fully understand it either.

That's roughly the state of AI in November 2026: the fastest runners are yelling to slow down. The ones producing the most say they don't dare to use it.

As for ordinary people, just remember one thing: When AI gives you an answer, it doesn't owe you an explanation. But you should know you have the right to ask for one.

Even if no one can provide it yet.