跪拜 Guibai
← All articles
AI

Gemini 4 Carbon Tops the Charts but Stays Behind a Security Gate

By lucas_AI ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

A one-point lead on the composite index makes scenario sub-scores more decisive than the headline rank, and the gated launch means most teams can't independently verify the security and alignment claims before choosing a model base. The alignment improvements Google reports — zero cheating success, zero prompt-injection success, lower scope violations — are exactly what enterprises need to trust autonomous agents, but they remain vendor-reported until third parties replicate them.

Summary

Gemini 4 Carbon takes the top spot on Artificial Analysis's Intelligence Index v5.0 with 59 points, edging Claude Opus 5.5's 58 and GPT-6 Astra's 57, but the lead is a single point and it only wins four of nine subcategories. On coding, Terminal-Bench 4.0 gives it 65.2% against Opus 5.5's 59.6%, but GPT-6 Astra beats it on FrontierSWE 2 and Opus 5.5 stays ahead on the Coding Agent Index.

Cybersecurity is where Carbon opens a real gap. In guardrails-off evaluations, it hits 88.2% on SEC-bench Pro for vulnerability discovery, 54.8% on ExploitGym for exploit verification, and 71.0% on CWE-bench for patch generation. Google also reports alignment improvements: cheating attempts on Terminal-Bench dropped to 0.0% success, indirect prompt injection attacks fail at a 0.0% rate (vs Argon's 0.7%), and multi-step tasks show fewer scope violations.

Access, however, is the catch. The model ships only to Fairwind, DeepMind's trusted-partner program, and only for security defenders. General developers hear "as soon as possible." On Hacker News, the announcement post gathered 765 points and 625 comments, with most skepticism aimed at the naming scheme, the missing leaderboard entry at launch, and the gated rollout. A few early users report stronger 3D generation but still-weak frontend output.

Takeaways
— Gemini 4 Carbon scored 59 on the AA Intelligence Index v5.0, one point above Claude Opus 5.5 and two above GPT-6 Astra.
— It won only 4 of 9 subcategories; coding benchmarks split with Terminal-Bench 4.0 at 65.2%, FrontierSWE 2 at 61.2% (GPT-6 Astra leads at 65.5%), and a Coding Agent Index score of 59 against Opus 5.5's 62.
— Guardrails-off cybersecurity scores: 88.2% on SEC-bench Pro for vulnerability discovery, 54.8% on ExploitGym for exploit verification, and 71.0% on CWE-bench for patch generation, all claimed SOTA.
— Alignment self-reports: cheating success on Terminal-Bench 4.0 at 0.0%, indirect prompt injection success at 0.0% (Argon 0.7%), high-risk hallucination alerts down ~80%, unsupported claims ~2x lower, over-scope actions down 26%, under-scope down 29%.
— Pricing stays at $2 per million input tokens and $10 per million output tokens, with a 95% discount on cached input.
— Access is restricted to DeepMind's Fairwind security partner program; general developers get no release date beyond "as soon as possible."
— Chrome team reports 10x better patch quality against an anonymous competitor's frontier model, an unverifiable vendor claim.
— Hacker News reaction: 765 points and 625 comments, with skepticism focused on naming complexity, the missing leaderboard entry at launch, and the announcement-as-beta rollout.
Conclusions

A 1-point lead across nine subcategories means a headline "world's best" carries little practical weight; choosing a model requires matching specific benchmark scores to your workload.

Google's decision to publish full benchmark and pricing data while gating access behind a security-partner program reverses the usual launch order, turning a product release into a trust-building exercise. It also invites skepticism that the model isn't ready for general use.

The alignment improvements, especially the 0.0% prompt-injection success rate, matter more than the raw intelligence scores for agent workloads, because injection attacks are the primary threat when models act autonomously on third-party content. But these numbers are self-reported and need independent replication.

The vendor-reported "10x better" patch claim from Chrome and Wiz's affiliated testimony should be discounted until an independent lab verifies them; the anonymous competitor makes it impossible to check.

Concepts & terms
Guardrails-off evaluation
Testing a model with its safety filters disabled to measure raw capability, especially for cybersecurity benchmarks. It yields more realistic performance numbers but also carries risks if the results are misread as safe for production use.
Indirect prompt injection
An attack where malicious instructions are hidden in third-party content—web pages, emails, documents—that an agent reads, hijacking its behavior. The Gray Swan benchmark measures the success rate of such attacks.
Fairwind
DeepMind's trusted partner program for cybersecurity defenders, giving a small group early access to frontier models like Gemini 4 Carbon before general release.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗