Claude Code vs GPT-5.4 Codex: How Computer Use Works Under the Hood

Claude Code vs Codex: Deep Analysis of Computer Use Implementation Principles

I. Core Concept Clarification

Before diving into technical details, first clarify two easily confused facts:

Product	Underlying Model	Computer Use Integration
Claude Code	Claude Sonnet 3.5/3.7, Claude 4 Opus/Sonnet, Claude 4.6, Opus 4.7	Implemented directly through Anthropic API's `computer_use` toolset
OpenAI Codex	GPT-5-Codex, GPT-5.4	GPT-5.4 is OpenAI's first general model with native computer use support; earlier CUA (Computer-Using Agent) was a dedicated browser agent trained with GPT-4o vision + RL, which has evolved into a general capability

In short: All of Claude's vision models have built-in computer use capabilities; while OpenAI went from a "dedicated CUA model" to "native support in GPT-5.4 general model".

II. Architecture Overview Comparison

2.1 Overall Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────┐
│                    Claude Code Architecture                            │
│                                                                         │
│  ┌──────────────┐   tool_use request   ┌─────────────┐   screenshot/action execution │
│  │  Claude API   │ ───────────────▶ │  Agent Loop  │ ──────┐            │
│  │  (Messages   │ ◀─────────────── │  (Client)    │ ◀─────┤            │
│  │   API + Beta) │   tool_result    └─────────────┘       │            │
│  └──────────────┘                                ┌────────┴──────┐      │
│  tools: [computer_20251124, bash, text_editor]   │ Sandbox environment │
│  Beta Header: "computer-use-2025-01-24"          │ Xvfb + Mutter  │      │
│                                                  │ + Firefox/...  │      │
│  Capability source: model base vision understanding + tool call alignment └────────────────┘
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│                    OpenAI Codex / GPT-5.4 Architecture                  │
│                                                                         │
│  ┌──────────────┐  computer_call request ┌─────────────┐   screenshot/action execution │
│  │  Responses   │ ─────────────────▶ │  Agent Loop  │ ──────┐           │
│  │  API         │ ◀───────────────── │  (Client)    │ ◀─────┤           │
│  │              │ computer_call_output└─────────────┘       │           │
│  └──────────────┘                                ┌────────┴──────┐       │
│  tools: [computer_use_preview]                   │ Sandbox environment │
│  params: display_width, display_height, environment│  network disabled by default │
│                                                  │                │       │
│  Capability source: GPT-5.4 native vision+reasoning+RL training │                │
└─────────────────────────────────────────────────────────────────────────┘

2.2 Agent Loop Comparison

Both use the model requests action → user executes → returns result → loop architecture, but the API protocols differ.

Claude (Messages API):

# User code controls the full loop
while True:
    response = client.beta.messages.create(
        model="claude-opus-4-6",
        tools=[...],           # tool definitions in each request
        messages=messages,     # explicitly pass full message history
        betas=["computer-use-2025-01-24"],
        thinking={"type": "enabled", "budget_tokens": 2000}
    )
    # check stop_reason == "tool_use"
    # execute tool → construct tool_result → append to messages

OpenAI (Responses API):

# Can automatically link context with previous_response_id
response = client.responses.create(
    model="gpt-5.4",
    previous_response_id=prev_id,  # automatically manage history
    tools=[{...}],
    input=[...current step...]     # only pass current turn input
)
# check computer_call in output
# execute action → construct computer_call_output → next request

Key differences:

Claude explicitly passes the full messages array
OpenAI supports previous_response_id automatic context linking, saving tokens
Claude uses tool_result type to return results; OpenAI uses computer_call_output type

III. Computer Tool Capability Comparison

3.1 Action Set Comparison

Action Type	Claude (`computer_20251124`)	OpenAI `Computer` interface
Screenshot	`screenshot`	`screenshot()`
Left click	`left_click` (x, y)	`click(x, y, button=left)`
Right click	`right_click` (x, y)	— ❌
Middle click	`middle_click` (x, y)	— ❌
Double click	`double_click` (x, y)	`double_click(x, y)`
Triple click	`triple_click` (x, y)	— ❌
Mouse move	`mouse_move` (x, y)	`move(x, y)`
Drag	`left_click_drag` (path)	`drag(path)`
Scroll	`scroll` (direction, amount)	`scroll(x, y, scroll_x, scroll_y)`
Type text	`type` (text)	`type(text)`
Key press	`key` (key_name)	`keypress(keys)`
Hold key	`hold_key` (key) + other actions	— ❌
Precise control	`left_mouse_down` / `left_mouse_up`	— ❌
Wait	`wait`	`wait()`
Custom environment	`display_number` (X11)	`environment` (browser/mac/windows/ubuntu)
Local zoom	`enable_zoom` ✅	— ❌

Claude advantages: richer action set (about 17 vs 9 core), supports hold_key (e.g., Ctrl+click), triple_click, precise mouse button control (press/release) and other fine-grained operations.

OpenAI advantages: environment parameter explicitly tells the model the current environment type, helping the model choose the right GUI interaction strategy.

3.2 Tool Declaration

# Claude — tools array declaration
tools = [{
    "type": "computer_20251124",       # versioned tool type
    "name": "computer",
    "display_width_px": 1280,          # coordinate space declaration
    "display_height_px": 720,
    "display_number": 1,
    "enable_zoom": True                # optional feature toggle
}]

# OpenAI — tools array declaration (Responses API)
tools = [{
    "type": "computer_use_preview",
    "display_width": 1024,             # coordinate space
    "display_height": 768,
    "environment": "browser"           # environment type hint
}]

IV. Underlying Model Capability Comparison

4.1 Model Evolution Timeline

Time	Anthropic	OpenAI
2024-10	Claude Sonnet 3.5 (new) first released computer_use	—
2025-01	—	Released CUA (GPT-4o + RL training), used for Operator
2025-04	Claude Opus 4 / Sonnet 4 released, computer_20250124	—
2025-09	—	Codex upgraded to GPT-5-Codex, CUA capability rolled down
2026-03	—	GPT-5.4 released, first general model with native computer use support
2026-05	Claude Sonnet 4.6 / Opus 4.6 released	—
2026-07	Claude Opus 4.7 released, higher resolution support	—

4.2 Vision Input Capabilities

Dimension	Claude 4.6 Family	Claude Opus 4.7	GPT-5.4
Max long edge	1568 px	2576 px	6000 px (`original`)
Max total pixels	1.15 MP	3.75 MP	10.24 MP (`original`)
Recommended resolution	1280×720	1920×1080	—
Image detail	API auto-scales internally	API auto-scales internally	`low`/`high`/`original` selectable
Coordinate space matching	Requires explicit `display_width_px/height_px`	same as left	`display_width/height`

Key insight: Both models internally downsample super-resolution images, but handle it differently:

Claude does implicit processing: API auto-scales internally and returns corresponding coordinates
GPT-5.4 introduces explicit detail level selection, giving developers more control

Claude officially emphasizes: you must manually scale screenshots to within API limits before sending, otherwise the API silently downsamples, causing a mismatch between "the image the model sees" and "the coordinate space it returns" — this is the #1 cause of inaccurate clicks.

4.3 Benchmark Scores

Benchmark	Claude 4.6	Claude Opus 4.7	GPT-5.4	Human
OSWorld-Verified	~48%	72.7%	75.0%	72.4%
WebArena-Verified	—	—	67.3%	—

⚠️ Note: These numbers come from official releases at different times; test settings (max steps, resolution) may differ, so they are not fully comparable.

V. Safety Mechanism Comparison

5.1 Prompt Injection Protection

Aspect	Claude	OpenAI
Detection layer	Built-in model resistance + additional classifier detects malicious instructions in screenshots	Three types of API-level safety checks
Trigger method	Automatically runs classifier → guides model to request user confirmation	Returns `pending_safety_checks` → application layer decides whether to confirm
Check types	Unified prompt injection classifier	`malicious_instructions` (malicious instructions)
`irrelevant_domain` (irrelevant domains)
`sensitive_domain` (sensitive domains)
User confirmation	If detected, model proactively requests user confirmation for next step	`pending_safety_checks` must be acknowledged via `acknowledged_safety_checks` before continuing
Can disable	Can contact Anthropic to disable extra protection	—

5.2 Sandbox Isolation

Aspect	Claude Code	Codex
Default isolation	Reference implementation provides Docker container	Default sandbox, network access disabled
Network policy	Domain whitelist recommended	Network disabled by default, can enable as needed
Permission control	Minimal-privilege VM/container recommended	Workspace filesystem restrictions

VI. Technical Implementation Details

6.1 The Critical Trap of Coordinate Scaling

Problems both sides need developers to handle:

Original screenshot resolution (screen_w × screen_h)
        ↓ downsample
Image the model sees (display_w × display_h)
        ↓ returns coordinates
Model output click coordinates (click_x, click_y) —— note these are in display coordinate system!
        ↓ scale back
Actual execution position (click_x * screen_w/display_w, click_y * screen_h/display_h)

Claude's officially provided code pattern:

# Model returns display coordinate system, need to scale back to actual screen
scale_x = screen_w / display_w  # display_w is the value passed to API
scale_y = screen_h / display_h
actual_x = int(click_x * scale_x)
actual_y = int(click_y * scale_y)

6.2 Claude's enable_zoom Mechanism

Claude 4.6+ adds enable_zoom: True:

Model can autonomously decide to "zoom in" on a screen region
Especially effective for small targets in dense UIs (checkboxes, system tray icons, etc.)
Zoom logic happens internally in the model; developers don't need extra handling

6.3 OpenAI Responses API's previous_response_id

GPT-5.4's conversation context management mechanism:

# No need to manually concatenate message history
response1 = client.responses.create(model="gpt-5.4", ...)
response2 = client.responses.create(
    model="gpt-5.4",
    previous_response_id=response1.id,  # automatic linking
    input=[...]  # only pass current turn
)

This is more efficient than Claude's approach of explicitly passing the full messages array each time, saving tokens.

6.4 Codex's Combinatorial Capability

Codex (GPT-5.4) can use multiple tools simultaneously:

bash — execute shell commands
computer — vision + mouse/keyboard
file_search — file search
web_search — web search
Custom MCP tools

Claude Code is similar, combining computer + bash + text_editor three beta tools.

VII. Developer Experience Comparison

Dimension	Claude	OpenAI
API type	Messages API (beta)	Responses API
Context management	Manually pass messages	`previous_response_id` auto-linking
Tool declaration	`tools` array + beta header	`tools` array
Beta status	Requires beta header marker	`computer-use-preview` model-specific
Reference implementation	Docker + Python, open source on GitHub	Azure OpenAI provides Playwright examples
SDK support	Python/SDK + CLI	Python OpenAI Agents SDK + JS Agents SDK
Composability	computer + bash + text_editor	computer + Playwright + any tool

VIII. Summary: Which Is Better?

8.1 Technical Feature Comparison Matrix

Dimension	Claude (4.6/4.7)	GPT-5.4	Winner
Action richness	⭐⭐⭐⭐⭐ (17 types)	⭐⭐⭐⭐ (9 types)	Claude
OSWorld success rate	72.7% (Opus 4.7)	75.0%	GPT-5.4
Image resolution cap	3.75 MP	10.24 MP	GPT-5.4
enable_zoom local zoom	✅	❌	Claude
hold_key compound operations	✅	❌	Claude
Safety check granularity	unified classifier	three check types	OpenAI
Context management efficiency	manual history	previous_response_id	OpenAI
Native support in general model	yes (via beta tools)	yes (native capability)	tie
Training basis	base vision model + tool call alignment	RL-trained CUA + general vision	different styles

8.2 Selection Recommendations

Choose Claude if:

Need rich mouse/keyboard fine-grained control (hold_key, triple_click, etc.)
Target UI is dense with small targets (enable_zoom significantly helps)
Need open-source reference implementation for quick start

Choose GPT-5.4 if:

Pursuing highest OSWorld and other benchmark scores
Need to handle high-resolution original screenshots (10.24MP)
Want a more standard API and more efficient context management
Already using Codex or OpenAI Agents SDK

Common engineering challenges for both:

Coordinate scaling — aligning model coordinate system vs actual screen coordinate system is the foundation of click accuracy
Image downsampling — must be manually controlled before sending; cannot rely on API auto-handling
Agent Loop termination conditions — need to set max iterations to prevent infinite loops
Prompt injection protection — both have mechanisms, but developers still need good isolation

IX. Core Conclusion

Both are based on the "vision model understands screenshot → outputs mouse/keyboard coordinates → client executes → returns new screenshot" Agent Loop paradigm, but design philosophies differ:

Claude: give the model more action options, let it express intent with rich vocabulary (17 actions + enable_zoom)
OpenAI: use better training and higher resolution to let the model see clearly (RL training + 10.24MP input)

Choosing one depends more on your specific scenario and existing tech stack.

Sources

Anthropic Computer Use API Docs — https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool
Anthropic Best Practices for Computer and Browser Use — https://claude.com/blog/best-practices-for-computer-and-browser-use-with-claude
OpenAI Computer-Using Agent release — https://openai.com/index/computer-using-agent/
OpenAI Introducing GPT-5.4 — https://openai.com/index/introducing-gpt-5-4/
OpenAI Agents SDK Computer interface — https://openai.github.io/openai-agents-python/ref/computer/
Azure OpenAI Computer Use — https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/computer-use
OpenAI Introducing Upgrades to Codex — https://openai.com/index/introducing-upgrades-to-codex/
OSWorld 2.0 Benchmark — https://arxiv.org/abs/2606.29537

Comments

Top 1 of 2 from juejin.cn, machine-translated. The original thread is authoritative.

AI向善

A troubleshooting tip to add: when Computer has issues, first confirm whether it's an environment difference — if it runs locally but fails in production, nine times out of ten it's the dependency version.

怕浪猫

Makes sense.