Claude Code vs GPT-5.4 Codex: How Computer Use Works Under the Hood
The choice between Claude and Codex for GUI automation now comes down to concrete trade-offs: Claude offers finer mouse and keyboard control with local zoom, while GPT-5.4 posts higher benchmark scores and a cleaner context management API. Builders integrating computer use need to handle coordinate scaling and image downsampling themselves regardless of vendor, so the integration code, not the model, often decides click accuracy.
Claude Code and OpenAI Codex both drive computer use through the same Agent Loop: the model reads a screenshot, outputs mouse and keyboard coordinates, the client executes the action, and a fresh screenshot goes back. The API contracts differ. Claude's Messages API requires passing the full message history each turn and uses a `tool_result` response type. OpenAI's Responses API supports `previous_response_id` to auto-link context, saving tokens, and uses `computer_call_output`. On the action side, Claude exposes about 17 actions, including right click, triple click, key hold, and precise mouse down/up, plus an `enable_zoom` flag for dense UIs. GPT-5.4 offers roughly 9 core actions but adds an explicit `environment` parameter and supports input images up to 10.24 MP.
The harder part is coordinate mapping. Both models return coordinates in the display space they were given, so a screenshot that gets silently downsampled breaks click accuracy. Claude's docs warn developers to pre-scale images to API limits; GPT-5.4 gives explicit `low`/`high`/`original` detail controls. Safety differs too: Claude runs a prompt-injection classifier that asks the user to confirm, while OpenAI returns `pending_safety_checks` that the application must acknowledge. On OSWorld-Verified, GPT-5.4 scores 75.0% vs Claude Opus 4.7's 72.7% and a human baseline of 72.4%, though the numbers come from different releases. The comparison concludes that Claude wins on input granularity and an open reference implementation, while GPT-5.4 wins on benchmark scores and API ergonomics.
The design philosophies split along a concrete axis: Claude arms the model with 17 distinct mouse and keyboard actions to express intent, while GPT-5.4 invests in higher resolution input and RL training to perceive more accurately.
OSWorld scores of 75.0% for GPT-5.4 and 72.7% for Claude Opus 4.7 sit near the 72.4% human baseline and come from different release dates, so treating them as a decisive vendor ranking overstates the difference.
Coordinate scaling emerges as the shared failure point: both systems silently downsample oversized screenshots, and unless the client pre-scales the image, the model's returned coordinates will be off. That places the accuracy burden on integration code, not on model selection.
A troubleshooting tip to add: when Computer has issues, first confirm whether it's an environment difference — if it runs locally but fails in production, nine times out of ten it's the dependency version.
Makes sense.