A Hospital 3D Twin Built With Codex: Where AI Fails on 'Does It Look Right'

Three.js Hospital 3D Big Screen in Practice: Codex + Blender MCP + React from Modeling to Interaction

Author: 夏天要喝冰可乐

theme: cyanosis

First, look at the design mockup. This is what I wanted to build.

Now, honestly: the live page is uglier than this. The materials aren't up to par, the vegetation is a few color blocks, and the back facade has basically no detail. But it can rotate, it can be clicked, it can walk from the courtyard into a floor, and the numbers add up.

Online version (recommended on desktop): hospital-twin
Code: github.com/Xxcool/hospital-twin

Let me be clear about how this project came about: modeling, coding, and bug fixes were mostly me giving Codex tasks, accepting or rejecting the output, and sending it back to redo. My job was to judge "does it look right" and "can it be used"; its job was to get the work done. The source is about 800 lines, the model is 180k triangles / 112 meshes, and we went back and forth many times.

Also clarify the boundaries: this is a personal demo, not an official hospital thing. The data is all simulated, the interior space is a schematic design, and you shouldn't use it for navigation or any operational decisions.

1. Don't rush to have AI write code; first think about how to delegate tasks

The first pitfall I stepped into was not in the code but in the process: if you just say "help me make a hospital 3D big screen," Codex will give you a runnable, pretty, but completely illogical thing — the model is randomly placed, the data is random numbers, and the camera is hardcoded.

What really works is splitting the work into six stages, each with a verifiable deliverable for it:

Stage	What I do	What Codex does
1. Research convergence	Lock scope to only outpatient and emergency; other buildings are just navigation	Search public data, organize spatial relationships, write research notes
2. Full-screen plan	Decide visual direction	Generate several mockups to choose from
3. Representative zone	Judge fidelity	Model via Blender MCP, export GLB
4. Sample validation	Scrutinize against real photos	Adjust materials and structure based on feedback
5. Zone-by-zone development	Define interaction rules	Write React panels + Three.js scenes
6. Acceptance	Raise requirements, screenshot check	Fix layout, fix performance, run tests

Two key points:

First, scope must be cut upfront. Hospital big screens easily pile up energy, security, work orders, and equipment, and AI will follow your lead and keep piling. I cut it down from the start to only two business areas (outpatient and emergency), and the medicine/technology building, inpatient, children's hospital, and parking building only got spatial navigation — I didn't fabricate topics I couldn't explain clearly.

Second, acceptance criteria must be verifiable. Saying "make it prettier" is meaningless to AI; saying "the entrance arc must match this real photo, the middle section retreats 16 meters" is meaningful. This experience saved me a lot of rework later.

2. The first thing AI screwed up: drew the entrance arc backward

The outpatient entrance is the most recognizable part of the campus: a low curved entrance in front, with the tall buildings behind.

In the first version I thought it looked quite good and even screenshotted it. Then I pulled up the close-up real photo of the entrance and compared — the arc direction was reversed.

The real entrance curtain wall retreats in the middle and extends forward at both ends. What AI made was a middle that bulges out and two ends that pull in. Looking from the top, the whole building's character was off.

Why did it get it wrong? Because it started from the phrase "curved entrance" and inferred a visually reasonable surface; the real building is a curve I hadn't even measured myself: y = -27 - 16 * (x/60)², retreating 16 meters at the middle relative to both ends. AI doesn't know those 16 meters, and I didn't say it, so it could only guess.

The cleanup was even more annoying. That curve is the reference for the entire building:

The curved roof cap had to be re-bent to match;
The curtain wall verticals had to be rearranged;
The canopy and doorway had to be repositioned;
The entrance paving and grass cracked at the point where the curve reversed, and needed to be remodeled to connect.

The parameter itself took a few minutes to change, but the roof cap, curtain wall elements, and the crack in the paving took almost half a day to patch. That experience completely changed one of my judgments: whether a 3D scene is credible has little to do with the fineness of a single model, and a lot to do with whether the models connect to each other. Later I deliberately added the connecting podium and paving between buildings, and only then did the campus go from "a few buildings placed together" to "a campus".

3. Technical point one: give the model a built-in ID, don't guess with screen coordinates

The classic failure in big screens: click a building, and the data on the right has nothing to do with that building. Data is data, model is model, two separate skins.

The solution is not to write smarter picking logic, but to embed identity into the model at the modeling stage. Each building, each floor group, each zone surface carries userData when exported, and we attach an empty object as an anchor:

// On load, get the anchor by convention; throw if missing; no silent degradation
for (const id of [...regions.map(r => r.id), 'floor_f1', 'floor_f2']) {
  const node = model.getObjectByName(`anchor_${id}`);
  if (!node) throw new Error(`Model anchor missing: ${id}`);
  anchors.set(id, node);
}

That throw is intentional. Early on I let it emit a warning when an anchor was missing, and after changing the model I forgot to sync; clicking some buildings moved nothing, and it took half an hour to discover the name had changed. Failing early is cheaper than failing late.

Picking also has two real pitfalls:

Pitfall one: Raycaster doesn't respect visible. intersectObject will still count hidden objects in hit results — even when the parent has visible = false. So you have to recursively check yourself:

const isVisible = o => !o.visible ? false : o.parent ? isVisible(o.parent) : true;

Without this line, when floors are expanded you might click on the neighboring building that's hidden.

Pitfall two: dragging can falsely trigger clicks. When the user holds the mouse to rotate the view, the displacement at release is almost zero, and click still fires. So I attach picking to pointerup and only recognize a hit when the displacement is less than 6 pixels:

if (!origin || Math.hypot(e.clientX - origin[0], e.clientY - origin[1]) > 6) return;
const rect = renderer.domElement.getBoundingClientRect();
pointer.set((e.clientX - rect.left) / rect.width * 2 - 1,
           -(e.clientY - rect.top) / rect.height * 2 + 1);

Note the use of getBoundingClientRect() instead of window.innerWidth — this point will save you in the adaptation section later.

4. Technical point two: don't hand-tune the camera, use bounding box + safe area

The first version of focusing had Codex hardcode a set of camera parameters for each building. It was obedient, and the parameters looked good.

The problem is that the building sizes in the campus differ wildly — the parking building and outpatient building differ by several times. With the same parameters, small buildings can't be seen clearly, and large buildings push behind the panel.

Later I changed it to let the camera compute itself. The core code is just a few lines, but there are three numbers in it that are worth a lot:

// Project the 8 corner points of the bounding box onto the screen's horizontal and vertical axes using camera basis vectors
for (const x of [b.min.x, b.max.x])
  for (const y of [b.min.y, b.max.y])
    for (const z of [b.min.z, b.max.z]) {
      const v = new THREE.Vector3(x, y, z).sub(center);
      minX = Math.min(minX, v.dot(right)); maxX = Math.max(maxX, v.dot(right));
      minY = Math.min(minY, v.dot(up));    maxY = Math.max(maxY, v.dot(up));
    }

const pixelsPerUnit = el.clientHeight / (camera.top - camera.bottom);
goalZoom = Math.min(
  1000 * scale / ((maxX - minX) * pixelsPerUnit),  // safe area width 1000 logical pixels
  540  * scale / ((maxY - minY) * pixelsPerUnit)   // safe area height 540
) * 0.92;

Three numbers:

1000 × 540: This is the area in a 1920×1080 canvas that's actually "not taken up by UI". The left navigation, right details, top title, and bottom time slot bar all occupy space. Filling the entire browser doesn't mean the user can see the building.
0.92: whitespace factor. Framing flush against the edge looks fine in a static screenshot, but feels cramped in motion.
scale: because the entire screen is uniformly scaled, the safe area must scale too; you can't hardcode pixels.

There's another subtle pitfall: the visual center of the UI is not the geometric center of the canvas. The panel layout makes the content area sit higher overall, so directly aiming at the geometric center pushes the building too low. So the target point needs a vertical offset:

goalTarget.copy(center).addScaledVector(
  up, (top + 550 * scale - el.clientHeight / 2) / (pixelsPerUnit * goalZoom)
);

This 550 is a logical coordinate measured from the design mockup. Hardcoding it is not elegant, but it's more reliable than fiddling with parameters — for such compensation values, I recommend writing a comment explaining where they come from, otherwise six months later you won't know why it's there.

5. Technical point three: floor expansion, framing must look at the final state

Floor expansion is more insidious:

The expansion animation starts from the collapsed state. If you compute the bounding box from the "current state," the camera will frame according to the collapsed size, and once the floors spread out, the rooftop will have long left the screen.

The approach is to first temporarily move the floor group to the exploded final state, compute the overall bounding box, then move it back:

for (const [id, group] of floorGroups) {
  group.visible = !zone || id === floor;
  group.position.y = lerp(group.position.y, manifest.floors.explode[id], .09);
  group.position.z = lerp(group.position.z, -manifest.floors.back[id], .09);
}

The explode / back offsets are written in the scene manifest manifest.json, not in code — because they are properties of the model, not of the page. Anyone editing the model can change them in passing, without having to come to the frontend developer.

Here's a detail about near-view handling: when you enter the waiting area view, you must hide the medtech, children's hospital, inpatient, parking, and emergency buildings; otherwise the connecting podium sits in front of the schematic interior, blurring into one mess in perspective. When to hide, which to hide, and how to restore on exit all follow the interaction state; this logic is more annoying than loading the model.

6. Technical point four: fake data trips you up the most

Big-screen articles love to hype "real-time digital twin," but I'm not hyping that. The data in this project is just four hardcoded demo snapshots: 08:00 opening, 10:00 outpatient peak, 15:00 afternoon dip, 20:00 emergency focus.

But "fake data" does not mean "haphazard data." The four time slots share one entry point, getFrame(), and the overview must be computed from the sub-items:

total.waiting = outpatient.waiting + emergency.waiting;
total.open = outpatient.open + emergency.open;

Sounds like a platitude. But in big-screen projects, the case where "the big number on top is 172, the left outpatient is 148, and the right emergency is 24" and they don't match is too common — usually three places each fill in their own fake numbers.

Viewers may not be able to say what's wrong, but they'll definitely feel "this screen is fake." Conversely, as long as the numbers add up, viewers instinctively trust it, even when every number is fabricated.

This is the most unexpected sensation I had in the whole project: credibility can be calculated, and no matter how fine the model is, it can't fill the hole of broken data logic.

The 20:00 slot also toggles night mode. The lights come on, the sky darkens, Bloom intensity rises, and the demo moves to "emergency focus," and the screen itself sinks.

Night design goal:

What actually ran (the gap is clear, but the linkage is real):

Material color adjustment relies on naming conventions: material names in the model like TJ_glass, TJ_warmGlass, floorEdge correspond to a day/night color palette. When switching to night, iterate over all materials and change colors and self-illumination, with warmGlass additionally lit to a warm yellow. Naming conventions again replace if-else.

7. Technical point five: the three pitfalls of full-screen scale adaptation

For adaptation I chose the simplest: a 1920×1080 logical canvas + overall uniform scaling. ResizeObserver computes the ratio, the root node uses transform: scale(), and non-16:9 windows keep the background centered.

The reason for not using fluid responsive is simple: a big screen is a "locked ratio, continuously displayed" thing, and the composition of the three panels is designed, not computed. If you truly open up fluid layout, the panels cram together in a small window.

But scale has three pitfalls, and I didn't dodge any of them:

Pitfall one: click coordinates. As long as you normalize with getBoundingClientRect(), scale is naturally compatible — the rect carries scale information. Using window.innerWidth is a recipe for inaccurate clicks.

Pitfall two: stacking context. transform creates a new stacking context, and position: fixed inside the scaled tree becomes positioned relative to the container. Don't put popovers in there.

Pitfall three: it's not responsive. No wrapping, no reflow; when the window shrinks, the text scales down proportionally. So layout must reserve safe margins.

There's also a very insidious problem unrelated to adaptation: when I used a headless browser to take screenshots for verification, the scene was always black. After much debugging, I found it was the launch argument --disable-gpu, which causes WebGL context creation to fail outright in newer headless Chrome. Removing that argument fixed it. If you do automated screenshots, check this first; it can save you half a day.

8. What Codex can and cannot do

This is what I really want to emphasize.

Things it does very well, and clearly faster than me:

Batch operations in Blender — uniformly changing materials across 112 meshes, organizing names, splitting and exporting, writing bpy scripts for batch processing;
Boilerplate code — GLB loading, OrbitControls configuration, Bloom post-processing, ResizeObserver, one shot;
Validation scripts and tests — consistency tests for the data model, anchor-missing checks; it writes them more thoroughly than I do;
Documentation and comments — manifest.json field descriptions, build commands, deployment config.

Things it obviously can't do, and I have to decide:

"Does it look right". The arc reversed, the building cluster too scattered, the entrance and paving cracking apart — it can't see these, because in its eyes there are only geometry bodies, no concept of "what a hospital looks like." This kind of judgment needs a reference, and the reference photos are in my hands.
Cross-file consistency. When changing one interaction state, it might change only the React panel and forget the Three.js side, or vice versa. With 800 lines spread across 8 files, it only sees part of it per round.
The acceptance criteria themselves. "When the user clicks a building, the camera flies over and data syncs" — anyone can say that. But a requirement like "after the camera flies over, the building must not be pressed behind the right-side panel" has to be thought up by me first before it has a chance to implement it.
Trade-offs. Whether to use R3F, whether to do a tour, how many floors to build — it can give suggestions, but never dares to decide for you.

So my habit when delegating now is: give it a target it can verify on its own each round — run tests, take screenshots, measure bounding box values, check that anchors are complete. Giving an instruction like "make it prettier" is equivalent to handing over the steering wheel and expecting it to find the way.

A very concrete example: the logic for layer show/hide was described vaguely at first ("don't block when entering close view"), and it implemented it in a messy way. Later I changed it to "when mode === 'floor' && zone is true, set visible = false for the five objectIds medtech / children / inpatient / parking / emergency" — one sentence, and it got it right in one go.

9. Why I didn't use react-three-fiber

I imagine someone will ask. R3F is good, but I didn't use it this time because the scene lifecycle is too heavy: GLB loading, material replacement, floor expansion, day/night switching, tour, post-processing, per-frame performance sampling — almost all of it is imperative and stateful. With R3F, most logic would still have to loop back through ref + useFrame, adding another layer of reconciler just to write <mesh> as JSX.

So I chose native Three.js, all tucked into one useEffect, using propsRef to sync the latest props and avoid re-renders interrupting the render loop, exposing only three methods to the outside: home / focus / zoom. The React side doesn't need to know how the scene turns.

The cost is clear: that file is 378 lines, and adding routes or particles would force a split. Declarative and imperative are not right or wrong; it's only about which side you're more willing to pay on.

10. Numbers, boundaries, and what's missing

Source about 800 lines (8 files), tsc --noEmit + custom data tests + GitHub Pages deployment
Model 180k triangles / 112 meshes, orthographic camera + ACES tone mapping + light Bloom, dpr capped at 1.5
Performance on the target device hasn't been separately verified — so I'm not reporting frame rates; it running on my machine doesn't mean it'll be stable on the on-site big screen
Model fidelity still has a clear gap from the mockup; the README shows real screenshots, not mockups

Next step is to refine each of the 12 asset packages the campus is split into: first make two samples with outpatient exterior and waiting room chairs, with AI generation tools producing candidates and then going back to Blender to correct. But roads, floor slabs, and interaction positioning surfaces will stay under manual control in Blender — if generated results come in as a whole block, spatial relationships and interaction bindings get destroyed.

Finally

The pitfalls this project stepped into were almost none at the level of "I don't know how to use Three.js" — they were all in the gap between "looks reasonable" and "actually works when interacting": the arc has to connect with the paving, the camera has to give way to the UI, fake data has to add up. And these gaps are exactly the places AI can't see, and only a human can watch.

So I'd like to ask a question to friends who are writing code with AI: which judgments do you keep for yourself, and which do you hand over completely?

My answer is — anything that can be verified by numbers, screenshots, or tests, hand it over; anything that requires "is it right, does it look like it, is it worth it," none of it gets handed over.

Comment below about your dividing line.

Tech stack: React 19 + TypeScript + Vite + Three.js 0.186, modeling in Blender (MCP-assisted), GLB + manifest data-driven, main tool Codex.
If you find it useful, give a like; I'll keep writing about the pitfalls in model refinement and the AI generation pipeline after I hit them.