A 600B open-weight MoE builds a full 3D Chinese Chess game in one day — here's where it shines and breaks
Open-weight Chinese flagship models have crossed into usable territory for individual developers, but the bottleneck is shifting from generation speed to verification capability. Anyone using these models for real work needs a perft-like acceptance process, because model-produced code will pass ordinary tests yet still contain version-specific and interaction-level bugs that only appear under real execution.
A developer tasked Step 5 Preview, StepFun's flagship open-weight model, with building a full 3D Chinese Chess game. The result: a 701KB single-file web app with seven uniquely shaped pieces rendered from Three.js primitives, procedurally generated Canvas textures, full rule enforcement (including cannon captures and horse-leg blocks), dynamic lighting, and a moving spotlight. Move generation passed the authoritative perft count (44/1920/79666) and 52 Node assertions guard against regressions. The whole thing went from zero to deliverable in one day.
The model's benchmark profile: 600B total parameters with 27B activated per token (4.5%), 92-layer narrow-deep Transformer, 1M token context, native text/image/video input. Open weights shipped on Hugging Face (BF16, 28 shards). Third-party scores show 44 AA Intelligence Index, 33.3% on Terminal-Bench 4.0, and 85.0% on Terminal-Bench 2.1. It lags Claude Opus 5 by 6 points on DeepSWE (67.7% vs 74.0%) at roughly one-eighth the official cost per task.
The developer's field notes catalog the failure modes: three.js r149 color management washing out textures, geometry coordinates sinking pieces into the board, reversed zoom direction, swallowed clicks during animations, and missing rotation resets on undo. The takeaway: the model handles the 90% that's straightforward, but the remaining 10% only surfaces when actually run — and that's where perft, assertions, and a human acceptance checker become non-negotiable.
Benchmark numbers without methodology are meaningless: the same model is 2.6x a rival on Terminal-Bench but tied on AA Index, and 6 points behind a closed flagship on DeepSWE. Reading total scores in isolation would lead to opposite conclusions about the model's agentic ability.
The failure modes reported are not about model ignorance but about version-specific environmental knowledge — color management in three.js r149, geometry coordinates, interaction edge cases. This sort of knowledge rarely appears in training data and requires real execution to surface, which is why human validation remains essential.
The 10% of bugs that appear only when running turn the development workflow from prompting into acceptance testing. The model accelerates the 90% that's well-represented in training data, but the last mile of correctness is a verification pipeline the developer must provide themselves.
The price-performance tradeoff is stark: paying one-eighth the cost for 6 points lower on DeepSWE is a rational trade for many batch or high-volume tasks, but useless for a mission-critical system where one wrong fix is expensive.
The open-weight release plus community quantizations changes the cost of experimentation for individual developers: they can now run a 600B-parameter model locally, something previously out of reach, even if it costs 8 GPUs for full precision.