跪拜 Guibai
← All articles
AI Programming · Gemini

Inkling 2 Mini Loops Its Middle Layers 3x for Deep Reasoning in a 20GB Laptop Memory

By 可夫小子 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

This brings high-performance agentic coding to commodity laptops, removing the cloud API dependency and opening long-running local agents for a wider developer base, while exposing a new trade-off: looped depth boosts reasoning tokens but not factual recall or strict format compliance.

Summary

Thinking Machines Lab's open-source Inkling 2 Mini takes a different route to deep reasoning: instead of stacking more layers, it reuses a 12-layer block three times per token. The 26.4B-parameter MoE model stores 24 physical layers, but each token passes through 48 logical layers because the middle two blocks (layers 7-18) loop three times. An Attention Residuals mechanism with a learned gate controls cross-round information flow, and the model activates 5.4B parameters per token when accounting for loops. The result is a ~20GB memory footprint that runs offline on 32GB MacBooks or Mac Minis through Ollama, without cloud APIs.

Training uses low-bit precision throughout: first 20 layers in native 4-bit NVFP4, last 4 in 8-bit MXFP8, avoiding post-training quantization loss. The team also pruned experts from 384 to 128 per layer mid-training, then distilled 1T tokens to recover accuracy, followed by SFT and 2.2T tokens of RL. Independent testing by Stanford's Hazy Research Lab measured 0.703 Intelligence per Watt on a DGX Spark, beating several larger rivals. Official self-reported benchmarks show strong agentic coding performance (e.g., 48.2% on DeepSWE v1.1, 79.2% on SWEBench Verified bash-only), though independent verification is still thin.

The trade-offs are clear: looped depth doesn't expand factual knowledge (closed-book QA scores barely improve), and multi-turn instruction following lags smaller dense models. The model excels at tool-using agentic tasks and consumes far fewer reasoning tokens (chain-of-thought length halves from 3703 to 1661 tokens with 5 loops). For developers with 32GB machines who want a local, offline coding agent, this is a rare sweet spot, but the team warns that internal silent reasoning becomes harder to monitor as loops scale.

Takeaways
— Inkling 2 Mini has 24 physical layers; layers 7-18 loop 3 times per token, giving 48 logical layers of depth.
— Total parameters: 26.4B; with MoE, activation per token is 3.4B without loops and 5.4B with loops.
— Memory footprint is ~20GB; officially supported on Macs with 32GB+ RAM via Ollama.
— Training uses low-bit precision: first 20 layers in 4-bit NVFP4, last 4 in 8-bit MXFP8, eliminating post-training quantization.
— Expert pruning cut 384 experts per layer to 128, then 1T token distillation recovered accuracy.
— Non-loop models need 140% more inference compute to match the same loss on a 512k-token agent trajectory.
— On GPQA Diamond, 5 loop passes cut chain-of-thought tokens from 3703 to 1661 vs. 1 pass.
— Official self-reported scores: DeepSWE v1.1 48.2%, SWEBench Pro v2 (Hard) 77.9%, SWEBench Verified (bash-only) 79.2%, Vals Index v2.2 46.5%.
— Third-party test on DGX Spark: 0.703 Intelligence per Watt, beating Qwen3.6-35B-A3B (0.430), Gemma-4-26B-A4B (0.159), Qwen3.5-27B (0.036).
— Limitations: closed-book QA (SimpleQA, WorldVQA) doesn't improve with loops; AA-Omniscience score -8.6, below Qwen3.8-27B's -6.4.
— MultiChallenge 50.0% and ComplexConstraints 33.4%, both behind Qwen3.8-27B.
— Looping increases internal silent reasoning; monitoring implicit reasoning remains an open research question.
Conclusions

The loop-then-prune strategy effectively trades training compute for inference compute: you train a small model once, then gain extra depth at runtime by paying more activation per token (5.4B vs 3.4B). This inverts the usual model scaling trade-off and could shift how local AI is optimized.

The dramatic drop in chain-of-thought tokens with more loops suggests the model internalizes reasoning, which could make agentic behavior harder to audit or steer, especially at larger scales—a problem the team flags as open.

Self-reported benchmarks and a single third-party measurement leave the real-world advantage over Qwen and Gemma models uncertain; developers should re-run their own workloads before switching.

Concepts & terms
Attention Residuals
A mechanism used in looped layers where a learned gating switch selectively merges the output from the previous loop iteration with the current input, controlling how much cross-round information flows through the network.
NVFP4
NVIDIA's 4-bit floating-point format used for both training and inference, reducing memory footprint and compute cost while maintaining numerical stability.
Intelligence per Watt
A metric that measures a model's reasoning capability per unit of energy consumed, helping compare efficiency across models running on different hardware.
From the discussion
Featured comments
用户368345607470

The loop approach fits local deployment better than simply stacking more layers. Reusing the middle layers lets 24 layers achieve 48 layers of depth, and combined with native 4-bit training, the size is squeezed to around 20GB, which is more convincing than post-hoc quantization. However, multichallenge is only 50.0% and complexconstraints 33.4%, so following complex constraints remains a weak spot. Before running long-context tasks locally, validate on your own codebase first, and don't just trust the official self-reported scores.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗