跪拜 Guibai
← All articles
Artificial Intelligence · Agent

Jev: The Decision Model That Made 'Judgment' a Function Call

By 米小虾 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams doing high-volume classification, routing, or guardrail checks can now run judgment on every request at a fraction of the cost of a frontier LLM. The catch: confidence scores only mean something when validated on your own distribution, and the industry's calibration gap is real.

Summary

Jev changes how models handle 'judgment' tasks by replacing token generation with a single forward pass over typed questions. The API takes a `state` (text or JSON) and a set of questions (choice, score, noul, box), and returns answers with full probability distributions plus a confidence score derived from the distribution. This eliminates the serial token-by-token cost, making it up to 278x cheaper than frontier models like GPT-6 Astra for routing tasks. But the headline numbers are measured against a strawman: against cheap models like GPT-6 Luna, Jev is only 2.8x cheaper, and its real advantage is latency and type safety.

Beneath the marketing, Jev's calibration is the hard part. Independent studies show systematic overconfidence on relevance judgments (only 66% of documents flagged >0.9 are truly relevant), 'confidence inversion' across datasets (AUROC 0.38, worse than random), and option-induced abstention (95% abstain when an 'unknown' option is present). The 'zero hallucination' claim only means answers stay inside your option set; it says nothing about correctness. TypeSafe's own RLCD training optimizes for calibrated probabilities, but that calibration is a batch property, not a per-request guarantee.

The ecosystem is already crowded: Microsoft, Cloudflare, OpenAI, and others shipped similar models within weeks. On third-party leaderboards, Jev 1.14 ranks mid-pack (Index 66.2) with 92ms latency, while open-weight models like mammoth-1 beat it on both Index (71.4) and ECE (0.034). The interface is becoming the standard—everyone is copying the 'state + questions' contract—but Jev no longer leads on quality. For teams, the playbook is to treat Jev's confidence as a routing signal, validate on 300–500 in-house examples, and reserve escalation for low-confidence outputs.

Takeaways
— Jev's API requires a `state` and typed questions, and returns answers with probability distributions over a closed option set, not generated text.
— Output tokens are counted but not billed; input costs $0.042 per million tokens.
— On a typical 300-token routing request, Jev costs $12.60 per million calls vs $3,500 for GPT-6 Astra (278x cheaper), but only 2.8x cheaper than GPT-6 Luna.
— Jev's 150ms median model-side latency is 12.6x faster than GPT-6 Astra's 1.89s, but network overhead can inflate end-to-end latency to ~1.2s.
— The `confidence` field is a statistic derived from distribution shape, not a probability of correctness.
— Independent tests show Jev overconfident on relevance: only 66% of documents with >0.9 probability were truly relevant.
— RSAJEV found cross-dataset confidence AUROC of 0.38, meaning the model is more confident when wrong.
— Adding an "unknown" option caused Jev to abstain on 95% of ambiguous samples in KoBBQ audits.
— Jev does not support fine-tuning; all accounts share the same weights.
— Recommended rollout: validate on 300-500 labeled examples, compute ECE, test option-order effects, and use dual-threshold escalation with a fallback model.
Conclusions

The mechanism of reading probabilities from a single forward pass is easily reproduced, as the "Inner Jev" paper shows; the durable advantage lies in calibration data, parallel sampling, and turning the API into a product.

Jev's interface is becoming the de facto standard while its own ranking slides, mirroring how early API definitions can outlive their creators' performance lead.

"Zero hallucination" is a category error: type safety prevents out-of-set answers, but high-confidence wrong answers are still hallucinations constrained to the answer set.

The cost story depends entirely on the baseline: saving two orders of magnitude against frontier models unlocks new workloads, but against cheap models the only reasons to switch are latency and type safety.

Concepts & terms
Decision model
A model that outputs probability distributions over a finite set of options instead of generating text, enabling fast, low-cost classification and routing.
RLCD (Reinforcement Learning for Calibrated Decisions)
A training method that penalizes high-confidence errors to make model probabilities match long-run accuracy.
ECE (Expected Calibration Error)
A metric that measures the average absolute difference between predicted confidence and actual accuracy across bins; lower is better.
Noul
A primitive in Jev for binary yes/no questions, returning a 0-1 probability without a separate confidence score.
From the discussion
Featured comments
米小虾 1 likes

Jev model ecosystem: https://github.com/supermanc417-alt/awesome-jev-hub, containing 58 application projects · 12 directions · 11 community lists...

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗