跪拜 Guibai
← Back to the summary

Jev: The Decision Model That Made 'Judgment' a Function Call

Jev Teardown: It Doesn't Write, Only Judges. How a "Decision Model" Took Over the Most Expensive Step in LLMs

On September 15, 2026, TypeSafe AI released Jev. Within six to seven weeks, Cloudflare, Microsoft, OpenAI, and Amazon all shipped similar products; on OpenRouter, 1.5 billion calls hit TypeSafe's model in a week; by October, decision models accounted for more than a quarter of Vercel's model routing requests.

But at the same time, Jev ranked 15–18th on the third-party JevBench managed API leaderboard; in a phishing detection test with 2,000 emails, it scored only 62.6%, losing to Claude Haiku 4.5's 81.3%.

These two facts are simultaneously true. This article pieces them together: what Jev actually is, where the speed and cheapness come from, what competitors exist now, which scenarios call for it, and how to get a working version running in half a day.

fig1-two-paths.png

Conclusion first:


1. What Jev Actually Does

In one sentence: extract "judgment" from "generation" and make it a single function call.

The conventional approach is to have a chat model output {"category": "billing", "confidence": 0.8} and then extract the field with regex or a JSON parser. This pipeline has three unavoidable problems:

  1. Output is serial. The model writes token by token, while classification does not inherently need a sequence.
  2. That 0.8 is text, not a probability. It comes from the same next-token prediction as the preceding words; it has no obligation to equal real hit rate.
  3. You only get the selected option, not the full distribution, so you cannot implement a tiered strategy like "auto-execute above 0.8, escalate between 0.5–0.8, route to human below".

Jev changes the contract. A single request has only three required fields:

{
  "model": "jev-latest",
  "state": "customer message, ticket, log, contract paragraph, or a JSON object",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this message?",
      "criteria": {
        "billing": "charges, invoices, refunds, payment methods",
        "technical": "bugs, errors, outages, integration issues",
        "account": "login, password change, permissions, account settings"
      }
    }
  }
}

There is no sentence in the response, only an answer and a distribution:

{
  "model": "jev-1.14.0",
  "answers": {
    "team": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.88, "technical": 0.09, "account": 0.03 },
      "confidence": 0.81
    }
  },
  "usage": { "input_tokens": 412, "output_tokens": 3 }
}

Note that usage.output_tokens is 3, not 0—output is still counted, but not billed.

Four primitives cover most "judgments"

Primitive Answers what Configuration Returns
Choice Which one of several options instructions + up to 255 options with descriptions choice + full probabilities + confidence
Score Where on an ordered scale instructions + 2–10 levels score (probability-weighted expectation) + legend + probabilities + confidence
Noul Whether a statement is true instructions + optional true/false descriptions noul (0–1 probability), no separate confidence
Box Where in an image a target is relative coordinates for four sides box: {top, bottom, left, right}, only in 1.14

Two easily overlooked details:

First, questions run in parallel. state is read once, and all questions share the same context. TypeSafe reports that "adding questions barely affects response time." This is an order-of-magnitude difference from "calling the LLM four times serially," and it's why Jev is ideal for "judge multiple dimensions in one call."

Second, confidence is not "the probability this answer is correct." It is a statistic computed from the shape of the distribution. For Choice the formula is:

confidence = (n x max_p - 1) / (n - 1)      n = number of options

With three options and max probability 0.88, (3 x 0.88 - 1) / 2 = 0.82, which roughly matches the doc example of 0.81 (rounding). So 0.81 means "the distribution is concentrated," not "81% likely correct." What actually answers "how likely is this correct" is the entry in probabilities, and even that only holds in a batch sense.

fig2-request-anatomy.png

Three differences from Structured Outputs

LLM + JSON Schema Jev
How output is produced Token-by-token generation, schema only constrains One forward pass, distribution over options directly
Can it violate schema? Yes, long output, truncation, model upgrades can all break it No, answer space is fixed in the request
What you get A single value + model self-reported confidence Full distribution + confidence derived from that distribution

The third row is the divider. The former gives you a number you must trust; the latter gives you a distribution, and your code can decide how much to trust it.

Names and origin

Both names have stories.

Jev is taken from William Stanley Jevons. This 19th-century economist observed that after steam engine efficiency improved and per-unit coal consumption dropped, Britain's total coal consumption actually soared—this is the "Jevons paradox." TypeSafe is betting on it: each order-of-magnitude drop in the unit cost of intelligence leads to several orders of magnitude more usage. So they price it at $0.042 per million tokens, not to steal LLM workloads, but to make traffic that "wasn't worth AI judgment before" now worth it. They admit they cannot prove this price isn't subsidized; only time will tell if it's sustainable.

System One comes from Kahneman's Thinking, Fast and Slow. System 1 is fast intuition, System 2 is slow reasoning. They admit the name is risky—"System 1" in psychology already implies "error-prone"—but their response is that System One models can be made more reliable than alternatives because their output space is closed.

Founder Diogo Almeida worked on RLHF and InstructGPT at OpenAI and is a co-author of RLHF. When the company emerged from stealth on September 15, 2026, DCVC led a $40 million seed round; then, per company announcements, it raised another $870 million at a $7.5 billion valuation. This funding cadence itself shows the heat—a model that only does multiple-choice questions has reached a valuation close to that of foundation-model companies.


2. Speed and Cheapness, Specifically Where

From O(n) to O(1)

Traditional LLM:
state -> [tok1] -> [tok2] -> ... -> [tokN] -> assemble JSON -> json.loads
            |          |                              |
            +----------+-- each step re-reads all context --+
latency proportional to output length; cost proportional to output tokens

Jev:
state -> one forward pass -> q1 choice + probability distribution
                            q2 noul + 0.95
                            q3 score + 1.4
                            q4 choice + probability distribution
latency constant; question count barely affects total time

Generation models aren't just compute-bound; there's also the memory-bandwidth cost of "re-reading the whole context for each token." Jev eliminates that: after reading the state, it produces logits over all options in one shot and normalizes to probabilities. The output length variable disappears from the formula, so latency goes flat.

Calculate costs yourself; don't memorize "444x"

Official pricing: $0.042 per million input tokens, output free. At that price, cost depends only on input length:

cost per call = input tokens * 0.042 / 1,000,000

Compare three typical request shapes with GPT-6 Astra ($10 / $50 per million):

Request shape Input / output tokens Jev per million Astra per million Multiple
Short routing (ticket classification) 300 / 10 $12.6 $3,500 278x
Medium moderation (email verdict) 1,500 / 20 $63 $16,000 254x
Long document judgment (contract clause) 3,700 / 30 $155 $38,500 248x

The third row's magnitude aligns with the third-party leaderboard's $156 per million decisions for Jev 1.14, so the arithmetic isn't off.

But here's a comparison many miss: if the opponent is GPT-6 Luna ($0.10 / $0.50), the same 300-token input costs $35 per million—Jev is only 2.8x cheaper, not 444x.

The official 193.6x / 444.6x figures come from their own workflow eval, with several assumptions: the comparison is against frontier models; the LLM is forced by a wrapper to output structured decisions (slower and costlier); and the reference answer is the average prediction of Astra and Fable 5.1, not human ground truth. TypeSafe itself wrote in the blog post that "these numbers sit at the high end of true savings."

So the right expectation is: against frontier models, two orders of magnitude cheaper; against cheap small models, only a few times cheaper. The former enables "run full judgments you never dared before"; the latter just means "run the same budget a few more times."

The reverse ledger: network latency can eat the advantage

Jev's own claim is 70–500ms. Krishnan et al. at CMU measured a median of 150ms in a preprint, versus GPT-6 Astra's 1.89s—that's a 12.6x difference, not 193x.

Worse is the network. Third-party leaderboard JevBench records Jev 1.13's median latency at 1.23s, which is eight times the model-side 150ms. The gap likely isn't in the model but in gateway, retries, and queueing. If your agent does one call per step, 20 steps serially, the 1.7s saved by the model gets repaid elsewhere.


3. RLCD: Jev's Real Differentiation, and the Hardest Part to Verify

TypeSafe founder Diogo Almeida was one of the early RLHF / InstructGPT researchers at OpenAI. His startup thesis is the opposite of treating RLHF as a universal cure: RLHF optimizes "what humans like to read," not "what software can trust."

The official docs lay out three post-training routes:

Route Where the reward signal comes from What it optimizes Typical side effects
RLHF human preference rankings making answers "read pleasingly" sycophancy, overconfidence, mode dropping
RLVR results that can be automatically verified (correctness, code passes) correctness and reasoning strong reasoning but slow, expensive; probabilities not necessarily trustworthy
RLCD alignment between decisions and predicted probabilities when it says 0.8, long-term accuracy is actually 80% needs lots of decision data with outcome labels; drifts across distributions

RLCD stands for Reinforcement Learning for Calibrated Decisions. Its objective isn't to reward "correctness" but to reward "probability honesty"—a 0.9 prediction that's wrong is penalized more than a 0.6 one. That's why Jev's output can be used directly for thresholding: the probability is a product feature, not a byproduct of ranking.

Calibration is a batch property, not a per-call guarantee

The official documentation is explicit: calibration is "a property computed across a batch of predictions."

This decides the right usage: treat probabilities as routing signals, not guarantees. Using thresholds to split traffic, distributions to rank, and low confidence to trigger escalation are things calibration supports; "the model said 0.95 so it must be right" is not.

Calibration can break, and it breaks in concrete ways

This is the section most worth remembering. Three independent measurements:

1. Systematic overconfidence on relevance judgments. A paper replicating BEIR ranking experiments aligned Jev's native probabilities with human relevance labels and found that among document pairs Jev scored above 0.9 on DL21, only 66% were truly relevant. The good news: this is mostly a "scale" problem. Fitting a two-parameter calibration on DL19's 2,257 document pairs removes most of the error and transfers to DL21; after calibration, Jev's Brier score is lower than FLAN-UL2 and three cross-encoders.

2. "Confidence inversion" across datasets. RSAJEV measured this in remote sensing classification: 97.71% accuracy in-distribution, but 7.35%–96.40% cross-dataset, and the confidence AUROC is only 0.38—worse than random. That means the model is more confident when it's wrong. This is not unique to Jev, but it reminds you: confidence thresholds must be validated on the target distribution, not borrowed from benchmarks.

3. Abstention can be induced by the options. In a list aggregating 53 independent studies, the KoBBQ calibration audit found that when an "unknown" option is present, Jev chooses to abstain on 95% of 300 ambiguous samples; remove "unknown," and accuracy drops to zero, with 79% of answers falling into dataset stereotypes. Adding an abstention option is usually good, but it can amplify "uncertainty" into a default action.

So "zero hallucination" should be taken apart: not fabricating things outside the options is type safety; choosing wrong with high confidence, or abstaining in the wrong places, is an accuracy problem. The two don't guarantee each other.


4. The Current JEV Family: Two Official Generations, Plus a Whole Category

First, clear up a naming confusion: Jev is TypeSafe's model name; "Jev-style" has become the community's umbrella term for decision models. Papers carrying Jev names include Inner Jev, Chinese-Jev, PixelJev, OpenJev-RLCD, RSAJEV, AnyJev—at least seven, most not official.

Official: 1.13 and 1.14

Jev 1.13 Jev 1.14
Release 2026-09-15 2026-10-14
Model ID jev-1.13.0 jev-1.14.0 (jev-latest points to it)
Price $0.042 / M input, output free same
Context 64k per request / 32k per question same; with TypeSafe-Danger-Zone-Long-Context: true header up to 262k
Input text, image text, image
Primitives Choice / Score / Noul adds Box (image localization)
JevBench Capability 77.1 (Intelligence 63.6 / Calibration 90.6) 75.0 (59.9 / 90.1)
AA Decision Index v2.1 — 66.2, $156/M, 92ms, ECE 0.054

1.14 deserves a separate note: it adds the Box primitive (drawing a region in an image), but overall JevBench capability dropped from 77.1 to 75.0, Intelligence from 63.6 to 59.9, calibration roughly flat (90.6 → 90.1). This is a classic "feature expansion buys quality regression." If you're text-only and have already tuned thresholds on 1.13, there's no reason to follow jev-latest drift—the official docs also recommend pinning the version once you've tuned.

Competitors: one table to see who sits where

The table below uniformly uses the Artificial Analysis Decision Index v2.1 (2026-11-14 snapshot), because it gives index, cost per million, per-call latency, and ECE all in one place. Note: the index is not accuracy; it's a weighted score across several decision benchmarks.

Model Format $/M calls Latency Index ECE
Drex Max v1.1 hosted 103 283ms 72.6 0.038
SubQ Snap 1.2 hosted 137 162ms 72.6 0.046
GLiDE v1.2 (Fastino) self-hostable 780 284ms 72.5 0.057
mammoth-1 27B open weights 264 106ms 71.4 0.034
d1-ultra (Liquid AI) hosted 133 301ms 70.6 0.021
pplx-decider v1.3 hosted 68 289ms 70.4 0.064
Microsoft-Decision-1.1 hosted 63 253ms 66.5 0.059
Jev 1.14 hosted 156 92ms 66.2 0.054
Gemini Decision 1 hosted 345 211ms 66.0 0.013
Microsoft-Decision-1 hosted 143 274ms 62.1 0.124
Celeris-1.1 hosted 14 61ms 62.0 0.104
Clef 27B (Cloudflare) open weights 863 354ms 60.3 0.071
OpenAI Decisions (GPT-6 Luna) hosted 339 206ms 60.2 0.110

fig3-landscape.png

Reading the table three times gives three conclusions:

First, Jev's edge is no longer "most accurate," but "one of the most accurate among the fastest." 92ms is the second-fastest in the table; Index 66.2 is mid-pack. It's now selling latency + ecosystem + interface familiarity, not absolute quality.

Second, the price war has already bottomed out. Celeris at $14 per million is 1/11th of Jev; pplx-decider is more than half the price with an Index 4 points higher. "Decision models = cheap" only holds when compared to frontier LLMs; within the decision-model category, Jev is mid-to-high priced.

Third, open weights are already competitive. mammoth-1 (27B, open) has Index 71.4 and ECE 0.034, beating Jev on both accuracy and calibration. Clef is Apache 2.0 but hosted at $863 per million, clearly a "weights for you, inference via me" play; self-hosting is where its value lies.

Another line: reproducing Jev

If you just look at JevBench (the third-party leaderboard maintained by Benchmark Heaven, v1.8.x), the picture is even sharper. Note that its Capability is the mean of Intelligence and Calibration, not directly comparable to the AA index above:

And the paper Releasing the Inner Jev (arXiv 2610.17205) does something even more audacious: it maps open instruction models' options to A/B/C letters and reads probabilities directly from the last token's logits, zero-shot. Gemma 4 31B scores 79.6 on OpenDecisions, while Jev scores 80.1; adding a simple LoRA, Jeff-1.0-Large reaches 84.4, overtaking.

The paper's conclusion is not "Jev is useless." It's that Jev's moat isn't the "single forward pass reads probabilities" mechanism; it's engineering quality—calibration data, parallel sampling, stability, and making the interface a product. The mechanism can be replicated; the data and SLOs are hard.

Incidentally, the community has split into several branches: Chinese-Jev (10 million Chinese decision samples, +1.24% on general tasks, +4.0% on medical, 20.3x speedup), PixelJev (native image decisions), OpenJev-RLCD (applying RLCD to reasoning models, Qwen3-1.7B matching or exceeding SFT/GRPO), SII-Decision (Fudan, open weights, with a decision benchmark suite), and kettei (Sqwish, $3M pre-seed). The ecosystem has clearly moved past "one model."


5. Which Scenarios Actually Fit

There are only four criteria: closed-set answers, high call frequency, can fall back on error, judgment mainly relies on reading the input. Only when all four hold do you get to model selection.

Scenario Typical question Primitive Why it fits Main risk
Content / email moderation is phishing, abuse, spam Noul + Choice high volume, closed set, must split by threshold adversarial samples; Jev doesn't default to treating state as hostile
Ticket / intent routing which queue, which team Choice options enumerable, wrong can go to human option-order bias; long-tail intent has no matching option
Injection / jailbreak guardrail does this attempt to override system instructions Noul runs on every request, latency-sensitive can't be the only defense; must combine with rules
RAG relevance filtering does this passage support that claim Noul / Score score dozens of candidates in parallel at tiny cost overconfidence, needs recalibration on target distribution
Agent tool selection which tool to call Choice tool set fixed, args filled in separately 255 limit; real errors live in the "answer to action" mapping
Rubric evaluation does the answer meet criterion k Noul / Score multiple criteria in parallel, cheaper than an LLM judge wavers when criteria are mutually exclusive or conflicting

Some real deployments: Cloudflare uses it to flag spam PRs in GitHub docs repos, identify phishing emails and domains, detect prompt injection, and judge whether backbone outage emails affect its services. Microsoft says Copilot's request routing is 2.2x faster and 4.4x cheaper at the same accuracy. TypeSafe uses it for real-time Doom decision-making at 10 decisions/second, roughly $7/hour.

Conversely, these five types should NOT use it

1. Tasks requiring arithmetic, counting, or date comparison. The official model-jaggedness doc lists this: Jev is not a calculator, miscounts, and treats dates as text. The right pattern is "model extracts, code computes."

2. Multi-hop reasoning. Double negatives, attributes of attributes, conclusions requiring several steps: accuracy drops sharply. The official advice is to remove indirection layers or split into atomic questions composed in code.

3. Tasks needing domain expertise. Fudan's Wei Zhongyu team's SII-Decision paper found that decision models slightly beat LLMs on document understanding and match on tool selection, but lag behind LLMs on tasks requiring medical, financial, or legal expertise. Reasoning ability is one factor but doesn't explain the whole gap.

4. Using it as a trading signal. Public tests: BTC/USD backtest holdout AUC of only 0.471–0.503, equivalent to random; an NQ order-book model had 67.8% hit rate over 339 decisions, but after fees it lost $62.69. Judgment accuracy doesn't equal decision profitability—odds, costs, and position sizing sit in between.

5. Tasks needing generation, explanation, or a single final decision-maker. It doesn't write; nor should it sign alone on irreversible, high-stakes decisions.

One more restriction that's easy to miss: Jev does not support fine-tuning with your data. Officially "not fine-tuned or LoRA-adapted with customer data," all accounts share the same weights. What you can change is limited to state, instructions, and criteria. That's the opposite of Sqwish's "continuous post-training on production data" pitch—the upside is your dirty data can't skew it, the downside is your domain jargon has to be forced through prompt text.


6. Getting Started: Five Steps from Signup to Production

fig4-rollout.png

Step 1: Get a working request

Sign up at console.typesafe.ai (early access includes $5 credit), get a key, then:

curl -X POST https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev-1.14.0",
    "state": "My order arrived broken and I want a refund.",
    "questions": {
      "wants_refund": {
        "type": "noul",
        "instructions": "Is this customer asking for a refund?",
        "criteria": {
          "true": "explicitly asking for money back, refund, return of payment",
          "false": "just complaining, no refund request"
        }
      }
    }
  }'

Python SDK:

pip install typesafe-sdk
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

TEAMS = {
    "billing": "charges, invoices, refunds, payment methods",
    "technical": "bugs, errors, outages, integration issues",
    "account": "login, password change, permissions, account settings",
}

with TypeSafeClient(model="jev-1.14.0") as client:
    r = client.system_one(
        state=ticket_text,
        questions={
            "team": Choice(
                instructions="Which team should handle this message?",
                criteria=TEAMS,
            ),
            "urgency": Score(
                instructions="How urgent is the customer's emotion?",
                criteria=["calm", "unhappy", "angry", "about to churn"],
            ),
            "wants_refund": Noul(
                instructions="Is the customer explicitly requesting a refund?",
            ),
        },
    )

team = r.answers["team"]
if team.confidence < 0.8:
    route_to_human(ticket_text)
else:
    route_to(team.choice)

JS/TS uses @typesafe-ai/sdk; Vercel has @ai-sdk/typesafe-ai, but note it renames noul to boolean and the field from noul to probability—adjust documentation snippets accordingly.

Step 2: Pick the right primitive

Step 3: Rewrite prompts as atomic questions

Jev is a literal reader. It answers the question you write, not the one you have in mind. Three rewriting principles:

  1. One question, one judgment. "Is this email urgent and needs a refund" must be split into two.
  2. Write boundaries into criteria. The sentence you'd later say as "I actually meant..." is the half-missing instruction.
  3. Don't let option descriptions fight the instruction. Reversed mappings like true->"no" degrade performance noticeably.

Step 4: Evaluate with 300 real traffic samples

Don't use benchmark scores for go-live decisions. At minimum do four things:

# 1) accuracy + 2) ECE (expected calibration error)
def ece(pairs, bins=10):
    total, err = len(pairs), 0.0
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        bucket = [(c, y) for c, y in pairs
                  if lo <= c < hi or (b == bins - 1 and c == 1.0)]
        if not bucket:
            continue
        conf = sum(c for c, _ in bucket) / len(bucket)
        acc = sum(y for _, y in bucket) / len(bucket)
        err += len(bucket) / total * abs(conf - acc)
    return err

# 3) Flip rate: shuffle option order and see if answers change
# 4) Coverage: how much traffic can auto-execute at your target accuracy

Each metric serves a purpose: accuracy decides whether it works, ECE decides whether thresholds are trustworthy, flip rate decides whether option order can hurt you (official acknowledges Jev biases toward the first option), and coverage decides whether the system actually saves money.

A rule of thumb: 300–500 labeled examples covering the long tail expose more than 5,000 random samples.

Step 5: Dual thresholds + escalation, not step-by-step

Full traffic
   |
   +-- confidence >= high threshold -> auto-execute (log, leave trace)
   |
   +-- low threshold <= c < high threshold -> escalate to LLM review / one more generation
   |
   +-- c < low threshold -> human / rule fallback

The CMU preprint tested this combination: Jev hands low-confidence samples to GPT-6. Total accuracy is slightly higher than using GPT-6 alone, at only 41% of the cost. This is the steadiest engineering pattern—a cheap model does the triage, and an expensive model only handles the uncertain cases.

Three details for rollout:

Cost estimation template

Monthly cost = monthly requests * average input tokens * 0.042 / 1,000,000

100k requests/month at 800 tokens average: 100,000 * 800 * 0.042 / 1e6 = $3.36/month.

The same volume via GPT-6 Astra (800 in / 20 out): 100,000 * (800*10 + 20*50) / 1e6 = $900/month.

That's the line between "dare to run full judgments" and not. It's not that Jev is smarter; it's so cheap you can afford to ask every single record.


7. My Verdict

First, Jev defined the interface but hasn't held the throne. It turned the "state + typed questions + parallel probabilities" contract into the de facto standard, and Microsoft, Cloudflare, and OpenAI all conform to that shape. But on leaderboards, a pile of models are more accurate, cheaper, or better calibrated; 1.14 even regressed from 1.13. Treating it as the "standard interface for a class of models" is more accurate than treating it as the "strongest model."

Second, the cost-saving narrative splits into two layers. Against frontier LLMs, it's two orders of magnitude cheaper—true, and it unlocks "never dared to run full-volume before." Against small models like Luna, it's only a few times cheaper, so the reason to choose it then is latency and type safety, not money.

Third, the real risk has shifted from "parse failure" to "high-confidence wrong judgment." The former raises exceptions you can catch; the latter is silent. So those 300 labels, the ECE, the option-reordering test before launch are not optional—they're part of the architecture, not accessories.

If you remember one sentence: Jev turned the most expensive step of an LLM into a function call, and the price is that you assume full responsibility for whether the judgment is correct.


Data Sources and Notes

Caveat: All "cost per million calls" in this article are model-side costs from input-token price; they exclude gateway, retries, and human review. Different leaderboards measure latency at different points (model-side vs end-to-end). When comparing across boards, compare order of magnitude, not decimals. Vendor-reported numbers are marked with sources; independent replication results are called out.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

米小虾 1 likes

Jev model ecosystem: https://github.com/supermanc417-alt/awesome-jev-hub, containing 58 application projects · 12 directions · 11 community lists...