Introduction

Jev has been quite popular lately. At first I thought it was just another new model, but after actually using it I realized it's different! It's a tool that lets you engineer a lightweight decision layer into AI workflows, cutting cost and improving efficiency.

My workflow comes from the test domain, where many tasks only need a clear classification or selection, such as labeling test cases, choosing the next action in UI test execution, or filtering retrieval results. These tasks can be done by a general large language model, but when executed frequently, response time, calling cost, and output processing all affect overall efficiency.

So I picked three everyday testing scenarios and used them on the AI Testing platform to try Jev's integration and practical landing. The results exceeded expectations!

This post records my process, runtime data, and implementation examples. It also lists related platform links, skill packages, reference frameworks, and scripts for anyone who wants to explore further.

2. Get to know Jev quickly

The difference between Jev and a general LLM

To intuitively show the output difference, I'll use a simple question: Is a bat a bird? Here are the answers from a general LLM and from Jev:

General LLM: output format is uncontrolled by default

Example answer: a generative model improvises and expands its explanation

Jev: returns a structured judgment

Example answer: fixed standard structure, crisp and decisive

From these two images, the first thing I noticed is the difference in output form: the general LLM gives a conclusion and then continues explaining its classification basis; Jev returns a judgment and a score based on a predefined "belongs / does not belong" set. This comparison only illustrates the output form, not a performance or accuracy evaluation under the same conditions. I also put input, output, latency, and price into a table for further comparison:

What is Jev?

From this comparison I got a more concrete understanding of Jev's positioning: for selection and judgment tasks with a clear answer space, it returns structured results with low latency and low cost.

To use it, I provide the content to be judged, the question, and candidate answers; Jev returns a result based on semantics. Predefining the answer space also lets downstream scripts directly process the output and continue the flow.

I want to distinguish model score from actual accuracy. "Confidence 100%" is just the score for that output; it doesn't mean the model is necessarily correct, nor is it a calibrated real-world accuracy. Whether to auto-adopt a result depends on task risk, manually labeled samples, and independent verification.

Where Jev fits

In requirement analysis, design, and coding, I usually need a general LLM for reasoning and generation. But in some workflow steps the requirement is much simpler: classify a case, judge content relevance, or pick the next action from candidates. A general LLM can do these too, and it can be constrained by prompts and structured output; however, when call frequency is high, the overhead of generation and result parsing still deserves separate evaluation.

For these steps, I care about two things: whether the model understands context correctly, and whether it can stably return results within a defined answer space. Directly usable output also helps reduce downstream parsing.

With that in mind, I scoped Jev to: handling small, well-bounded judgment tasks in scripts, and driving subsequent actions based on the results. Whether this split works needs validation on real tasks. Next I focused on integration latency, cost, and result handling.

After Jev's release, posts showing it off on X multiplied so much that someone built a community practice aggregation page. As of September 23, it had collected 5,721 works from 113,386 posts by 4,424 authors.

Combining these practices, I've compiled several directions close to everyday business:

• Intent recognition and agent dispatch

• Hooks (security review, hallucination judging, completion governance)

• Bulk data sorting, labeling, filtering

• Quick decisions

• Category ranking

• Trend prediction

• Code review

This time I chose three scenarios: test-case labeling, UI automation, and retrieval filtering, all validated on the AI Testing platform. Each scenario's implementation and results follow.

3. Scenario 1: Test-case labeling - fast batch execution

Scenario and approach

I first chose test-case labeling. After test cases are generated, they usually need priority labels plus a flag for potential financial-loss impact, so that later execution scope can be filtered by label. This kind of judgment has a fairly clear boundary and works well for observing batch efficiency.

This time I used a Python script to concurrently process 300 test cases, labeling priority as P1, P2, P3, and adding the "financial-loss" tag where relevant.

Then, through Python I read the cases, concurrently called the Jev API, and finished labeling. The script can also be written with AI help: after pulling the ep-aitesting-jev skill from the AI Testing platform's skill market, the AI can follow the skill description to complete the interface integration:

After running the script, you can see Jev processing cases concurrently:

Results

This run generated a labeling result file within seconds:

The result includes a "suspected financial loss - to be reviewed" label, which appears after my adoption threshold in the script. Judgments below the threshold are kept for manual review; Jev's probability and confidence are used as one routing signal.

To assess the benefit of this integration, I compared two approaches processing the same 300 cases in terms of time and cost:

This time the DeepSeek approach took 4 minutes 5 seconds, while the Jev approach took 5 seconds; the former is about 49x slower. Cost was roughly 0.31 yuan vs 0.01 yuan, so the latter is about 31x cheaper. These numbers show that for this labeling task, Jev reduced overall time and call cost. The cost figures are rounded, so ratios are only indicative; the comparison covers the full script run of both approaches.

One difference in request style: the DeepSeek approach merged multiple cases into one request, totaling 42 requests; the Jev approach sent one request per case, 300 requests. Jev also supports multiple judgment items in a single request: put cases and IDs into state, and define corresponding questions in questions. Whether to merge requests I decide based on input length, concurrency limits, and measured results. This comparison should be understood in light of each approach's request style.

4. Scenario 2: UI automation - faster UI case execution

Scenario and approach

The second scenario is interactive UI automation. When using tools like Playwright, the model needs to locate target elements based on the current page and the test steps, then let the browser tool execute the action. Compared to fixed scripts that change with page layout, interactive approaches try to choose the next action based on the current page state.

What I care about is the accumulated overhead of step-by-step judgment: as operation steps grow, each step's model response affects total latency. So I tried having the browser tool first collect visible controls and necessary text, then let Jev choose an operation target from a limited candidate set based on the test steps.

Implementation

For this, I referenced the public npm package easy-jev-e2e. It uses text-based test cases and Jev to select the next page element to operate. The developer mainly maintains text steps, and the framework handles actual execution.

Following that idea, I selected a typical scenario on the AI Testing platform: enter account and password, verify the site login result.

From the middle results, you can see Jev choosing the next action based on the text test step and current page state, then the browser tool executing it:

As a control, I handed the same login case to another sub-agent using a general LLM for traditional interactive UI testing:

Results

I summarized the time and cost of both approaches:

This time the general LLM approach took 90 seconds, while Jev took 7 seconds, so the former is about 13x slower; cost was roughly 0.93 yuan vs 0.01 yuan, making Jev about 93x cheaper. For tasks where you choose an operation target step by step, these results provide a basis for further validation. Cost figures are rounded so ratios are approximate; the comparison covers the full UI test runs.

5. Scenario 3: Knowledge base recall - efficient reranking via vector search

Scenario and approach

The third scenario is retrieval result filtering. When an Agent analyzes requirements or supplements test scenarios, I usually first retrieve historical requirements and baseline test cases as analysis references.

This time my query was the "XXX business". The goal was to find cases that actually verify XXX-business-related receipt/payment, payment channels, refunds, reconciliation, compliance blocking, or channel switching. So what I care about is the business chain the case verifies, not whether the text contains keywords like "XXX business", "bank", or "payment".

Vector search can recall semantically similar candidates, but those candidates aren't always suitable for the current business goal. To illustrate what I wanted to filter, here are two anonymized examples (they don't represent actual recalled data from this run):

• Bind a supported domestic bank card, complete shopping mall RMB payment, verify payment success and order generation. This case contains both "bank" and "payment", but it verifies domestic payment acquiring, involves neither the target business's bank nor the cross-border chain, and is far from the target business.

• Open the target business bank's exchange rate display page, switch currency, verify the rate and bank name display correctly. This case even contains "target business bank", but it actually verifies page display, not cross-border payment transactions; business name and institution name match, yet the case still can't prove business relevance.

If these candidates go directly into an Agent, irrelevant payment flows and display checks also pollute the context. I wanted to reduce this noise at this stage, avoid later analysis referencing unsuitable material, and make genuinely relevant business constraints more likely to be noticed.

So after vector recall I added a judgment step: let Jev evaluate business relevance and content quality, then let the script filter and route. Concretely, use noul to judge whether the verification target is directly relevant, choice to distinguish business topic, and score to assess whether scenario, operation target, and expected result are complete. This lets me handle two kinds of failures: business mismatch and content needing completion.

I split the final output into three buckets: cases that match the business and meet quality requirements go to the Agent; irrelevant candidates are rejected; relevant but incomplete candidates are kept separately for completion.

Implementation

On the AI Testing platform, I chained this process in a Python script: call the knowledge base API to recall candidates, call Jev to judge, then return filtered results to the Agent.

In this chain, I placed Jev after vector recall, responsible for candidate reranking and quality filtering.

Actual chain: vector Top-10 recall → Jev three-dimensional judgment → script filter and route

This knowledge base held 16 baseline test cases. I queried "XXX business", ran vector Top-10 recall, then used a Python script to request the local Jev gateway one by one, judging whether each candidate's verification target belongs to the target business.

In each request, I put one case into state and defined three judgment items in questions, all sharing the same case content:

• relevance/noul: judge "does the verification target directly belong to the XXX business?", returning the probability the proposition holds. Judgment is based on core chain, channel, and transaction scenario, not just keywords.

• topic/choice: choose the most accurate from four mutually exclusive topics: XXX business, other cross-border payment, domestic payment acquiring, non-payment domain; return probability distribution and confidence for candidates, distinguishing business boundaries.

• quality/score: using an ordered scale "missing or vague elements / basically complete but insufficient detail / three elements complete and directly executable", judge content quality around the three elements of scenario, operation target, and expected result.

These three dimensions answer "is it relevant?", "which business category?", "can it be used directly?". In the first image below, relevance is 98%, the target topic probability is 100%, but quality's "basically complete but insufficient detail" bucket is 84%. This also shows the need for separate quality assessment: business relevance doesn't mean the case content is fully complete.

Python example: directly call Jev API for three-dimensional judgment

This is the Python example for the three-dimensional judgment. The example starts from an existing vector retrieval result file, doesn't implement vector recall; the filter thresholds and sorting are only for demonstration. The script reads noul, choice, score from data.answers. Before running, replace JEV_URL with the actual address and configure AUTH_TOKEN; if connecting directly within an AI Testing session, you can switch to reading the session's ANTHROPIC_AUTH_TOKEN as per the integration docs.

import json
import os
from urllib.request import Request, urlopen

JEV_URL = "..."
QUESTIONS = {
    "relevance": {
        "type": "noul",
        "instructions": (
            "Does the case's verification target directly belong to the XXX business? "
            "It counts as relevant only if the core chain, channel, or transaction scenario involves "
            "the XXX business's receipt/payment, cross-border channels (including agent banks, "
            "business channels, refunds, reconciliation, compliance blocking, channel switching); "
            "merely containing XXX business, bank, payment keywords but verifying other business "
            "does not count."
        ),
    },
    "topic": {
        "type": "choice",
        "instructions": "Which business topic does the case most accurately belong to?",
        "criteria": {
            "target_business": "XXX business (channel, receipt/payment, refund, reconciliation, compliance, switching)",
            "other_crossborder": "Other cross-border payment (other channels not the target business)",
            "domestic_payment": "Domestic payment acquiring",
            "non_payment": "Non-payment domain (rate display, account opening, logistics, login, marketing, etc.)",
        },
    },
    "quality": {
        "type": "score",
        "instructions": "As a baseline test case, are the three elements - scenario, operation target, and expected result - complete and clear?",
        "criteria": ["Elements missing or vaguely stated", "Basically complete but insufficient detail", "Three elements complete and directly executable"],
    },
}


def judge_case(case):
    payload = {"state": json.dumps(case, ensure_ascii=False), "questions": QUESTIONS}
    request = Request(
        JEV_URL,
        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ["AUTH_TOKEN"],
        },
        method="POST",
    )
    with urlopen(request, timeout=30) as response:
        body = json.load(response)
    # Business errors may also return HTTP 200; must check the envelope code.
    if body.get("code") != 200:
        raise RuntimeError(f"Jev call failed: {body.get('code')}{body.get('message')}")
    return body["data"]


def refine(candidates):
    accepted, review, rejected = [], [], []
    total_cost = 0.0
    for case in candidates:
        data = judge_case(case)  # one request per candidate, three judgments at once.
        answers = data["answers"]
        relevance = answers["relevance"]["noul"]
        topic = answers["topic"]["choice"]
        quality = answers["quality"]["score"]
        row = {"case": case, "answers": answers, "trace_id": data.get("u_trace_id")}
        total_cost += data.get("usage", {}).get("cost", 0)
        # The thresholds and sorting below are only examples, not the actual config in the screenshot.
        if relevance < 0.8 or topic != "target_business":
            rejected.append(row)
        elif quality < 2:
            review.append(row)
        else:
            accepted.append(row)
    accepted.sort(key=lambda x: x["answers"]["relevance"]["noul"], reverse=True)
    return {"accepted": accepted, "review": review, "rejected": rejected,
            "jev_cost": total_cost}

# Run this in an AI Testing session that has Jev enabled; reuse the session token.
# Input file comes from the existing vector retrieval script; it's a JSON array of Top-10 candidate cases.
if __name__ == "__main__":
    with open("vector_top10.json", encoding="utf-8") as f:
        candidates = json.load(f)
    print(json.dumps(refine(candidates), ensure_ascii=False, indent=2))

For result handling, I kept three sets - adoptable, to-be-completed, rejected - and recorded answers to review probabilities and buckets; if the response carries u_trace_id, it's saved too. The example stops on call errors to avoid mistaking an API failure for an irrelevant case.

The second image shows this round's artifacts: kb_baseline_cases.json stores 16 baseline cases, retrieve_baseline_cases.py executes vector recall and Jev fine-filtering, and result.json stores the run result. Each judgment was actually executed via the local Jev gateway, with trace info attached. This round's Top-10 ended with 6 adopted, 1 to-be-completed, 3 noise rejected, and total Jev cost $0.000336.

Results

After judging the 10 candidates, the final outcome was 6 adopted, 1 kept for completion, 3 noise rejected. The script took about 4 seconds; Jev total cost was $0.000336. For this scenario, what matters more is completing relevance and quality filtering at low cost before data enters the Agent's context.

6. Conclusion

After these three scenarios, I have a clearer picture of the applicability boundary: they all let you define the answer space first, then drive the next step based on the judgment result. When integrating, I first clarify input, options, adoption conditions, and fallback path, then evaluate whether Jev is the right fit for that step.

These practices also helped me understand Jev (TypeSafe System One) as a dedicated structured decision model: it doesn't generate natural language; it returns judgment results with probabilities.

For future integrations, I'll keep starting from well-bounded judgment tasks in scripts, validating accuracy, latency, and cost separately, then decide whether to expand usage. I've compiled relevant capabilities, limitations, and integration points below.

Three judgment capabilities to focus on:

• Noul: yes/no judgment, returns a 0~1 probability.

• Choice: picks one from a predefined set (up to 255 options), with each option's probability.

• Score: rates on a customized ordered scale, returns a score and bucket meanings.

Features to consider when evaluating adoption:

• Low latency: end-to-end 70-500ms, suitable for real-time routing and interception.

• Low cost: input 0.3 yuan / million tokens, output free.

• Strongly structured output: directly returns machine-readable results, skipping LLM text output parsing and JSON repair steps.

• Built-in probability confidence: configurable thresholds; low confidence automatically degrades to manual / strong LLM.

• Suitable for high-frequency trivial judgments: ticket routing, content labeling, LLM output security checks, agent action routing, risk grading.

Limitations to consider in selection: cannot do open-ended generation, long-horizon reasoning, or complex planning; has no memory; complex multi-step tasks can't rely on Jev alone. Quality depends on a clear option/score definition; the prompt must spell out each option's semantics, and vague definitions visibly degrade accuracy. It only fits scenarios where the answer space is pre-fixed, not open-ended questions where the answer is unknown.

Division of responsibility in agent flows: in an Agent flow, leave analysis, planning, and tool calls to a general LLM, and hand well-bounded routing branches, result validation, risk scoring, and input filtering to Jev. Results that don't meet the adoption threshold go to the main model or manual review.

Points to check when integrating: pass the content to be judged as state, and define both the judgment question and the option descriptions. Option descriptions should be clearly distinct to avoid ambiguity. Set adoption, review, and fallback conditions based on the returned probability; don't blindly trust all results. Suitable for batch, high-frequency, simple decisions; leave complex reasoning to a general LLM.

My advice is to start with a high-frequency, clearly bounded step. Define the answer space and judgment criteria, test accuracy and failure paths on a batch of samples, and record latency and cost. After confirming benefits and a review mechanism, gradually expand scope. That's the approach I used in these three runs.

Previous posts

1. Domestic compute · Ascend 910C LLM inference optimization (Part 2): source modification and operator tuning, 30% faster | Dewu Tech

2. Domestic compute · Ascend 910C LLM inference optimization (Part 1): image update and parameter tuning, 40% faster | Dewu Tech

3. Agent practice with dynamic rules: using deterministic engineering to tame LLM asset acceptance

4. Send an image and it "jumps" out of the screen? A quick read of Dewu's 3D spatial image technology

5. Stop only competing on vector retrieval - how Dewu transaction search uses "generative" recall for a paradigm leap

Author: Yuansi

Follow Dewu Tech for weekly technical dry goods every Thursday.

If you find this post helpful, feel free to comment, share, and like.

Unauthorized reproduction without Dewu Tech's permission is prohibited; violators will be pursued legally.