Embedding Swaps Should Be Priced Per Successful Task, Not Per Vector Query
A model that wins a retrieval leaderboard can still make a knowledge-base Agent slower and more expensive through retries, ignored evidence, or expanded context. Counting index rebuilds and failed attempts per successful task is the only comparison that tells you whether a swap actually saves money in production.
Public retrieval leaderboards and per-vector price lists don’t predict what an embedding swap does to an Agent’s total bill. The cost appears downstream: whether the Agent finds enough evidence in the first retrieval, rewrites a query, expands a document, or retries a tool after failure. Each of those actions spends generation tokens, and a cheap vector call can be the most expensive choice once the full task is counted.
The evaluation separates retrieval quality, Agent behavior, and task-level cost. Retrieval metrics like Recall, NDCG and MRR tell whether evidence enters the candidate set and ranks correctly; Agent logs record why a second search happened, such as correcting a wrong document versus gathering a second hop of evidence; task accounting divides all costs, including failed attempts and failure handling, by tasks that pass acceptance. Index rebuild amortization belongs in the same ledger. The included Python 3 script runs nine synthetic records and asserts the bookkeeping: a candidate with lower online spend (1.0480 vs 1.0631) becomes more expensive per success (0.509333 vs 0.354367) once a 12-dollar rebuild is amortized over one hundred tasks and then spread across four.
For a concrete comparison, the candidate set includes Nemotron 3 Embed 1B BF16 and 8B BF16, with vendor-reported RTEB numbers treated as historical, not as proof of business results. Hardware constraints matter too: 8B BF16 weights are roughly 16GB before activations and caches, and NVFP4 throughput claims for Blackwell can’t be ported to an RTX A6000. Fair evaluation freezes task sets, corpus snapshots, chunking, filters, generation configs, and Agent prompts, then varies only the embedding model and its required input adaptation.
The 78.5% NDCG@10 figure is dated 2026-07-15 and comes from a vendor announcement; treating it as current would be a category error, and the article’s refusal to re-run the model keeps the claim honest.
The ledger’s inversion is the useful part: a model with lower online cost per task can still be the worse deal once index rebuild is amortized over a small task count, which is exactly the situation a single vector-price comparison misses.
Requiring a stable task_id and variant for every log row is the practical precondition for all of this; without it, retries, failures, and acceptance results cannot be attributed to a specific model.
Layered failure classification prevents a common false inference: if retrieval-empty, vector timeout, generation throttling, tool error, and acceptance failure are all stored as one failure bucket, an embedding swap can accidentally take credit for a drop in generation throttling.
Including indexing cost in the per-successful-task metric is solid. Adding a common pitfall from practice: when the corpus updates frequently, the retrieval set itself drifts. Using a fixed batch of old queries for regression comparison makes it easier to pinpoint whether degradation occurs at the indexing layer or the model layer than rebuilding the eval set each time.