How to Evaluate an Embedding Replacement: Count Index Cost into Successful Tasks

A retrieval model with a higher score on a public leaderboard can still make the knowledge base more expensive after integration. The reason often lies after retrieval: whether the Agent can answer with the segment it got, whether it continues rewriting the query, whether it expands the entire document, and whether it calls a tool again after failure. These actions consume generation-model tokens and time. Looking only at the unit price of one vector request cannot observe the changes across the whole chain.

This article uses the public information about Nemotron 3 Embed as a candidate entry point, delivers a paired-measurement method with fixed retrieval conditions, and a bounded log-accounting program that can be copied and run directly. The program processes manually constructed task records to verify the ledger’s accounting; A and B in the text are hypothetical scheme labels, not the win or loss of any two real models. This round did not load Embedding weights, run a GPU, or obtain a real vendor bill.

Specifications and Confirmed Information

As of this verification, NVIDIA’s official 1B BF16 model card lists about 1.14B parameters, 2048-dimensional output, and a maximum sequence length of 32768 tokens; the 8B BF16 model card lists about 8B parameters and 4096-dimensional output. The model cards state evaluation on 34 languages, and the license is marked OpenMDW-1.1. Language coverage states which public tasks were evaluated and cannot be read as equal quality in every language or industry.

These numbers affect service cost, but they are not proof of business benefit. Vector dimensionality determines the storage volume of raw vectors, the sequence limit affects truncation strategy, and parameter count affects weight and runtime burden. A business question that can be answered in eight hundred characters of a document will not automatically improve because the model supports longer input; encoding irrelevant attachments along with it can also increase computation and recall noise.

The official BF16 usage notes distinguish query and passage prefixes, and require re-running L2 normalization on the sliced vectors. In comparison, each candidate should keep the input adaptation that fits its own model card; fix the adaptation rules, then fix the business content. Making all models mechanically use the same wrong prefix turns an interface mismatch into a model quality difference. Changing output dimension is also an intervention and needs a separate configuration; it must not be silently changed while keeping the same model name.

In the 1B card read this round, the local BF16 example lists dependencies including Transformers 5.2.0 and above, Sentence Transformers 5.4.1 and above; the vLLM recommendation includes 0.25.0. These are version boundaries in the official example; those deployment steps were not executed on this machine. In actual runs, model revision, tokenizer, normalization implementation, and backend image digest should be locked; a floating model name is not a complete experiment version.

Three Candidates and Hardware Boundaries

The following specifications were attributed to official announcements and model cards checked on 2026-10-15; these are public specs, and no model was loaded on this machine.

Candidate	Parameter scale / output dimension	Max sequence	Applicability boundary
8B BF16	about 8B / 4096	32768	Quality candidate; actual latency and memory still to be tested
1B BF16	about 1.14B / 2048	32768	Low-cost candidate; not guaranteed to be better for this business
1B NVFP4	about 1.14B / 2048	32768	Native NVFP4 deployment targets Blackwell; cannot extrapolate A6000 throughput

NVIDIA’s 2026-07-16 announcement article states that its Figure 1 comes from the 2026-07-15 RTEB screenshot and reports that 8B BF16 ranked first at the time with an average NDCG@10 of 78.5%, and 1B BF16 at 72.4%. This is the historical result caliber from the vendor’s publication, not a re-run leaderboard for this round, and does not represent the current ranking or business task success rate. This article does not derive actual savings from these numbers. NVFP4 specifications and the vLLM roadmap are in the official model card.

The NVIDIA RTX A6000 specification is Ampere, 48GB GDDR6 ECC. Using only the estimate of 8x10^9 parameters and two bytes per BF16 parameter, 8B weights are about 16x10^9 bytes, i.e. 16GB in decimal; this is not full process memory. Activations, attention, batch, framework cache, and concurrency increase the peak, and 48GB does not automatically guarantee long-sequence or concurrent configurations work. On an A6000, BF16, FP16, or a quantization path the framework explicitly supports can be candidate test routes; do not cite Blackwell’s maximum two-times throughput as the A6000 expectation. No GPU measurement was performed this round.

What RTEB Can Answer

RTEB is a retrieval benchmark that can help decide whether a candidate is worth entering business testing. NVIDIA’s announcement and evaluation article reports the public retrieval performance and Agent experiments of Nemotron 3 Embed. The rankings, scores, and throughput in it belong to a specific vendor evaluation caliber; this article does not rewrite a publication-date ranking into a permanent conclusion, nor treat it as interchangeable with its own business task success rate.

The most direct thing a public benchmark measures is ranking: whether material relevant to the question enters the top few results, and whether relevance is correctly ordered. Online knowledge bases also have permission filtering, expired documents, duplicate versions, table parsing errors, and separation between code and explanations. If the page with the answer never entered the corpus, a stronger vector model still cannot recall it. Check corpus coverage before discussing vector quality, so missing data is not attributed to the model.

Business problems can be divided into five categories: fact location, version comparison, code lookup, multi-hop composition, and no-answer. Fact location looks at the position of the first correct segment; version comparison looks at whether two valid versions are retrieved; code lookup also checks whether the relevant function and its calling context appear together; multi-hop problems require checking evidence at each hop; no-answer problems observe whether approximate wording is wrongly treated as an answer. An average retrieval score cannot replace these layered judgments.

No-answer tasks affect cost especially. Without evidence, the Agent may keep rewriting queries until the budget is exhausted, or it may quickly return a wrong answer. The latter looks like fewer tokens and faster speed, but it does not meet the task-quality condition. The cost table must therefore carry acceptance results and refusal rules at the same time. If savings come from giving up early or answering incorrectly more often, that is not acceptable replacement benefit.

Why Stronger Retrieval May Lower Downstream Cost

In the vendor article, the search Agent uses Nemotron 3 Ultra, and only the retrieval Embedding model is changed; downstream token cost is estimated from input and output counts with the stated pricing formula. What it shows is an observation under a specific Agent, dataset, and pricing caliber, not a real vendor bill, and it does not prove the same benefit for any self-built RAG system.

What can be brought back to a project is a falsifiable hypothesis: when correct evidence appears earlier, the Agent may reduce re-queries, expand fewer documents, lower input tokens, and thereby reduce task cost. To verify this hypothesis, the intermediate steps recorded cannot be only one total cost number. At minimum, associate the task ID, query text, recalled segment IDs, first valid evidence position, re-query count, input tokens, and final acceptance result.

For the same question “how does the new version cancel a task through the interface”, A recalls the old guide in the first round and the Agent searches another round; B finds the new version’s cancellation instructions in the first round and directly gives a verifiable call. If both pass the same acceptance, B’s one fewer query is the valid behavioral difference. If B returns an unverified summary, even with fewer inputs, that task cannot be counted as a successful task of the same quality.

This mechanism can also be covered by other components. When a Reranker already puts the correct evidence in first place, changing the first-stage recall model may not affect generation; when the Agent’s termination policy always requires two searches, improved first-round recall does not necessarily reduce calls; for problems that need cross-document composition, the second search may be a necessary action. In such results, record “retrieval improvement did not propagate downstream” instead of using the leaderboard to explain the savings.

The other counterexample is context bloat. Higher recall makes the Agent receive more relevant but duplicate long segments, answers become more complete, but input tokens increase. At this point, check whether segment deduplication, citation filtering, and context budget match the task. Whether higher cost is worth accepting is decided by predefined acceptance conditions; do not afterward pick only the metrics favorable to the new model.

Evaluation Metrics Must Be Split into Three Layers

Layer 1: Retrieval Quality

Recall reflects whether the needed evidence enters the candidate set, NDCG reflects the ranking quality of relevant evidence, and MRR focuses on the position of the first correct result. Multi-hop tasks cannot mark only one “correct page”; keep a minimal evidence set for each hop. No-answer tasks are counted separately for wrong citations after false recall; Chinese, code, version, and long-document layers are reported separately, so a large number of easy questions cannot hide a few key regressions.

Annotation must first decide whether the evidence really supports the answer. Containing the same noun is not relevance; an old-version document may answer a different version of the question; a table row without its header may lack constraints. Binding relevance labels to document version and segment position, then freezing the task set, reduces the bias of annotators changing standards after seeing candidate results.

Layer 2: Agent Behavior

Re-queries, query rewrites, Rerank calls, context expansion, and tool retries must be recorded, and the reasons for re-queries must be distinguished. A correction after first finding the wrong document and an active second search for evidence in a multi-hop task are both called a second search, but they have different meanings. Reducing only the total number of searches may reward an Agent that did not complete necessary evidence gathering. Behavior logs should make it possible to replay the decision “why to keep searching”.

Failures should also be layered. Empty retrieval results, vector service timeouts, generation-model rate limits, tool execution errors, and answer acceptance failures should be kept separately; do not write all of them as a single model failure. Otherwise, if changing the Embedding happens to reduce generation-service rate limiting, it can easily be misread as better retrieval. When infrastructure anomalies occur in paired tasks, the cost still enters the ledger; mark them separately in attribution and do not delete unfavorable samples.

Layer 3: Task and Cost

The final report simultaneously includes the number of successful tasks, total attempts, full cost, cost per successful task, end-to-end latency, and index cost. Acceptance must land on concrete conditions: code tasks pass agreed tests; fact questions cite valid evidence; no-answer questions refuse correctly. A model text reporting “done” is not task success. Manual review uses the same standard and, as much as possible, does not expose candidate names.

Task latency includes queueing, retrieval, reranking, generation, and tools; it is not labeling one model response time as the task time. Small samples can list per-item timings, but that cannot support a stable P95 claim. Long-document tail latency under concurrency, cache state, cold starts, and service rate limits should be measured later in load testing. The bounded accounting program in this article has no latency input and therefore outputs no throughput or tail-latency conclusions.

Single-Task Comprehensive Cost

Cost can be accumulated for one complete task: query embedding, vector search, reranking, generation input, generation output, tool calls, failure handling, plus index rebuild amortization. The costs already produced by retries must be accumulated per attempt; do not add a “retry total price” that contains the same consumption again, or billing will be duplicated. Failure handling records only additional manual or business overhead; if it cannot be valued, list the quantity separately and do sensitivity analysis instead of defaulting to zero.

Cost per successful task is the cost of all tasks in this round divided by the number of tasks that pass acceptance. Money spent on failed tasks stays in the numerator; a task that succeeds after retries appears once in the denominator. This caliber avoids packaging a model that fails more but has cheaper single requests into an appearance of overall savings. When all tasks fail, the denominator is zero; report it as undefined and retain the total investment; do not display zero dollars or delete that configuration.

Index rebuild amortization can also change the conclusion. Set rebuild cost at twelve hypothetical dollars, and suppose one hundred tasks are processed over the expected lifecycle; each task amortizes 0.12. If only ten tasks are actually used, the amortization rises sharply. In offline evaluation, the experimental expense of “building an index once to try a candidate” is not the same caliber as production continuous-service amortization; they should be in separate ledgers, then give high and low scenarios for expected task volume.

When comparing old and new schemes, distinguish total cost of ownership from incremental cost. The old index is already built, but that does not mean long-term rebuild, storage, and maintenance are forever free; the new index should not carry all historical costs that already belong to shared data cleaning. The numerical example sets A’s additional rebuild this time to zero and B’s additional rebuild to twelve, showing the incremental ledger for one replacement rather than a long-term production cost model.

Runnable Minimal Paired Accounting

The following code only needs the Python 3 standard library. Copy it to paired_embedding_cost.py and run python3 paired_embedding_cost.py in the directory where the file is saved. Nine manually written records cover A/B attempts for four tasks: both sides succeed on three and fail on one; A retries once on the second task. Query count can be larger than attempt count, meaning the Agent searched again within one attempt. Amounts use Decimal, and all unit prices are hypothetical.

The program first checks that the task sets match, attempt numbers are consecutive, and only the last attempt is terminal, then accumulates full consumption. It refuses to count two successful terminal states as two successes for the same task, and refuses to let a task that remains at retry enter completed statistics. Real logs need stable task_id and variant to join; the sample data is written directly in the code so readers can re-run without a service dependency.

from decimal import Decimal as D
from collections import defaultdict

# Synthetic accounting fixture; A/B are labels, not measured model results.
# task, variant, attempt, query calls, input tokens, output tokens, final status
ROWS = [
    ("t1", "A", 1, 1, 2000, 400, "success"),
    ("t2", "A", 1, 2, 4000, 500, "retry"),
    ("t2", "A", 2, 1, 2000, 300, "success"),
    ("t3", "A", 1, 2, 5000, 600, "failed"),
    ("t4", "A", 1, 1, 1500, 200, "success"),
    ("t1", "B", 1, 1, 1800, 350, "success"),
    ("t2", "B", 1, 1, 1800, 300, "success"),
    ("t3", "B", 1, 1, 4000, 450, "failed"),
    ("t4", "B", 1, 1, 1200, 200, "success"),
]
# All prices below are invented USD amounts for arithmetic only.
EMBED = {"A": D("0.001"), "B": D("0.003")}
REBUILD = {"A": D("0"), "B": D("12")}
EXPECTED_TASKS = 100

def summarize(rows):
    grouped = defaultdict(list)
    for task, variant, attempt, queries, inp, out, status in rows:
        if variant not in EMBED or status not in {"retry", "success", "failed"}:
            raise ValueError("unknown variant or status")
        if any(type(x) is not int or x < 1 for x in (attempt, queries)):
            raise ValueError("attempt/queries must be positive integers")
        if any(type(x) is not int or x < 0 for x in (inp, out)):
            raise ValueError("tokens must be non-negative integers")
        grouped[variant, task].append((attempt, queries, inp, out, status))
    task_sets = [{t for v, t in grouped if v == variant} for variant in ("A", "B")]
    if not task_sets[0] or task_sets[0] != task_sets[1]:
        raise ValueError("A/B task sets must match and be non-empty")
    summaries = {}
    for variant in ("A", "B"):
        cost = D("0")
        successes = attempts = query_count = 0
        for task in sorted(task_sets[0]):
            events = sorted(grouped[variant, task])
            if [e[0] for e in events] != list(range(1, len(events) + 1)):
                raise ValueError("duplicate or missing attempt")
            if any(e[-1] != "retry" for e in events[:-1]) or events[-1][-1] == "retry":
                raise ValueError("exactly one terminal outcome required")
            for _, queries, inp, out, _ in events:
                cost += queries * (EMBED[variant] + D("0.0005") + D("0.0008"))
                cost += D(inp) / 1000 * D("0.002") + D(out) / 1000 * D("0.004")
                cost += D("0.002")  # tool cost charged once per attempt
                attempts += 1
                query_count += queries
            if events[-1][-1] == "success":
                successes += 1
            else:
                cost += D("1")  # assumed extra failure handling cost, once per task
        tasks = len(task_sets[0])
        total = cost + REBUILD[variant] * tasks / EXPECTED_TASKS
        summaries[variant] = (tasks, successes, attempts, query_count, cost, total)
    return summaries

def reject(rows):
    try:
        summarize(rows)
    except ValueError:
        return True
    return False

result = summarize(ROWS)
assert result["A"] == (4, 3, 5, 7, D("1.0631"), D("1.0631"))
assert result["B"] == (4, 3, 4, 4, D("1.0480"), D("1.5280"))
assert reject(ROWS[:-1])  # unmatched task set
assert reject(ROWS + [ROWS[0]])  # duplicate attempt
assert reject(ROWS[:1] + [("t1", "A", 2, 1, 1, 1, "success")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, -1, 1, "success")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, 1, 1, "retry")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, 1, 1, "unknown")] + ROWS[1:])
assert reject([])
for variant, (tasks, successes, attempts, queries, online, total) in result.items():
    per_success = f"{total / successes:.6f}" if successes else "undefined"
    print(f"{variant}: tasks={tasks} success={successes} attempts={attempts} queries={queries}")
    print(f"  online_plus_failure_usd={online:.4f} amortized_total_usd={total:.4f}")
    print(f"  cost_per_success_usd={per_success}")
print("self_checks=9 passed; fixture_only=true; GPU_run=false")

The actual stdout from this run is below; exit code is zero, stderr is empty. The environment is local Python 3.13.5; this round only runs the accounting and nine assertions and does not call model services. After changing prices, task volume, or terminal states, the output should change accordingly, and the original exact-value assertions also need updating; this result cannot be reused to prove a new experiment.

A: tasks=4 success=3 attempts=5 queries=7
  online_plus_failure_usd=1.0631 amortized_total_usd=1.0631
  cost_per_success_usd=0.354367
B: tasks=4 success=3 attempts=4 queries=4
  online_plus_failure_usd=1.0480 amortized_total_usd=1.5280
  cost_per_success_usd=0.509333
self_checks=9 passed; fixture_only=true; GPU_run=false

A has seven total queries and B has four; B’s online consumption plus failure handling drops from 1.0631 to 1.0480. After adding the 0.48 rebuild amortization for these four tasks, B’s total becomes 1.5280, which is more expensive when divided by three successful tasks. It demonstrates the possibility that “downstream savings are not enough to cover the change cost”; it cannot prove that a real Embedding model would generate these call counts.

The nine assertions cover two sets of amount expectations, a missing task set, duplicate attempt, continuing after success, negative token, no terminal state, unknown status, and empty data. They help check the ledger constraints for this bounded input. Real collection must handle cross-service event duplication, currency conversion, late bills, and cache read/write pricing; before connecting real logs, expand input validation and billing reconciliation instead of treating this example as a complete cost platform.

Fairly Comparing Nemotron, Qwen3, and BGE

The first step is to freeze the business task set, corpus snapshot, chunk rules, metadata filters, top-K, vector database parameters, and Reranker. The second step is to freeze the generation model, system prompt, tool permissions, termination policy, and token and time budgets. The third step is to define the only variable: replacing the Embedding in this round and the input adaptation it requires. If Chunk and Agent Prompt are changed at the same time, start another experiment and separate the effect of the combined change from the effect of model replacement.

The Qwen3-Embedding-8B model card provides an Instruct format and a Query format on the query side, while the document side does not need the same instruction; the BGE-M3 model card distinguishes Dense, Sparse, and Multi-vector capabilities. If only BGE’s Dense output is compared, state clearly that this is the tested functional subset; enabling its hybrid retrieval while the other candidate only has Dense turns the comparison into a retrieval-system-level comparison, and the conclusion can no longer be attributed only to model weights.

Indexes with different dimensions cannot be mixed in one retrieval. Build an independent versioned index for each candidate, and verify that document count, segment count, filter conditions, and build configuration are consistent. Candidate dimension, normalization, and distance function must match the candidate’s output contract; fixing the same wrong distance function for all candidates is not a fair comparison. The first round can use the same business protocol with each candidate’s legal adaptation, and the second round can tune within an explicit budget separately, with different names for the two rounds.

Paired experiments first run every task on both A and B, interleaving the order to reduce load and cache differences over time. A random seed is only a traceable configuration and does not guarantee fully deterministic services; when generation behavior is unstable, use a pre-agreed number of repetitions and keep task-level results. The report can list four categories: both succeed, only A succeeds, only B succeeds, and both fail, and examine token and call differences separately without first deleting a candidate’s failure records.

A reusable record table should include experiment_id, task_id, variant, corpus_sha, chunk_config_sha, model_revision, query_adapter_sha, index_build_id, harness_sha, attempt_id, query and rerank counts, input and output tokens, terminal state, acceptance evidence, and pricing version. Missing items are written as “not collected”; do not fill in default values to fake the same conditions. In particular, after harness_sha changes, old results can be kept, but comparisons should be regrouped.

Realistic Boundaries of RTX A6000

For an existing GPU, selection first asks whether legal inference can run under the target length, batch, and concurrency, then looks at throughput. A simple estimate of weight bytes does not include activations, runtime cache, vector output, and service queues and cannot serve as memory peak. This article did not run loading or retrieval experiments on an RTX A6000, so it gives no throughput, maximum batch, or concurrency capacity for it.

NVIDIA’s announcement article puts the native acceleration of 1B NVFP4 in the Blackwell deployment scenario. The throughput ratios there cannot be transplanted into A6000 expectations. BF16 and quantization configurations also need to record backend, device, and precision separately; being loadable on one backend does not mean equal support on another. Before actual testing, check the official model card and backend docs; when failure happens, keep the error and configuration instead of changing the label and continuing to fill in a score sheet.

Service capacity also affects downstream cost. When a large Embedding model makes the queue longer, the Agent may trigger timeouts, retries, or fallbacks; those extra attempts should enter the full task ledger. Average vector latency on an idle single machine only describes that condition and is insufficient for online task latency. Record hardware facts, deployment feasibility, retrieval quality, and task benefit separately to know which evidence is actually needed next.

Common Mistakes

The easiest mistake is choosing a model by leaderboard and then computing tokens from a few successful answers. This misses failed costs, abnormal retries, and index changes. The correction is to freeze the task set first, connect all attempts back to their tasks, and count successes with the same acceptance standard. Small samples may publish per-item results, but avoid percentage slogans that claim to represent the entire business.

The second mistake is treating retrieval improvement as answer improvement. A segment with higher similarity may be a duplicate of the same version, and a citation that looks complete may add no information; the correct segment may also be ignored by the Agent. The recalled list, final citations, and acceptance evidence need to be connected. Saving only vector scores cannot check this hop, and saving only the final answer cannot distinguish retrieval failure from generation failure.

The third mistake is writing estimated costs as payment records. Hypothetical pricing is suitable for sensitivity analysis, vendor usage responses are suitable for consumption accounting, and actual bills are for reconciliation; their evidence strength differs. Token costs converted by a vendor pricing formula do not include your vector storage, manual review, cache discounts, or index lifecycle. The report should note the pricing version, units, and excluded items next to the amount.

Decision Rules

Write quality constraints before looking at results, for example: key version questions must not mis-cite old documents, and no-answer tasks must refuse correctly; the general task success threshold is set by business need, and this article does not set a universal percentage. Then check whether candidates meet these conditions. A scheme that does not pass the threshold, even if cheap, does not enter the conclusion of “same quality but cheaper”.

For candidates that pass, compare total cost per successful task, end-to-end latency, and migration amortization, while looking at regressions within task types. If quality is the same and online consumption is lower but rebuild cost is high, give the payback boundary under expected task volume; if the cost reduction depends on a larger cache hit rate, add cold-cache conditions; if the benefit appears only in code tasks, restrict the conclusion to that traffic subset and test it first.

Finally, keep a rollback-able old index and explicit switch conditions. An Embedding change means the document and query encoding contracts change together; rollback must restore the corresponding index and adapters, not just change the model name. First use a bounded task set to confirm quality, then use representative load to confirm capacity, and only after cost reconciliation decide whether to replace. What can be done immediately is to re-run the numerical example in this article, map your own task records into the same caliber, and list the evidence still not collected.

References

• NVIDIA Nemotron 3 Embed 1B BF16 official model card
• NVIDIA Nemotron 3 Embed 8B BF16 official model card
• NVIDIA announcement and vendor Agent evaluation notes
• Qwen3-Embedding-8B official model card
• BGE-M3 official model card