JavaScript · Architecture

Harness Engineering: Taming LLM Output with Parallel Sampling and Automated Judging

By Jasper_L · Aug 11, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

LLM outputs are non-deterministic and often wrong in ways that break downstream systems. Harness engineering gives teams a practical, model-agnostic way to raise reliability without touching weights — but only if they understand that the judge itself is the weakest link and must be constrained with rule-based verification.

Summary

A Harness pipeline runs the same prompt N times in parallel, scores every candidate with an automated LLM judge, and picks the highest-scoring result. The approach trades compute for quality: if a single generation has a 30% chance of correctness, five independent attempts push the success rate above 80%, all within the wall-clock time of one request. The three stages — generate, evaluate, select — are decoupled so each can be hardened independently.

Productionizing the pattern requires several additions. A rule-based verifier (unit tests, schema checks) should filter candidates before the LLM judge ranks them, because LLM judges carry systematic biases — position bias, verbosity bias, and self-preference — that inflate pass rates by as much as 17 percentage points over script-based verification. Multi-judge ensembles using different model families reduce that bias, and every run must log the full evaluation trail for bad-case analysis and prompt iteration.

The technique works best for code generation and mathematical reasoning, where correctness is verifiable. It breaks down for open-ended creative tasks and high-stakes decisions where judge bias dominates. When a particular N value consistently delivers, the selected outputs become training data for rejection-sampling fine-tuning, which bakes the selection pressure into the model weights and eliminates the N× inference cost.

Takeaways

— Best-of-N sampling raises the probability of at least one correct answer from p to 1-(1-p)^N; with p=0.3 and N=5, that reaches roughly 83%.

— Promise.all keeps N parallel generations within the wall-clock time of a single request, making the compute-for-quality trade viable.

— LLM judges carry four systematic biases — position, verbosity, self-preference, and style — that can inflate pass rates by 17 percentage points over script-based verification.

— Rule-based verifiers (unit tests, type checks, schema validation) should filter candidates before the LLM judge ranks them, confining judge bias to a small pool of already-valid outputs.

— Multi-judge ensembles using 2–3 different model families reduce bias more cost-effectively than doubling N, because judging is cheaper than generating.

— Every Harness run must log the prompt, all candidate outputs, each judge score and rationale, and the final selection to enable bad-case clustering and prompt iteration.

— Rejection-sampling fine-tuning can bake the selection pressure into model weights, eliminating the N× inference cost once a stable N value is identified.

— Harness is suited for code generation and math reasoning where correctness is verifiable; it fails for open-ended creative tasks and high-stakes decisions where judge bias dominates.

Conclusions

The 17-percentage-point gap between script-based verification and LLM-judge pass rates is not a footnote — it quantifies how much a naive Harness pipeline overestimates its own quality, and it explains why rule-based pre-filtering is non-negotiable in production.

Position bias in LLM judges means that simply reordering candidates in a prompt can flip the winner, which makes single-judge Harness pipelines dangerously unstable for any task where the ranking order matters.

The conservative principle from the RHO system — rejecting the top candidate unless it strictly beats a baseline — is a cheap hedge against judge noise that most Harness implementations overlook.

Harness engineering inverts the usual ML intuition: instead of spending budget on a single high-quality inference, it spends on many cheap inferences and a cheap judge, betting that selection pressure is cheaper than model improvement.

Concepts & terms

Best-of-N Sampling

Running the same prompt N times in parallel and selecting the best output according to some scoring function. The probability of at least one correct answer becomes 1-(1-p)^N, where p is the single-shot success rate.

LLM as Judge

Using a language model to score or rank the outputs of another language model, enabling automated evaluation loops without human review. Prone to position bias, verbosity bias, self-preference, and style bias.

Rejection Sampling Fine-tuning

A technique where high-scoring outputs from a Best-of-N pipeline are collected and used as training data to fine-tune the base model, embedding the selection pressure into the weights so that a single inference later produces quality comparable to the original N-shot pipeline.

Multi-Judge Ensemble

Using multiple LLM judges from different model families to score the same candidate, then averaging the scores. Reduces individual judge bias at a cost lower than doubling the number of generated candidates.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗