Judge prompt engineering: Reducing evaluation bias

LLM evals look clean on a slide; in production they get messy. A single prompt tweak can swing scores, and bias sneaks in before anyone notices.

This guide shows a pragmatic way to run prompt-based evaluations that people trust. Start with structure, add domain signal, and clamp down on bias. Then back it with experiments so the numbers hold up outside a notebook.

Setting the stage for prompt-based evaluations

Start with highly structured instructions: roles, goals, and explicit rubrics. Spell out scales and ties; ban style points. The C.R.A.F.T. and role-based patterns in the Prompt Evaluation Framework are a solid starting point (techniques for effective prompts). Lenny’s PM evals guide shows how concrete rubrics beat vibe checks (PM evals).

Make the judge fluent in the domain. Load terms, edge cases, and failure modes. Use few-shot examples that match the exact task. The custom metrics playbook shows how to encode this signal, and the ACL industry paper underlines why consensus and references matter once models leave the lab (custom LLM metrics; ACL industry).

Bias shows up early. Cap rationale length and reward evidence over word count to curb verbosity bias (arXiv). Randomize candidate order to fight positional bias and forbid tie-breaking by length or tone (prompt evaluation framework).

If using an LLM as a judge, set clear criteria and strict formats. Require short rationales, citations when relevant, and a fixed schema. A meta-judge can review the judge for rule adherence and consistency (LLM-as-judge strategies).

Anchor results to reference outcomes and concrete thresholds. Start with pass or fail against a reference, then grade nuance. Borrow guardrails and G-Eval ideas from the custom metrics guide (custom LLM metrics).

Here’s what to lock down upfront:

• Business success metrics tied to decisions, not just offline accuracy. HBR’s primer on online experiments is still the north star (HBR on A/B tests).

• Human calibration with labeler agreement tracked over time. Bayesian extrapolation notes help interpret small, noisy samples (Bayesian extrapolation).

• Selection bias checks. If overperformers still lose slots, that’s a red flag Paul Graham flagged years ago (detect bias).

Teams that run offline evals side-by-side with experiments in Statsig tend to catch shaky wins early and avoid painful rollbacks.

Mapping out bias types and their impact

With LLM-as-judge setups, three biases hit hardest and fast:

• Positional bias: head-to-head order skews outcomes. Randomize placement and blind IDs (prompt evaluation framework).

• Verbosity bias: longer text gets higher scores even when quality drops as length rises (arXiv). Cap rationale tokens and set penalties for fluff.

• Conformity bias: models gravitate to consensus, not truth. Meta-judges help by checking criteria, not popularity (prompt evaluation framework).

Chain of thought is helpful, yet it can mislead. Overweighting steps can cement bad logic; small flaws cascade across criteria. Keep roles crisp and rubrics tight; use step audits sparingly and with purpose (LLM-as-judge strategies). Practical moves:

• Score facts first, steps second. The final answer must stand on its own.

• Hide intermediate text when ranking candidates to avoid halo effects.

• Cross-check steps with a second model when you truly need them, then reconcile differences (ACL industry).

Bias also hides inside scoring guidelines. Uneven thresholds across groups quietly warp outcomes; outperformers still lose spots, a pattern Paul Graham called out (detect bias). Bake audits into the evaluation and track disparities by category, as recommended in PM evals and custom metrics guides (PM evals; custom LLM metrics).

Crafting balanced prompts for consistent judgments

Role-based prompts keep judges focused. State what matters and what does not so style never outruns substance.

A compact judge template:

• Role: factual arbiter; ignore tone; cite sources; flag missing information.

• Criteria: correctness first; instruction-following second; presentation last.

• Output: numeric score on a defined scale; short rationale capped at N tokens; quotes or links as evidence; risk tag for hallucination or safety concerns (prompting strategies; few-shot examples).

• Review cadence: sample disagreements regularly and recalibrate using prompt design guidance from The Sequence (designing prompts).

Shift feedback toward structure. Penalize verbosity; reward grounded claims. Call out known biases in the instructions so they are not ignored later (verbosity bias; positional bias).

Adopt an iterate–compare–recalibrate loop. Track judge-human agreement on targeted samples, review misses, then tighten the rubric. Meta-judges are useful for protocol checks and catching drift before it hits production (meta-judge patterns). Teams that tie these loops to rollout decisions in Statsig usually ship faster with fewer surprises.

Integrating cross-checks for unbiased scoring

Single-judge setups are brittle. Add cross-checks and make them earn their keep.

• Multi-judge consensus with diverse models reduces single-model quirks. A meta-judge can synthesize and call fouls on process violations (Prompt Evaluation Framework).

• Calibration and contrastive pairs help counter verbosity and style effects. The ACL industry notes and verbosity study outline workable patterns (ACL industry; arXiv).

• Concise rationales only. Reject votes that do not cite criteria or evidence (LLM-as-judge strategies).

Pair automation with human review for accountability. Route safety, hallucination, and correctness escalations to the right reviewers; both LLM-as-judge playbooks and PM eval guides recommend this step (LLM-as-judge strategies; PM evals).

Validate improvements against historical ground truth and track drift by cohort. Treat this like A/B test hygiene: rerun surprising wins and confirm with online experiments (HBR on experiments). Use Bayesian extrapolation to interpret sparse data and keep an eye out for selection bias patterns highlighted by Paul Graham (Bayesian extrapolation; bias test). Teams that wire these checks into Statsig usually spot regressions early and keep shipping.

Closing thoughts

LLM evals work when the rules are explicit, the bias traps are handled, and results tie back to real business goals. Start with roles, rubrics, and domain signal. Layer in cross-checks, meta-judges, and human reviews. Then validate with experiments so the numbers hold outside the notebook. Structure beats style, every time.

More to dig into:

• Prompt Evaluation Framework on roles, rubrics, and bias types (r/PromptEngineering)

• HBR’s explainer on why online experiments settle arguments (HBR)

• The ultimate guide to custom LLM metrics and few-shot setups (r/PromptEngineering)

• ACL industry notes on production-grade judging and consensus (ACL industry)

• Verbosity bias findings and mitigation ideas (arXiv)

• Bayesian extrapolation for reading small, noisy experiments (post)

Hope you find this useful!