Understanding results

What each number means, how scoring works, and how to use the results to decide what to deploy.

The headline metrics

Every evaluation run reports four numbers per benchmark file.

Accuracy

The share of benchmark samples the model got right. What counts as "right" depends on the task_type in config.yaml. Different tasks need different scoring logic - a classification task is scored differently than a summarization task.

ECE (Expected Calibration Error)

How closely the model's confidence tracks its actual accuracy. If a model is 90% confident across a bucket of predictions, ECE measures how close the model's actual accuracy on that bucket is to 90%.

Lower is better. ECE under 0.10 means the model's confidence is trustworthy. ECE over 0.20 means the model is confidently wrong often enough to matter - usually worse than being uncertain.

Latency P50 and P99

Median and 99th-percentile per-sample inference time. P50 tells you what most requests will experience. P99 tells you what your slowest 1% will experience. The gap between them matters as much as either number alone - a large gap means unpredictable tail latency under load.

How scoring works

XReduce picks a scoring method per benchmark based on the task_type in your config.yaml. Each method answers the question "did the model get it right?" differently:

If you need scoring that doesn't fit a built-in task type - say, semantic equivalence on paraphrased answers, or domain-specific output validation - that's where LLM-as-judge comes in.

LLM-as-judge

Some outputs can't be scored by string comparison. A model asked "what's the capital of France" might answer "Paris," "It's Paris," or "The capital is Paris" - all correct, none exactly matching a single reference. For these cases, XReduce runs a separate judge model that reads the question, the model's answer, and the reference answer, and decides whether they mean the same thing.

What XReduce uses today

The judge is openai/gpt-oss-120b, an open-weight large model run via Groq. It's configured via the judge_model field in config.yaml, and produces binary pass/fail scores for semantic correctness.

LLM-as-judge runs asynchronously alongside the main evaluation. Scores are written to the judge_results table and surface in the XReduce dashboard next to the primary accuracy number.

Where this is going

The current LLM-as-judge surface is narrow - one judge model, one scoring mode, task-type-automatic selection. We're expanding it in the next release cycles:

• Configurable judge model. Pick any XReduce-supported model as the judge. A task that needs reasoning will use a stronger judge; a high-volume task can use a cheaper one.
• Custom judge prompts. Supply your own judging prompt with {question} and {answer} template slots. Useful for domain-specific correctness - a legal correctness judge reads differently than a customer-support correctness judge.
• Correctness cutoffs. For judges that produce numeric scores (1–10 or 0–100), set the cutoff above which a response counts as correct.
• Additional computed metrics. BLEU, ROUGE, and METEOR as opt-in metrics alongside LLM-as-judge, for teams that want reference-based scoring for text generation and translation.
• Per-rubric scoring. Make the rubric used for each run explicit in the output - so you know which scoring logic applied to which benchmark.

None of this changes the shape of the evaluation run - you'll still use xreduce evaluate. What changes is how much of the scoring behavior you control.

Reading the results

A few practical points when looking at a result block:

• Sample count matters. Accuracy on 20 samples swings by 5% or more between runs just from randomness. Accuracy on 200 samples is stable enough to compare two models. Accuracy on 1000+ samples is stable enough to compare two versions of the same model.
• Compare runs, not absolute numbers. A model that scores 78% on your benchmark isn't "78% good" - it's 78% relative to your reference answers. The useful comparison is this model vs. that model on the same benchmark, not this model's number against some external target.
• Use --baseline. Pass --baseline <run-uuid> to see your current run as a delta from an earlier one. Catches regressions that absolute numbers hide.
• Use --show-outputs when debugging. Aggregate accuracy tells you whether the model is working. Per-sample outputs tell you how it's failing - which categories of inputs it gets wrong, where the mistakes cluster. Usually more actionable than the headline number.

What's next

• Go deeper on custom scoring. Custom model loaders documents how to wire in domain-specific validation logic.
• Compare across models. The XReduce dashboard shows the cost-quality matrix - every model, every benchmark, every run, side by side.