Request demo

Evaluate

xreduce evaluate runs a model against your benchmark data, scores the outputs, and emits accuracy, latency, energy, and calibration metrics. This is the main act of the workflow.

By the time you reach this step you should already have a registered model with a populated config.yaml and a benchmark JSONL in benchmarks/. If not, work through Initialize a project, Register a model, and Benchmark data first.

Basic usage

xreduce evaluate --config my-org/my-model/config.yaml

Runs the model against every .jsonl file in my-org/my-model/benchmarks/, scores each one, and prints per-benchmark and combined metrics.

Evaluating Qwen/Qwen3-4B...
   Benchmark files: 1

Evaluating text2sql.jsonl...
   text2sql.jsonl complete
      Samples: 100
      Accuracy: 66.00%
      ECE: 0.092
      Latency P50: 421 ms
      Latency P99: 1088 ms
      Energy/sample: 3.4 mWh

All benchmarks complete (1 files)

Evaluating multiple models against the same benchmark

--config is repeatable. Pass it multiple times to evaluate several models in sequence against a shared benchmark. Each model is loaded, evaluated, and released before the next is loaded.

xreduce evaluate \
  --config models/qwen3-4b/config.yaml \
  --config models/qwen2.5-coder-7b/config.yaml \
  --config models/smollm2-1.7b/config.yaml \
  --benchmark shared/benchmarks/text2sql.jsonl

Use --fail-fast to abort on the first config failure instead of continuing through the remaining models.

Flags

Flag	Description
`--config` / `-c` (required, repeatable)	Path to `config.yaml`. Repeat for multiple models.
`--benchmark`	Benchmark to evaluate. Three forms:
`--optimization-config` / `-o`	Optimization strategy to apply. Defaults to `baseline`. See config.yaml reference for the full list.
`--baseline` / `-b`	UUID of a previous evaluation. When provided, this run's results are reported as deltas from that baseline.
`--show-outputs`	Print per-sample predictions and misses. Useful for seeing what the model actually produced.
`--show-errors`	Print only the samples the model got wrong, with error breakdown.
`--verbose` / `-v`	Show detailed logging.
`--fail-fast`	Stop on first config failure (only meaningful with multiple `--config` flags).

Filename only, e.g. text2sql.jsonl - resolved from <config_dir>/benchmarks/.
all - runs every .jsonl in <config_dir>/benchmarks/.
Shared or external path, e.g. shared/benchmarks/text2sql.jsonl - used as-is.

Baseline runs and deltas

To compare a new run against a previous one, pass the previous run's UUID via --baseline:

# First run - establish the baseline
xreduce evaluate --config my-org/my-model/config.yaml
# → Run ID: 8f3a1e22-4d57-4c1a-a8e3-6f02b7e9c0a1

# Later - same model, after config tweaks, compared to that baseline
xreduce evaluate \
  --config my-org/my-model/config.yaml \
  --baseline 8f3a1e22-4d57-4c1a-a8e3-6f02b7e9c0a1

For richer head-to-head comparison across models or benchmarks, use xreduce compare instead - see Compare.

Seeing per-sample outputs

Add --show-outputs to see the model's actual predictions alongside the expected outputs. Add --show-errors to see only the samples that scored below threshold:

xreduce evaluate --config my-org/my-model/config.yaml --show-outputs
xreduce evaluate --config my-org/my-model/config.yaml --show-errors

What's next

Read the results. See Understanding results for what each metric means.
Compare across runs and models. xreduce compare renders the cost-quality matrix as a formatted table - see Compare.