Request demo

Compare

xreduce compare renders the cost-quality matrix as a head-to-head comparison table. Use it after you've evaluated two or more models to see how they stack up on accuracy, latency, energy, and cost.

Two modes

Head-to-head: pass two or more model identifiers positionally, plus a --benchmark filename to compare them on. Models can be specified by UUID (from xreduce model list) or by a substring of the model name.

xreduce compare qwen3-4b qwen2.5-coder-7b --benchmark text2sql.jsonl

Fleet-wide: pass --task-type to compare every model registered to your account for that task type. Mutually exclusive with positional model arguments.

xreduce compare --task-type text-to-sql

Example output

benchmark: text2sql.jsonl

                          qwen3-4b      qwen2.5-coder-7b   smollm2-1.7b
   Accuracy               66.0%         32.0%              30.0%
   ECE                    0.092         0.184              0.211
   Latency P50            421 ms        892 ms             142 ms
   Latency P99            1,088 ms      2,341 ms           388 ms
   Energy / sample        3.4 mWh       7.1 mWh            1.2 mWh
   Cost / 1K              $0.0042       $0.0089            $0.0016

Lower-is-better metrics (ECE, latency, energy, cost) are flagged in the rendered table. Pass --json for the raw matrix rows instead of a formatted table.

Flags

Flag	Description
`models` (positional)	Two or more model identifiers (UUIDs or name substrings) to compare. Required unless `--task-type` is set.
`--task-type`	Compare all account models registered with this task type. Mutually exclusive with positional model arguments.
`--benchmark`	Benchmark JSONL filename (basename match - e.g. `text2sql.jsonl` matches both `text2sql.jsonl` and `shared/benchmarks/text2sql.jsonl`). Required when models are listed positionally; optional in `--task-type` mode (results group by benchmark).
`--profile`	Credentials profile to use. Defaults to `default`.
`--json`	Emit raw matrix rows as JSON instead of a formatted table.

What gets compared

Metric	Direction	Notes
Accuracy	Higher is better	Share of samples scored correct against expected output.
ECE	Lower is better	Expected calibration error. Measures confidence alignment.
Latency P50 / P99	Lower is better	Median and 99th-percentile per-inference wall-clock time.
Energy / sample	Lower is better	Energy used per inference, in µWh / mWh / Wh depending on scale.
Cost / 1K	Lower is better	Cost per 1,000 inferences, derived from energy × electricity cost + infra amortization.

Resolving model identifiers

Both UUIDs and name substrings work. If a substring matches more than one model, compare tells you which models matched and asks you to disambiguate. List your models any time with:

xreduce model list