Supported models
The complete matrix of what XReduce supports today and what's coming. If your model isn't listed, - we're actively expanding coverage based on customer demand.
Status legend
Run through XReduce end-to-end by our team.
Loads via our existing HuggingFace transformers path. Should work; not all variants formally tested.
Active engineering work, targeted for upcoming release.
On the roadmap.
Distilled specialist models
Small, fine-tuned models built for specific tasks. High accuracy on narrow domains at a fraction of the cost of general-purpose models.
| Provider / Family | Examples | Status |
|---|---|---|
| Distil Labs | text2sql-qwen3-4b, and other Distil Labs specialists | Verified |
| Customer fine-tuned variants | Your own specialists on open-weight base models | Compatible |
Open-weight base and instruct models
General-purpose models from major open-weight families. These load through the standard HuggingFace transformers path.
| Family | Notable models | Status |
|---|---|---|
| Qwen | Qwen3-0.6B, Qwen2.5-Coder-7B-Instruct | Verified |
| Qwen (other sizes) | Qwen2, Qwen2.5, Qwen3 in all sizes | Compatible |
| SmolLM | SmolLM2-1.7B-Instruct | Verified |
| GPT-2 | gpt2 (useful for smoke-testing) | Verified |
| Llama | Llama 3.1, 3.2, 3.3 (requires HF license acceptance) | Compatible |
| Mistral | Mistral 7B, Mixtral 8x7B, Mistral Small, Ministral | Compatible |
| Gemma | Gemma 2, Gemma 3 (some variants gated) | Compatible |
| Phi | Phi-3, Phi-4 | Compatible |
| DeepSeek | DeepSeek-V2, DeepSeek-V3, DeepSeek-Coder | Compatible |
| Yi | Yi, Yi-1.5 | Compatible |
| Falcon | Falcon, Falcon 2, Falcon 3 | Compatible |
| OLMo | Allen AI's OLMo series | Compatible |
| GPT-OSS | OpenAI's open-weight models | Compatible |
| Command | Cohere's open-weight Command family | Compatible |
| StarCoder | StarCoder, StarCoder2 | Compatible |
| DBRX | Databricks' DBRX | Compatible |
| MPT | MosaicML's MPT family | Compatible |
Serving runtimes
Most production self-hosted inference runs through a serving runtime rather than loading weights in-process. Support for hitting these endpoints is our top near-term priority - today, XReduce loads weights directly; the runtime path is coming.
| Runtime | Description | Status |
|---|---|---|
| vLLM | High-throughput serving engine for open-weight LLMs | In progress |
| TGI (Text Generation Inference) | HuggingFace's serving runtime | In progress |
| Ollama | Local model runtime with OpenAI-compatible API | In progress |
| llama.cpp / GGUF servers | CPU-friendly quantized inference | In progress |
| SGLang | Fast serving for LLMs with structured output | Planned |
| TensorRT-LLM | NVIDIA's optimized inference runtime | Planned |
| MLC-LLM | Universal deployment across devices | Planned |
| TorchServe | PyTorch-native serving | Planned |
| LocalAI / LM Studio | Local OpenAI-compatible servers | Planned |
| Custom OpenAI-compatible endpoints | Any in-house server exposing the OpenAI API pattern | In progress |
Hosted APIs
XReduce's primary focus is self-hosted inference, but hosted APIs appear in many real-world routing scenarios as comparison points or fallbacks. These are planned for later phases.
| Provider | Notable models | Status |
|---|---|---|
| OpenAI | GPT-4o, GPT-4o-mini, o-series | Planned |
| Anthropic | Claude Opus, Claude Sonnet, Claude Haiku | Planned |
| Gemini 2.5 Pro, Gemini 2.5 Flash | Planned | |
| Fireworks | Fireworks-hosted open-weight models | Planned |
| Together AI | Together-hosted open-weight models | Planned |
| Groq | Groq-hosted open-weight models | Planned |
| DeepInfra | DeepInfra-hosted open-weight models | Planned |
| OpenRouter | Unified access to multiple providers | Planned |
Model formats
| Format | Description | Status |
|---|---|---|
| HuggingFace Hub | Models pulled from huggingface.co by identifier | Verified |
| Local HF-format directory | HuggingFace-format weights stored locally | Compatible |
| GGUF | Quantized format used by llama.cpp and Ollama | In progress |
| GPTQ / AWQ | Common quantization formats | Compatible |
| ONNX | Open Neural Network Exchange format | Planned |
| TensorRT engines | NVIDIA-compiled inference engines | Planned |
| PyTorch .pt checkpoints | Raw PyTorch model files outside HF format | Planned |
Task types
Scoring logic and evaluation rubrics supported today and expanding. See Benchmark data for per-task JSONL formats.
| Task type | Description | Status |
|---|---|---|
| Classification | Intent detection, categorization | Verified |
| Text completion | Short-form text completion | Verified |
| Question answering | QA with contains-match or LLM-as-judge scoring | Verified |
| Tool calling | Function calling with JSON equivalence scoring | Verified |
| Text-to-SQL | Structured SQL generation | Verified |
| Summarization | Text summarization with word-overlap scoring | Compatible |
| Translation | Language translation with exact-match scoring | Compatible |
| Structured generation | JSON, schema-constrained output | In progress |
| Multi-turn conversation | Dialogue evaluation across multiple turns | In progress |
| RAG (retrieval-augmented generation) | Faithfulness and answer relevance scoring | Planned |
| Code generation | Syntactic + execution-based scoring | Planned |
| Vision-language | Multimodal model evaluation | Planned |
Evaluation metrics
Scoring methods and metrics available today and on the roadmap. See Understanding results for how each is used.
| Metric | Description | Status |
|---|---|---|
| Accuracy (exact match) | Share of samples with exactly-correct output | Verified |
| JSON equivalence | Order-insensitive structured output match | Verified |
| Contains match | Reference string found in output | Verified |
| Word overlap | N-gram overlap threshold (summarization) | Verified |
| LLM-as-judge (default) | Semantic evaluation via gpt-oss-120b | Verified |
| Configurable judge model | Choose any supported model as judge | In progress |
| Custom judge prompts | User-defined LLM-as-judge templates | In progress |
| Correctness cutoffs | Numeric judge scores with configurable thresholds | In progress |
| ECE (Expected Calibration Error) | Confidence calibration measurement | Verified |
| Latency percentiles (P50, P99) | Per-sample inference time distributions | Verified |
| BLEU | N-gram precision metric for translation | Planned |
| ROUGE | N-gram overlap for summarization | Planned |
| METEOR | Synonym-aware text similarity | Planned |
| Ragas Faithfulness | Context grounding for RAG pipelines | Planned |
| Ragas Answer Relevance | Response relevance for RAG pipelines | Planned |
Hardware
| Hardware | Description | Status |
|---|---|---|
| NVIDIA GPUs (CUDA) | A100, H100, L40S, RTX series | Verified |
| CPU (x86) | For small models and smoke-testing | Verified |
| Apple Silicon (MPS) | M-series chips for local development | Compatible |
| AMD GPUs (ROCm) | MI-series GPUs | Planned |
| Google TPUs | v4, v5e, v5p | Planned |
| AWS Inferentia / Trainium | AWS-specific accelerators | Planned |
Don't see what you need?
This matrix reflects our current state and near-term roadmap. Priorities shift based on customer demand - if something you need is marked Planned or isn't listed, and we'll tell you honestly where it sits in our backlog.