Decision models provide an alternative to free-form text generation by large language models (LLMs): instead of producing token-by-token responses, they evaluate an input state against a set of predefined answer primitives and return typed decisions together with probability distributions. A common use case is customer-support routing: from the message "I was charged twice for my subscription. I need a refund today." an application needs to determine which team should handle it, how urgent it is, and whether the customer requests a refund. Decision models directly return these answers as probabilities rather than generating free text.
How they differ from traditional approaches
The decision-model interface can be summarized as: State + Questions + Allowed Answers → Decisions + Probabilities. It supports three fundamental primitives:
- Choice: a discrete distribution over options (e.g., Billing, Technical, Sales). The model assigns a logit to each option and softmax yields probabilities.
- Score: for ordered scales, the model returns an expected value computed from the probability distribution over levels (e.g., 0=low, 1=medium, 2=high), and reports that expectation as the score.
- Noul: a Bernoulli-style binary primitive that returns the probability the answer is true (for yes/no questions, e.g., "Is the customer requesting a refund?").
Decision models let you ask multiple typed questions about the same state and receive probabilistic, typed answers for each.
Why not just ask an LLM to emit JSON?
Many applications already prompt LLMs to produce JSON or structured responses for classification and routing. Shortcomings of that approach include:
- Models may not follow the required schema reliably (invalid JSON, unexpected keys, or answers outside allowed choices).
- Constrained decoding or grammar enforcement fixes schema issues but introduces computation and engineering overhead.
- Autoregressive text generation is unnecessary when you only need to pick from predefined options.
Decision models make decisions and probabilities first-class, non-generative outputs, which can be optimized and calibrated for reliable behavior.
Confidence and how it’s computed
Besides returning probability distributions, the interface gives a "confidence" value for Choice and Score. For a Choice with K options, confidence is computed as
C = (p_max - 1/K) / (1 - 1/K)
where p_max is the largest predicted probability. Intuitively: a uniform distribution yields confidence 0; a distribution concentrated on a single option yields confidence 1. Confidence summarizes how concentrated the model’s belief is around the top option.
How Hugging Face supports decision models
Hugging Face aims for a common decision-model interface while allowing diverse internal implementations. Key pieces of the ecosystem:
- Serving with Transformers
- Transformers added DecisionPipeline, AutoModelForDecision and DecisionConfig classes tailored for decision models.
- There is a /v1/systemone endpoint and local serving via "transformers serve" (with the serving extras installed).
- Example responses include typed answers (department as choice, urgency as score, refund as noul) with per-option probabilities, confidence and usage statistics. In one demonstration the API returned "input_tokens": 148 and "output_tokens": 0.
- The Transformers implementation can also wrap compatible autoregressive LLMs by constructing decision prompts, evaluating label logits in a batched forward pass, and applying softmax without generating text. This enables experimentation with the interface without necessarily training new models.
- Serving with llama.cpp
- llama.cpp supports the same /v1/systemone API and can launch GGUF decision models (for example: "llama serve -hf ggml-org/Kev-4B-GGUF"). The default server port used in the example is 9931.
- Because the API is compatible, different runtimes can serve the same decision requests interchangeably.
- Discovering models on the Hub
- Hugging Face introduced a dedicated "decision-model" tag for discovering compatible models. The tag captures a variety of implementations (encoder-based like ModernBERT and models derived from larger generative backbones).
- Training decision models with Hugging Face tooling
- Decision models integrated in Transformers can be fine-tuned with the Trainer API. The article shows a small toy example fine-tuning an "autotrust/JEV-9B" checkpoint to route support requests: each training example is a decision request (state + questions) paired with a gold answer, the decision_config prepares inputs, and a choice question is trained with cross-entropy over the options.
- First-class TRL support is forthcoming, and recipes exist (e.g., Unsloth) to adapt generative LLMs into decision models using LoRA and a decision-scoring head.
Practical consequences
Decision models do not invent probabilistic classification or runtime-defined labels—classical classifiers and zero-shot methods already offer parts of this functionality—but they unify these features into a typed, probability-first API with native primitives for multiple question types and non-generative scoring. This unification makes it easier to build production systems that need calibrated decisions (routing, triage, yes/no detection, ordered ratings) and to reuse tooling across runtimes (Transformers, llama.cpp) and models discovered on the Hub.
Summary
Decision models elevate decisions and their probabilities to primary, machine-readable outputs. Hugging Face and associated runtimes are standardizing an interface (DecisionPipeline, /v1/systemone, decision-model tag) so that models with different internals can be discovered, served and fine-tuned consistently. For structured application logic—routing tickets, scoring urgency, answering binary questions—this approach reduces overhead from generative decoding and provides richer, calibrated signals for downstream systems.
