Decide, don't generate

Most systems that added a model in the last three years did not add a writer. They added a classifier, a router, a validator or a triage step, and they wrapped it in a chat-shaped API because that was the only shape available. The wrapper is not free. It costs the latency of generating tokens nobody reads, the price of a general-purpose model, and the ambiguity of a free-text answer that has to be parsed back into the enum the system actually needed.

What the traffic actually is

Look at what a typical assistant is asked to do in production and the distribution is embarrassing. Route this support ticket to one of eleven queues. Decide whether this invoice matches its purchase order. Classify this incident as access, capacity or data. Check whether this generated answer is supported by the retrieved passage. None of these need a paragraph. All of them need a label, and most need a number saying how sure the system is, so that the uncertain cases can be sent somewhere else.

That is not a small optimisation. It changes three properties of the system at once. The output space becomes finite, which means the failure modes can be enumerated and the tests can be written against a schema. The cost per call becomes roughly independent of the output length, which matters when the call volume is in the millions. And the audit becomes possible: a decision can be recorded as a decision, with the model version and the calibrated probability attached, instead of a transcript that a human has to re-read.

What a decision model is

The category is new enough that the definitions still move, but the shape is consistent across the implementations I have seen described. The model returns a typed choice from a declared set, plus a probability distribution over that set. There is no free-text channel, so there is nothing to parse and nothing to inject into the downstream system. Some implementations constrain the output with a solver or a compiler so that the returned value is structurally valid by construction rather than by retry.

The earliest public example in this wave is Jev, announced in early access by TypeSafe in September 2026 [TypeSafe 2026]. The claim in the announcement is that a decision model can be two orders of magnitude faster and cheaper than a frontier chat model on structured tasks, because it does not generate tokens. A second implementation, Laya, targets the same shape with open weights and local inference [Innovations 2026]. The two are interesting together precisely because they disagree about deployment: one managed and calibrated centrally, one self-hosted and tuneable.

The quote is a marketing-shaped sentence, and I am quoting it because it is also the best short description of the architectural problem [Waehner 2026]. The interesting part is not that a smaller model can classify. It is that once the decision is a first-class object, the surrounding system can be designed around it: thresholds, abstention, escalation to a general model, human review for the tail.

Calibration is the product

Raw model confidence is a number that looks like a probability and is not one. A classifier that is right ninety-four percent of the time can be systematically wrong in the region where it is most confident, and there is no way to know from the logits alone.

Calibrated output changes what an operator can build. If the probability means what it says, then a threshold is a policy: route everything above 0.9 automatically, send 0.6 to 0.9 to a human queue, and treat anything below 0.6 as “the taxonomy does not cover this” rather than forcing a label. That third branch is the one most systems are missing, and it is the one that prevents the slow accumulation of confident nonsense.

Three ways to get a label out of a model, and what each one costs the surrounding system.

The numbers, and what they prove

The vendor announcement reports large multiples: on the order of two hundred times faster and four hundred times cheaper than a frontier model on the tasks it was built for. A separate write-up reports running sixty agents for a day in the tenths of a dollar, against tens of dollars for the same workload through a frontier API [Folkman 2026].

These numbers are plausible and they are not evidence of general superiority. They are measured on structured tasks, which is exactly the regime where the comparison is easiest to win: short outputs, a closed label set, no reasoning trace required. The fair reading is narrower and more useful. On tasks that fit the shape, the cost and latency gap is large enough to change architecture — because it becomes cheaper to decide on every request than to batch decisions overnight. On tasks that do not fit the shape, the numbers say nothing at all.

The architecture: a decision layer, not a model swap

The version of this that survives contact with production is not “replace the LLM”. It is a decision layer with an explicit contract:

Declared taxonomy

The label set lives in version control, with an owner and a change process. A decision model trained or prompted on a taxonomy nobody maintains will drift in ways no metric catches.

Calibration check

Before a threshold is used in production, the probability has to be checked on held-out data: reliability curves, not accuracy.

Abstention path

A first-class outcome meaning the taxonomy does not cover the input. It routes to a human or to a general model, and its rate is monitored.

Escalation budget

How many cases per day may leave the fast path, and who handles them. Without a budget the middle band grows until the fast path is decorative.

Recorded decision

Input hash, model version, label, probability, and the policy that was applied. This is what makes the layer auditable and priceable.

flowchart LR
request["Request"] --> decision["Decision model"]
decision -->|"p >= 0.9"| act["Act automatically"]
decision -->|"0.6 <= p < 0.9"| review["Human review queue"]
decision -->|"p < 0.6 or out of taxonomy"| generate["General model, generation"]
generate --> review
act --> record["Decision log"]
review --> record

Failure modes the fast path introduces

The economics improve so much that the temptation is to route everything. That is where the failure modes live, and they are boring in the way that production failures usually are.

Taxonomy drift. The label set was written for the input distribution of six months ago. New products, new abuse patterns and new customer segments create inputs the taxonomy cannot express, and a decision model with no abstention path will label them anyway. The symptom is not an error rate; it is a growing pile of confident misroutes that nobody notices until a customer escalates.

Label quality debt. The supervised signal is only as good as the people who produced it. If the training labels came from an under-specified rubric, the model will reproduce the ambiguity with better consistency and worse honesty.

Silent degradation. A general model that starts answering worse is visible in the transcripts. A decision model that drifts produces plausible labels and a slowly worsening distribution that only shows up when someone plots confusion against time. Nothing about the interface tells you.

The middle band becomes the whole system. If the abstention threshold is set badly, or the review capacity is smaller than the ambiguous volume, the fast path becomes a queue and the economics evaporate. The escalation budget is a capacity plan, not a configuration value.

What I would test before adopting

The test is not a benchmark. It is a small harness on your own traffic, run for two weeks:

Cost per decision

Not cost per token. Include retries, review and the engineering time the abstraction saves or costs.

Latency tail

The median improves trivially. Look at p99 against the user-facing budget, because a decision layer sits in front of everything.

Abstention rate

If it is zero, the threshold is lying. If it is huge, the taxonomy does not fit the workload.

Calibration on held-out data

Reliability curve by label, not a single accuracy number.

Agreement with humans

On a sample of reviewed cases, including the ones the model was confident about.

Where this leaves the chat model

Nothing in the argument says the general model disappears. It says the general model stops being the default. Generation remains the right tool for explanation, summarisation, translation, code and anything where the output space is genuinely open. What changes is the order of operations: decide first, generate only after the decision says generation is needed, and let a human own the band in between.

That is a less exciting claim than the benchmarks, and it is the one that survives an audit. The decision layer is where the cost, the latency and the accountability all land. A system that knows which of its model calls are decisions, and treats them as decisions, is cheaper, faster and — the part that matters — explainable to someone who was not there when it was built.