In September TypeSafe released Jev, a “decision model.” You give it some text and a typed question (yes/no, pick one, or a 1–5 score), and instead of writing a reply it gives back a probability for each option. It’s fast and cheap: in my run it cost 40 to 50 times less per decision than GPT-6.1 Sol. Within six weeks there were about a dozen models on OpenRouter that work the same way, from Perplexity, Microsoft, Baseten, Liquid and others.

Every public test I found compared Jev to much bigger frontier models. It matches them when the answer is written in the text and falls about ten points behind when it has to reason. That mostly tells you big models are better at reasoning. What I wanted to know was whether the decision part does anything at all, or whether these are normal LLMs with a different pricing model.

Two of them say what they’re built on. Baseten’s Elephant 1 and Microsoft-Decision-1 are both trained on top of Qwen3.5-9B. So I could run plain Qwen3.5-9B on the same questions and compare it directly to the two decision models made from it. Same model underneath, the only difference is the extra training.

What I expected

I didn’t know. The public tests made Jev look like a cheap fine-tune, and a preprint by Sun et al. had shown it changing answers when you rename the yes/no labels, which is how LLMs behave. If the decision versions of Qwen were no better than plain Qwen, “decision model” would just be a pricing idea. If they were better, the next question was which one is best.

I ran eight models plus a simple baseline on 450 questions I generated myself, five times each. After seeing the first results I added one more run: plain Qwen3.5-9B allowed to think before answering, and the one-token version under two other prompts. Everything went through OpenRouter from one machine on October 29 and 30.

Group	Model	Why it’s here
Decision model	TypeSafe Jev 1.14	The original. Base not disclosed.
	Perplexity Decider V1.2	27B, base not disclosed. Compared with Qwen3.5-27B.
	Baseten Elephant 1	Built on Qwen3.5-9B.
	Microsoft-Decision-1	Built on Qwen3.5-9B.
	Liquid d1-ultra	Probably not built on Qwen.
LLM, one token	Qwen3.5-9B	The main comparison. Answers with one token; probabilities come from the logprobs.
	Qwen3.5-27B	Compared with Decider.
LLM, thinks first	GPT-6.1 Sol	The upper bound.
	Qwen3.5-9B, thinking (added later)	Same model as above, allowed up to 12,000 tokens of thinking before the same one-token answer.
Baseline	No model	Keyword matching on tier 1, most common answer on tier 2.

The questions

I didn’t use public benchmarks because these models have probably seen them. Every question is generated by code, so the right answer is known for sure. They come in three tiers based on where the answer is:

1. Tier 1, in the text (200). Fake support tickets, invoices, memos, logs and contracts with the answer written in somewhere, plus filler. “Does this ticket mention a refund?”
2. Tier 2, needs reasoning over the text (200). Everything you need is there but you have to put it together: meeting-room conflicts, expense reports checked against five rules, putting events in order, small logic puzzles. A solver makes the answer and a separate checker confirms it.
3. Tier 3, outside knowledge (50). Trivia written by Claude Fable 5.1 and checked by GPT-6 Astra. This one is weaker and mostly there for comparison.

Each tier is about 40% yes/no, 40% pick-one (4–8 options) and 20% score from 1 to 5. Every model gets the same text and question.

The decision training mostly teaches the format, and Jev isn’t the best one. Making Qwen3.5-9B answer in one token drops it by 46 points on tier 2. The two decision models built on it win back 15 and 10 of those points. Letting the same Qwen think first gets it to 0.992, higher than every decision model here, but it costs about 50 times more and takes 60 to 90 times longer. On tiers 1 and 2, Perplexity’s Decider is the best decision model on accuracy, consistency and calibration, and also the cheapest. Jev is the weakest at reasoning, changes its answer on one in eight tier-2 questions between runs, and costs three times what Decider does.

Tier 1: everyone gets it

When the answer is in the text, every model scored 0.94 or higher, and Decider and Microsoft-Decision-1 got all 1,000 right. The keyword baseline got 0.54, so the questions weren’t trivial. This tier doesn’t separate the models, but it does show that for simple lookups a decision model is as good as anything else at a fraction of the price.

Tier 2: where they split

GPT-6.1 Sol got everything right at $1.49 per thousand and 2.4 seconds per question. Thinking Qwen3.5-9B was one point behind at 0.99, at half the price but about ten times slower. Then the decision models, from Decider at 0.84 down to Jev at 0.61. Last are the two plain Qwen models answering in one token, at 0.53 and 0.52. On just the yes/no questions in this tier, plain Qwen3.5-9B gets 0.59, where guessing would get 0.50.

The 27B model did no better than the 9B one here. Answering in one token without thinking seems to waste whatever the bigger model would have added. Decider, compared against that 27B, is 31 points higher, which gets it about two thirds of the way to what the 9B reaches when it can think.

Decision model vs the model it’s built on

This is the main comparison. For each base model there are three ways to answer: the decision model built on it, the base model forced to answer in one token, and the base model allowed to think first.

Against the one-token base, the decision models do better. On tier 2, Elephant 1 is 15 points higher and Microsoft-Decision-1 is 10 points higher, and both differences are clearly real. Their probabilities are also more accurate (Elephant’s Brier score is 0.43 vs 0.60).

Against the base when it thinks, they do much worse. Thinking Qwen gets 0.992, so Elephant is 31 points behind it (CI 25 to 37) and Microsoft-Decision-1 is 36 behind (CI 30 to 43). So answering in one token costs the base 46 points, and the decision training only wins back 15 and 10 of them. It doesn’t bring back the reasoning. What you get instead is price and speed: $0.016 and $0.014 per thousand decisions instead of $0.77, and 261 and 368 ms instead of 23.5 seconds. Thinking Qwen also got stuck in a loop on 3.8% of questions and never answered; those count as wrong.

The two 9B decision models aren’t the same either. Microsoft-Decision-1 changed its answer between runs on 4.5% of tier-2 questions, and its ECE was a bit worse than plain Qwen’s even though its Brier score was better. Elephant never changed its answer.

Consistency and renamed labels

Model	Tier 2 flips	Label-swap changes	Tier 2 Brier
Decider V1.2	0	0.0%	0.23
Elephant 1	0	0.0%	0.43
d1-ultra	13.5%	0.4%	0.38
Microsoft-Decision-1	4.5%	2.4%	0.52
Jev 1.14	12.5%	8.8%	0.43
Qwen3.5-27B, one token	0	6.0%	0.58
Qwen3.5-9B, one token	1.5%	13.6%	0.60
GPT-6.1 Sol	0	0.0%	—

Flips: share of questions where the 5 runs didn’t all agree. Label-swap changes: share of yes/no answers that changed when the labels were renamed. GPT reports its own confidence, so it has no Brier score here.

Jev is the least consistent decision model when you count both columns. It changed its answer on one in eight tier-2 questions between runs, and on about one in eleven yes/no questions when I renamed the labels. d1-ultra changes answers between runs even more often, but it ignores the labels. Reacting to label names is what plain LLMs do: Qwen3.5-9B is the worst here at 13.6%, and the two decision models built on it get that down to 0% and 2.4%. So the decision training does fix this. Jev just fixes it less.

One result didn’t hold up. Sun et al. found Jev’s AUC dropping from 0.81 to 0.58 when labels were swapped. On my questions it only went from 0.945 to 0.940.

Routing

If a model’s confidence is reliable, you can let it answer when it’s sure and send everything else to a stronger model.

With a cutoff of 0.96, Decider sends 42% of questions to GPT and gets 0.98 for $0.68 per thousand. GPT alone gets 1.00 for $1.49, so that’s less than half the cost for two points. Jev has to send 60% to GPT to reach 0.98, which costs $0.97.

All the questions, results and code are in the repo folder. The raw responses for every call (about 180 MB) are on my machine if anyone wants them.

Decision models are real, but they’re less than they look. Compared with the same LLM forced to answer in one token, the extra training helps: 10 to 15 points on questions that need reasoning, better probabilities, and far less sensitivity to how the options are labelled. Compared with the same LLM allowed to think, they fall well short. The training teaches the model to answer in the right format, not to reason.

Accuracy isn’t the whole comparison though. On tier 2 a decision model answered in about a third of a second for one to three cents per thousand decisions. The same 9B thinking took about 23 seconds and 77 cents per thousand, and GPT-6.1 Sol took 2.4 seconds and $1.49. So the decision models are 15 to 40 points behind, but tens of times cheaper and many times faster. Whether that’s worth it depends on what you’re deciding.

The products also differ a lot. Decider was the best on almost everything and the cheapest. Jev, the one that started the category, was the weakest at reasoning and the least consistent.

What I’d use

• When the answer is written in the text, use a decision model. Routing, tagging, checking whether something is mentioned: every model got 0.94 or better here, and the decision models do it in a third of a second for almost nothing.
• When it needs some reasoning and has to be fast, use Decider and let it defer. It answers when it’s confident and passes the rest to a stronger model. With a cutoff of 0.96 it handled 58% of questions itself and got 0.98 overall for less than half of GPT’s cost.
• When it needs reasoning and getting it right matters more than speed, use a model that thinks. GPT-6.1 Sol if you need the answer in a couple of seconds, or a small thinking model like Qwen3.5-9B for half the price if you can wait 20-some seconds. Decider could defer to that instead of GPT too, though I didn’t test it.

Whichever you use, set the cutoff on a few hundred of your own labelled examples. A model’s confidence on my questions won’t match its confidence on yours.

Two things I’d try with more time. Baseten’s Mammoth 1 is built on Qwen3.5-27B, which would give a clean 27B comparison. And nobody has made a decision model on anything bigger than 27B, so the only way to see if it helps at 70B is to train one. Amazon (with its open Strands Decider) and Baseten have both published how they trained theirs.