Answers from the documentation; checking which model is on duty
Hi - I answer from the OpenSmartRoute documentation: routing, the API, plans and quotas, self-hosting. Ask away, or open a support ticket if you need a person.
Grounded in the docs - follow a source before acting on it.
New Decision Models for Automated Evaluation - OpenSmartRoute
New decision models like Jev and OpenAI Decisions offer cheaper alternatives to LLM judges for automated testing. The article compares their cost, performance, and calibration on a leasing agent dataset.
Key points
OpenAI Decisions achieved the highest F1 score at 0.87 in the evaluation.
Perplexity Decider cost 0.82 cents per 100 eval runs compared to OpenAI's 3.72 cents.
Default decision thresholds of 0.50 often miss critical failures in real data.
Calibration adjustments can fix probability scores that do not match reality.
Why it matters: Teams save money and improve quality by tuning thresholds and comparing costs before buying models.
By OpenSmartRoute editorial · written through the router by writer-small
From Hamel Husain - “How To Use Decision Models for Evals”
Decision models have appeared recently in the automated testing space. TypeSafe released Jev, Perplexity launched Decider, Liquid released d1 and d1-ultra, Cloudflare released Clef, and OpenAI released the Decisions API. These models promise fast and cheap classification tasks. They offer a low-cost alternative to large language model judges for automated evaluation. A team must check the cost, performance, and latency requirements of their use case before choosing one. The default decision threshold of 0.50 is often used without testing. This can leave performance on the table. The post will show how to tune thresholds to get the best results from decision models. It will then compare cost and performance to help assess which model fits the use case. Finally, it explores how to interpret model scores and how calibration can make them more useful as confidence estimates.
Testing Decision Models on a Leasing Agent Dataset
The team used 100 traces for this test. A trace is a log of a conversation and the actions taken. The task is to identify traces that contain an error. Examples of errors include a failed booking or a missed handoff to a human. The same dataset and broad failure-finding task were used in a prior post on automated evals. Each decision model returns a failure probability for each trace. The models tested were Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna). Each model received the same evaluation criteria. The appendix shows the prompt and the structured version used for Jev. Performance is measured by comparing predictions with traces that people have reviewed and labeled as passing or failing. This measures how well the model catches failures while avoiding false alarms. Cost is calculated from token usage and the provider’s prices. Latency is left aside because it depends on factors outside the experiment’s control. The task is a binary pass/fail eval. This is recommended because it encourages clear criteria and is easier to align with human preferences. Each decision model returns a score from 0 to 1. This score estimates how likely a trace is to contain a failure. The APIs turn this score into a decision using a default decision threshold of 0.50. Traces with scores at or above the threshold are flagged as failures. Using this default without checking it against the data can give subpar results. A model could assign every failure a score of 0.40 and every passing trace a score of 0.10. Even though this model classifies traces perfectly at a threshold of 0.30, the default threshold of 0.50 would catch nothing.
Deepseek released V4.1 Flash 1017 which beats larger rivals in coding and agent tasks. The model also cuts per-task costs significantly while producing shorter outputs.
TypeSafe released Jev, a closed decision model, and 20 alternatives appeared within weeks. This guide compares their sizes, licenses and benchmarks.
Comparing Cost and Performance Across Models
In practice, scores for failures and passing traces will overlap. Lowering the threshold catches more failures but can also raise more false alarms. Choosing a threshold means balancing precision and recall. Precision is the share of flagged traces that contain a known failure. Recall is the share of known failures that the model catches. The right tradeoff depends on business goals. For example, criminal trials set a high bar for conviction to avoid convicting innocent people, emphasizing precision. Cancer screening may emphasize recall, accepting more false alarms to miss fewer cases. For this comparison, the team gave precision and recall equal weight using the F1 score. F1 is the harmonic mean of precision and recall. F1 ranges from 0 to 1, and higher is better. The curves show how each model scores failures (orange) and passing traces (blue). Taller parts of a curve mean more scores from that class fall in that range. Scores at or above the dashed threshold line are flagged. The chart shows how precision and recall change at different thresholds. The shaded region marks where both reach at least 80%. Perplexity, Liquid d1-ultra, and OpenAI Decisions meet the requirements. The team restricted tuning to thresholds that meet both 80% minimums on the training data whenever possible. The other models were left unconstrained to illustrate the effect of tuning. With labeled traces and saved model scores, a coding agent can tune the thresholds. An example prompt asks the agent to tune a decision threshold to maximize F1 while requiring at least 80% precision and 80% recall. The agent uses stratified five-fold cross-validation. It chooses thresholds on the training examples and measures performance on the remaining examples. If comparing models, it uses the same splits for each. It only selects thresholds that meet both minimums on the training examples. If none qualifies, it reports that rather than relaxing either requirement. It reports validation precision, recall, and F1, along with any splits where no threshold qualified or validation performance fell below the minimums. It compares with the default threshold of 0.50. It reuses the saved scores without making new model calls.
F1 tuning changed each model’s performance. The candidate models gained less from F1 tuning than Liquid d1 and Clef Flash, which fell below the minimums. Even a small shift in precision and recall can affect business costs or the user experience. In practice, it is unlikely that precision and recall are equally important. The F-beta score lets you give precision more weight with beta below 1, or recall more weight with beta above 1. Optimizing a different metric could lead to different thresholds and results. The chart compares each model’s F1 (after tuning) against cost. The dashed line shows the Pareto frontier among the candidates. For any given F1 score, it identifies the cheapest candidate that meets or exceeds that score. Perplexity was the cheapest of these candidates at 0.82 cents per 100 eval runs, with an F1 of 0.79. OpenAI Decisions reached an F1 of 0.87 at 3.72 cents per 100 eval runs, about 4.5 times the cost. Beyond pass/fail evals, a decision model’s score can be used as a confidence estimate. For a score of 0.80 to mean an 80% chance of failure, roughly 80% of traces with similar scores should contain failures. The chart shows each model’s probability calibration. This measures how closely its predicted probabilities match observed outcomes. Each point represents about 20 traces with similar scores. A point on the diagonal means the predicted probability matches the failure rate in that group. Above the line, the model underestimates the risk of failure. Below it, the model overestimates that risk. If the predicted probabilities do not match the observed outcomes, they can be adjusted using another model. One method is Platt scaling. This fits a simple curve to make that adjustment. Perplexity, Liquid d1, and Clef Flash move closer to the diagonal after calibration. For example, one group of Clef Flash traces had an average predicted failure probability of about 8%, but 60% contained failures. Jev and Liquid d1-ultra were already well calibrated on this dataset and needed little adjustment. Platt scaling isn’t always the best fit. For example, OpenAI’s probabilities got slightly worse with the version of Platt scaling used. A coding agent can perform calibration with a prompt. The agent tests whether Platt scaling improves each model’s probabilities. It uses stratified five-fold cross-validation with the same splits for every model. It fits the calibration mapping on each training split and applies it to the remaining traces. It fits on log-odds, clipping probabilities away from 0 and 1 first. It plots predicted probabilities against observed failure rates before and after calibration. It reports Brier scores (lower is better) and explains which models improved or got worse. It reuses the saved scores without making new model calls.
The Danger of Default Decision Thresholds
The team kept the eval task broad to make the differences between models easier to see. In practice, it is better to build a separate judge for each failure mode. With a narrower task and more specific guidance, the performance gaps between models will shrink. The team recommends starting with a larger model that has acceptable performance on a task. Then they try smaller models to see if they can match the larger model’s performance for less cost. Prompt tuning can also be helpful if the team has high-quality labels. The team discusses how to craft a good judge prompt in another post. Once the judge’s prompt is satisfied, the team should set minimum precision and recall requirements for the use case. They should rule out models that cannot meet these. They should tune each model’s decision threshold before comparing performance. They should choose a precision-recall tradeoff that fits business goals. They should be intentional and not just use the F1 score. If they want to use scores as confidence estimates, they should check whether the probabilities match observed outcomes. Calibration can help, but they need to measure whether it improves them. They should weigh performance against cost and latency for their workload. This process is iterative. They may improve the prompt, then need to tune the threshold again or revisit calibration. The team focused on threshold tuning and calibration because they have seen many people use decision models off the shelf without considering these factors. They also wondered how decision models stack up to LLMs. OpenAI Decisions uses GPT-6 Luna, which delivered the highest F1 at the highest cost. This suggests that harder eval tasks may justify paying more for an LLM. In a broader comparison, Randy Olson found that decision models could match LLMs on classification and routing at lower cost. LLMs pulled ahead on tasks requiring knowledge and reasoning. None of the ideas discussed in the post are new. Choosing a precision-recall tradeoff, tuning thresholds, and comparing performance vs. cost are part of evaluating any classifier. To learn more about these fundamentals, the team recommends Machine Learning with Python Cookbook, 2nd Edition by Kyle Gallatin and Chris Albon. Finally, if the reader is a data scientist or ML engineer wondering where they belong in the AI age, these are skills they already have. It is time to get revenge.
Why it Matters
For this particular dataset, OpenAI had breakout performance at a higher cost. Perplexity and Liquid d1-ultra offer lower-cost options with lower performance. The team observed that model performance varies substantially across tasks and datasets. This means it is important to measure things yourself. Once the judge’s prompt is satisfied, set minimum precision and recall requirements for the use case. Rule out models that cannot meet these. Tune each model’s decision threshold before comparing performance. Choose a precision-recall tradeoff that fits business goals. Be intentional, do not just use F1 score because this post did! If you want to use scores as confidence estimates, check whether the probabilities match observed outcomes. Calibration can help, but you need to measure whether it improves them. Weigh performance against cost and latency for your workload. This process is iterative. You may improve the prompt, then need to tune the threshold again or revisit calibration. In this post, the team focused on threshold tuning and calibration because they have seen many people use decision models off the shelf without considering these factors. You might also be wondering how decision models stack up to LLMs. OpenAI Decisions uses GPT-6 Luna, which delivered the highest F1 at the highest cost. This suggests that harder eval tasks may justify paying more for an LLM. In a broader comparison, Randy Olson found that decision models could match LLMs on classification and routing at lower cost. LLMs pulled ahead on tasks requiring knowledge and reasoning. None of the ideas discussed in this post are new. Choosing a precision-recall tradeoff, tuning thresholds, and comparing performance vs. cost are part of evaluating any classifier. To learn more about these fundamentals, the team recommends Machine Learning with Python Cookbook, 2nd Edition by Kyle Gallatin and Chris Albon. Finally, if you are a data scientist or ML engineer wondering where you belong in the AI age, these are skills you already have. It is time to get revenge.
How OpenSmartRoute helps
OpenSmartRoute is an open-source router for AI requests. It has a hosted platform at https://opensmartroute.ai. It sends each request to the best-fit model, agent, tool or skill from a catalogue the team defines. It scores every candidate on quality, cost, speed and safety. The team sets the weights per request. Hard rules pin a request: text with personal data stays on an on-premises model, a region or a cost cap is never crossed. It learns from outcomes, so a model that answers well gets more traffic and one that fails gets less. A new model is one catalogue entry and competes on the next request; nothing else changes in the app. It works with any OpenAI-compatible provider, with open-weight models served locally, with MCP tools and A2A agents. A savings ledger shows what each routed request cost next to what the most expensive model would have cost. An input guard spots prompt injection and personal data before a request leaves. 'osr eval' measures routing accuracy on the team's own prompts and can fail a build when it drops. The hosted platform keeps a models catalogue with prices and public rankings built from real traffic. For decision models, a team can add them to the catalogue and let the router choose the best option based on cost and performance. The router can enforce a minimum F1 score or a maximum cost per evaluation. If a model fails to meet the threshold, the router can route the request to a more expensive LLM judge instead. The savings ledger will then show the cost difference between the decision model and the fallback judge. This helps the team decide if the tradeoff is worth it.
What to Do
Start with a larger model that has acceptable performance on a task. Then try smaller models to see if they can match the larger model’s performance for less cost. Prompt tuning can also be helpful if you have high-quality labels. Once you are satisfied with your judge’s prompt, set minimum precision and recall requirements for your use case. Rule out models that cannot meet these. Tune each model’s decision threshold before comparing performance. Choose a precision-recall tradeoff that fits your business goals. Be intentional, do not just use F1 score because this post did! If you want to use scores as confidence estimates, check whether the probabilities match observed outcomes. Calibration can help, but you need to measure whether it improves them. Weigh performance against cost and latency for your workload. This process is iterative. You may improve the prompt, then need to tune the threshold again or revisit calibration. In this post, the team focused on threshold tuning and calibration because they have seen many people use decision models off the shelf without considering these factors. You might also be wondering how decision models stack up to LLMs. OpenAI Decisions uses GPT-6 Luna, which delivered the highest F1 at the highest cost. This suggests that harder eval tasks may justify paying more for an LLM. In a broader comparison, Randy Olson found that decision models could match LLMs on classification and routing at lower cost. LLMs pulled ahead on tasks requiring knowledge and reasoning. None of the ideas discussed in this post are new. Choosing a precision-recall tradeoff, tuning thresholds, and comparing performance vs. cost are part of evaluating any classifier. To learn more about these fundamentals, the team recommends Machine Learning with Python Cookbook, 2nd Edition by Kyle Gallatin and Chris Albon. Finally, if you are a data scientist or ML engineer wondering where you belong in the AI age, these are skills you already have. It is time to get revenge.
Researchers found Large Language Models frequently answer using search snippets without reading the actual web pages. This behavior leads to outdated answers and misleading citations across multiple model types.
