How to Choose an AI Model for Production
Saurav Jagdale
Technical Lead, Unico Connect
In this article
- Quick Answer
- Key Takeaways
- Step One, Build the Eval Set
- Step Two, Define What Passing Means
- Step Three, Run Candidates Across Price Tiers
- Step Four, Compute Cost Per Successful Task
- Step Five, Set the Latency Budget
- The Routing Decision
- What This Does Not Decide
- Keep the Decision Fresh
- Where These Numbers Come From
- Frequently Asked Questions
- Conclusion
Most teams choose a model the same way. Somebody reads a benchmark table, picks whichever name is highest, wires it in, and finds out six weeks later that it is too slow, too expensive, or wrong in a way the benchmark never tested. Then they swap models and repeat the process.
The problem is the question. Which model is best has no answer, because best is not a property of a model, it is a property of a model against a task at a price and a latency you can live with. A model that wins every public leaderboard can still be the wrong choice for extracting fields from your invoices.
This is the method we use to pick models for production systems. It is deliberately boring, it takes about a week, and it produces a decision you can defend and repeat. If you want a vendor by vendor verdict instead, read Claude versus GPT versus Gemini. This post is about how to run the choice yourself, which matters more, because the verdict expires and the method does not.
Quick Answer
Choose a model in five steps. Build an evaluation set of 50 to 200 real cases from your own traffic, with correct answers written down. Define pass and fail at the level of the whole task, not the token. Run three or four candidate models across price tiers against that set and record accuracy, cost and latency together. Compute cost per successful task, which is the only cost number that means anything, because a cheap model that fails a third of the time is not cheap. Then set a latency budget at the 95th percentile and eliminate anything that misses it. The model you keep is usually not the largest one, and it is almost never the one you would have picked from a leaderboard.
Key Takeaways
- A benchmark measures a model, an eval set measures your problem. Public benchmarks are useful for narrowing a shortlist to three or four candidates and useless for choosing between them. Your eval set is the only evidence that transfers to production.
- Cost per successful task is the real unit. Divide the total spend for a run by the number of tasks that actually passed. A model at a third of the price that fails twice as often costs more, and the token price never shows you that.
- Retries are part of the price. If your system retries on failure, the failure rate multiplies your bill and your latency at the same time. Cheap and unreliable compounds in both directions.
- Measure latency at the 95th percentile, never the average. Average latency hides the tail, and the tail is what users experience as the product being broken. Reasoning heavy models in particular have a long right tail.
- The decision expires. Model versions change under you, prices move, and your traffic drifts. Keep the eval set in version control and rerun it on a schedule, so the next choice costs a day rather than another six weeks.
Step One, Build the Eval Set
Everything depends on this and most teams skip it.
Take 50 to 200 real examples from actual traffic. Not synthetic examples, not the happy path from your demo, and not cases you wrote while thinking about the prompt. Real inputs, including the malformed ones, the ones in the wrong language, and the ones a user pasted from a spreadsheet.
For each one, write down what a correct output looks like. This is the tedious part and it is the part that creates the value, because you are encoding your actual standard rather than an abstract notion of quality. Where several outputs would be acceptable, record the rule that makes them acceptable instead of a single golden string.
Three properties matter. Hold some of it back. Keep 20 to 30 percent unseen while you iterate on prompts, or you will tune your way to a number that does not survive contact with new traffic. Include the cases you currently get wrong, because a model that fixes your known failures is worth more than one that is marginally better on cases you already handle. Keep it in version control next to the code, so it grows as production surfaces new failure modes.
A hundred labelled real cases beat every leaderboard in existence for this decision, and you can assemble them in a day.
Step Two, Define What Passing Means
Decide before you run anything, because deciding afterwards means grading on a curve you drew around the result you liked.
Grade at the level of the task. For extraction, all required fields correct or the task failed. For classification, the label matches. For a support reply, it answers the question, contains no invented fact, and follows the tone rules. Partial credit feels fairer and is worse, because production is usually pass or fail. An invoice with four of five fields right still needs a human.
Where the output is open ended and a human would have to judge it, you can use a strong model as the grader, but two conditions apply. Write the grading rubric explicitly rather than asking for a quality score out of ten, and check the grader against human judgement on a sample before you trust it on the rest. An unchecked model grader mostly measures how much the grader likes the candidate.
Keep a small set of hard rules separate from the graded score. Anything that must never happen, such as leaking another customer data, inventing a policy, or returning malformed JSON, is a gate rather than a metric. A model that trips a gate is eliminated regardless of how well it scores.
Model tiers, and the workload each one actually wins
| Tier | Wins when | Cost shape | The failure mode to plan for |
|---|---|---|---|
| Frontier model | Low volume, high stakes, or genuinely hard reasoning | Highest per call, often lowest per successful task at low volume | A long latency tail that misses an interactive budget |
| Mid tier model | The default for most production tasks once prompts are tuned | Balanced, and usually the best cost per successful task | Quietly good enough, so nobody ever tests whether a smaller tier would do |
| Small fast model | High volume, low stakes, narrow and well specified tasks | Lowest per call, worst per successful task if accuracy drops | Retries and human escalation erase the saving without showing on the invoice |
| Small model plus fallback | High volume where a validation gate can catch failures reliably | Close to the small model, plus the escalation rate times the large model | A weak gate, which passes bad output through and hides the problem |
Which should you choose
Tiers here are workload categories rather than named products. No vendor list prices appear in this post by design, since published rates move and differ across cloud platforms for the same model. Substitute your own measured accuracy, cost and latency.
Step Three, Run Candidates Across Price Tiers
Shortlist three or four models and make sure they span tiers rather than clustering at the top. A useful default is one frontier model, one mid tier model, and one small fast model. The point is not fairness to vendors, it is that you cannot find the cheapest model that clears your bar without testing models you expect to fail.
Hold everything else constant. Same prompt, same tools, same retrieval, same temperature settings, one variable at a time. Then record three numbers for every run and never look at any of them alone.
Accuracy is the pass rate against your eval set under the rules from step two. Cost is the total spend for the run, taken from actual token usage rather than an estimate. Latency is the full distribution, not a single number.
One caveat that catches people. Prompts are not portable. A prompt tuned over months against one model is not a fair test of another, and the honest fix is to give each candidate a short, equal tuning pass before you record its numbers. Otherwise you are measuring your prompt history rather than the model.
Step Four, Compute Cost Per Successful Task
Here is the arithmetic that changes decisions.
Cost per successful task equals total run cost divided by the number of tasks that passed. That is it. It folds the failure rate into the price, which is the thing token pricing structurally cannot show you.
Work through an illustrative example. Suppose a large model passes 94 percent of your eval set and a small model passes 71 percent, and the small model costs one fifth as much per call. On the token price the small model looks like an obvious win. Per successful task, the large model costs the run divided by 0.94 and the small model costs one fifth of the run divided by 0.71, so the small model is still cheaper, by roughly two and a half times rather than five.
Now add the thing teams forget. If failures are retried once, the small model runs 29 percent of its volume twice, which raises its real cost and its real latency together. If failures instead escalate to a human, price the human minutes, and the small model usually loses outright. If failures reach the user, the cost is a support ticket and some trust, which is real even though it does not appear on the invoice.
The rule that falls out. For high volume and low stakes work, a smaller model with a failure path is usually correct. For low volume and high stakes work, buy the accuracy, because the volume is too small for the token saving to matter and the failure is too expensive to absorb. Most teams get this backwards, running an expensive model on bulk classification and a cheap one on the thing that touches a customer.
Step Five, Set the Latency Budget
Work backwards from the interface.
Something a user waits for, staring at a spinner, needs to feel immediate. Something that streams can take longer, because the perceived wait is the time to first token rather than the time to completion. Something in a background job can take as long as the queue tolerates. Write the number down before you test, because a budget invented after the results is not a budget.
Then measure the 95th percentile, not the average. Averages conceal exactly the cases that make users think the product is broken, and reasoning heavy models have a genuinely long right tail. Measure under realistic concurrency too, since a model that is fast for one request at a time may not be fast when a hundred arrive together.
If a candidate misses the budget, it is out, however good its accuracy is. Accuracy you cannot deliver in time is not accuracy.
The Routing Decision
You do not have to pick one model, and past a certain scale you should not.
One model everywhere is the right starting point. It is simplest to reason about, cheapest to operate, and you should stay here until you have measured evidence to leave. Most systems never need to.
Tiered routing sends easy cases to a small model and hard ones to a large model, which works when you can classify difficulty cheaply and reliably. The trap is that the classifier is itself a model that can be wrong, so budget for its error rate rather than assuming it away.
Fallback on failure runs the small model first and escalates only when the output fails a validation gate. This is the pattern we reach for most often, because the escalation trigger is a real check on a real output rather than a guess about difficulty made in advance.
For how these compose once several models are actually in production, see multi model production AI.
What This Does Not Decide
Model choice is one variable and usually not the biggest one. If your answers are wrong because the model cannot see the right information, no model swap fixes that. Retrieval quality, prompt structure and tool design typically move accuracy further than moving up a tier does, and they are cheaper.
Test in this order. Fix retrieval, fix the prompt, fix the tools, and only then change the model. Teams that reverse it spend a lot of money discovering that the frontier model is also unable to guess a fact that was never in its context. Our guides to RAG versus fine tuning and fine tuning versus prompt engineering cover the adaptation side of that.
The number I ask every team for is cost per successful task, and almost nobody has it. They have a token price from a pricing page and an accuracy figure from a different run, never divided into each other. Once you compute it the argument usually ends in about ten minutes, because the model people were defending on price turns out to be the expensive one.
Saurav Jagdale, Technical Lead, Unico Connect
Keep the Decision Fresh
Treat this as a fixture rather than a one time project. Model versions change beneath you, prices move, and your own traffic drifts as users learn what the product can do.
Keep the eval set in version control, rerun it when a vendor ships a new version or when you change the prompt, and log accuracy, cost per successful task and 95th percentile latency in production continuously rather than only at selection time. Production is where drift shows up first, and instrumenting it is the subject of our guide to AI observability.
Because the eval set already exists, the second selection costs a day. That is the actual return on building it.
Where These Numbers Come From
There are no vendor prices in this post, deliberately. Published rates move, they differ across the Anthropic, Amazon Bedrock, Google Vertex AI and Microsoft platforms for the same model, and any table we printed would be wrong within a quarter. The 94 percent, 71 percent and one fifth figures in the cost section are illustrative inputs chosen to demonstrate the arithmetic, not measurements of any specific model, and the point of that section is that you substitute your own. For current vendor positioning see Claude versus GPT versus Gemini, and read list prices from the vendor pricing page on the day you need them. The method itself is our own delivery practice across AI systems we run in production.
Frequently Asked Questions
How do I choose which AI model to use in production?
Build an eval set of 50 to 200 real cases from your own traffic with correct answers recorded, define pass and fail at task level, run three or four candidates spanning price tiers, then compare on cost per successful task and 95th percentile latency rather than on token price or benchmark scores. Keep the eval set in version control so the next decision is cheap.
What is cost per successful task?
Total cost of a run divided by the number of tasks that passed your quality bar. It folds the failure rate into the price, which token pricing cannot do. A model at one fifth the price that fails twice as often is nowhere near one fifth the cost, and if failures are retried or escalated to a human it can be more expensive outright.
Should I use the biggest model available?
Usually not. Buy accuracy where volume is low and the cost of being wrong is high, and use a smaller model with a validation gate and a failure path where volume is high and stakes are low. Many teams do the opposite, spending on bulk internal classification while a cheaper model handles the customer facing path.
How many test cases do I need to choose a model?
Between 50 and 200 real cases is enough to separate candidates for most tasks. Below about 50 the differences fall inside noise. What matters more than count is that the cases come from real traffic, include your known failures and the malformed inputs, and that 20 to 30 percent is held back while you iterate on prompts.
Can I use benchmarks instead of building my own evaluation set?
Use them to narrow a shortlist to three or four candidates, then stop. Benchmarks measure general capability on public tasks, and your decision depends on one specific task, your prompt, your retrieval and your latency budget. They also leak into training data over time, which erodes what a high score means.
Should latency be measured as an average?
No. Measure the 95th percentile, because the average hides the slow tail that users experience as the product being broken. Measure under realistic concurrency as well, since throughput under load differs from single request speed. If you stream the response, measure time to first token separately, since that is what determines whether the wait feels acceptable.
How often should we revisit the model choice?
Rerun the eval set whenever a vendor ships a new model version, whenever you materially change the prompt or retrieval, and on a fixed schedule regardless, quarterly being a reasonable default. Also monitor accuracy, cost per successful task and latency in production continuously, since traffic drift shows up there before it shows up in a scheduled run.
What should we fix before changing models?
Retrieval, then the prompt, then tool design. If the model cannot see the information needed to answer, no upgrade fixes it, and those three are cheaper to change than a tier jump. Change the model last, once you know the remaining errors are genuinely capability limits.
Conclusion
Model selection looks like a research question and is really a measurement question. Assemble a hundred real examples, decide what correct means before you look at any output, run a few candidates across price tiers, and divide cost by successes. The answer usually turns out to be a smaller model than the team expected, wrapped in a validation gate that catches the cases it gets wrong.
The lasting artifact is the eval set. Build it once and every future model decision, prompt change and vendor announcement becomes a day of work instead of a debate.
If you want this run against your workload rather than described, see our AI development services and AI integration, or contact us.
