The demo was flawless. The agent answered everything, the committee applauded, the budget was approved. Six months later the pilot is frozen: inconsistent answers, a couple of embarrassing mistakes in front of a client, and nobody can say whether today's version is better or worse than last month's. The model didn't fail. Something more basic did: nobody was measuring.
The number in this article's title is one we use in our own presentations: roughly 73% of AI projects never reach production. And after several years building agents for Mexican companies, at SISCON the pattern is clear: projects almost never die for lack of technology. They die for lack of evidence. With no systematic way to measure quality, every change is a gamble, every mistake is a crisis, and every status meeting is a debate between opinions.
The demo is not the product
Every demo is built on happy paths: the ten questions the team already knows work well. Production is something else — it is the full distribution of what your real users ask, with typos, missing context, edge cases and the occasional bad actor. An AI system that looks brilliant across twenty cases can run at 60% accuracy across a thousand real ones, and nobody will know until the expensive mistake shows up.
Traditional software systems have a safety net we take for granted: tests. Nobody ships a billing system without automated tests. Yet with AI, for some reason, companies accept deploying probabilistic systems — which by definition get things wrong sometimes — with no equivalent test suite at all. It is exactly backwards: the less deterministic a system is, the more it needs systematic measurement.
What "measuring" an AI system actually means
Measuring is not "we played with it for a while and it looks fine". The discipline — which Chip Huyen systematizes in her book AI Engineering (O'Reilly, 2025), the current reference in the field — rests on three pieces:
- Golden sets. A curated set of real cases with the expected correct answer, validated by your business team — not by the vendor. A hundred well-chosen cases are worth more than a thousand invented ones. This is your yardstick: fixed, versioned and shared.
- Automated judges (LLM-as-judge). An evaluator model, calibrated against human judgment, that grades every system response: is it correct? is it grounded in the source documents? did it respect the format and tone? This lets you evaluate hundreds of cases in minutes instead of weeks of manual review.
- Regression tests. Every change — a tweaked prompt, an updated model, a new document in the knowledge base — runs against the full golden set before it reaches production. If quality drops below the agreed threshold, the change doesn't ship. Just like a CI/CD pipeline, but for behavior.
The cost of not measuring is silent (until it isn't)
What makes running AI without evals treacherous is that regressions give no warning. You change one line of a prompt to fix a case and unknowingly break twelve others. Your vendor updates the model behind the API — which happens several times a year — and your system's behavior shifts overnight without anyone touching anything. An agent that answered well for three months starts inventing figures in an edge case that was never in the manual tests.
Without measurement, each of these events is discovered in production, in front of a user or a client. And at that point the cost is no longer technical: it is trust. A single visible mistake in front of the executive committee can freeze an entire AI program — not because the system is bad, but because nobody can prove with data that it is good.
Evaluation is not a project: it is a discipline
The common mistake is treating evaluation as a one-off deliverable: "we ran the initial benchmark, that's done". Valuable evals are the ones that live in your pipeline: they run on every change, they run overnight against the day's traffic, and they feed a dashboard where business and technology see the same truth. That is how you answer in seconds the question that kills pilots: "is it getting better or worse?"
That same suite is also what lets you make bigger decisions on evidence rather than marketing: is it worth switching models? does the open-source version match the commercial API's quality at a fraction of the cost? would fine-tuning improve things enough to justify itself? Without evals, those questions get answered by gut feeling. With evals, they get answered with numbers from your cases and your data.
Where to start this week
- Gather 50–100 real cases from your operation (tickets, emails, queries) with the answer you consider correct. That is your first golden set.
- Run your current system against them and write down the result. That number — the one you don't have today — is your baseline.
- Define the deployment threshold: no change reaches production if the score drops below X%. Write it down and communicate it.
- Automate the run in your pipeline with tools like Langfuse or promptfoo, so measuring doesn't depend on someone remembering.
- Review the failures every week and turn new real errors into golden set cases. The set grows with your operation.
This is exactly the kind of work we formalized in our AI Evaluation and Auditing service: building the suite with your cases, calibrating automated judges against your criteria and leaving it running in your pipeline — as part of the harness that makes an agent reliable. Because the conclusion from years of watching pilots die is simple: AI that isn't measured can't be defended. And what can't be defended doesn't survive the committee.