Your Pilot Is Not Failing on the Model

Last Wednesday I sat in a room in Amsterdam with a few dozen people who build AI systems for a living. Lena Shakurova of ParsLabs opened by asking how many of them would trust their own LLM outputs. A handful of hands went up.

Her title slide said the rest.

That is the honest state of the field, and it is the same answer I get from enterprise teams sitting on a stalled pilot with a budget question to answer. The demo worked. The production system does not. The instinct is to blame the model, so the team swaps in a newer one, rewrites the prompt, and runs the demo again.

The model is rarely the problem. The problem is that nobody wrote down what good means, so nobody can prove the new version is better than the last one.

Unreliable is a different problem at enterprise scale

Shakurova started where any risk conversation should start, with the fact that the same input does not produce the same output. Everything downstream follows from that.

She split the consequences by likelihood rather than lumping them together, which is the useful move. Hallucinations and inconsistent output arrive free with the technology, so every team has them. Jailbreaks, injection attacks, and data leaks are far less common, and when they land they are critical. A risk register that treats those four as one category will underfund the second pair.

Her line on what that means for a product was the sharpest of the night. Working most of the time is not good enough. For a consumer novelty, most of the time is a feature. In finance, healthcare, or any regulated operation, it is the reason legal will not sign.

The ladder every team climbs, and the step they stop on

Most teams follow the same progression. They start in the playground, checking whether the thing works at all. That is vibe testing, and it is fine as a sanity check. It fails because you test different things in different ways every time, and you forget to re-test what already worked, so regressions ship silently.

Next comes structured manual testing, usually a spreadsheet the team walks before each release. Better, and unaffordable. It costs human hours on every change.

Then automated evaluation, whether that is a local script, a platform, or checks wired into the pipeline on every pull request. This is where most well-run programs stop and declare the problem solved.

It is not solved, and her summary slide said why in one line.

Automation does not replace the human. It extends a judgment a person made, across a volume no person could read. Take the person out and you have a system that reliably measures the wrong thing.

The step before the tooling

Which puts the real first move outside engineering entirely. Before any test set exists, someone has to write down what an acceptable answer looks like in your domain.

She was blunt that this cannot be delegated to the system being tested. Ask the model what matters and you have automated your own opinion out of the process.

The method is unglamorous. Run a workshop with the team. Read production logs by hand. Bring in the subject matter experts. Mine the standard operating procedures you already have. The output is a plain-language checklist, written by people who would be accountable for a bad answer, saying what the system must and must not do.

Buyers skip this because it looks like the least technical step in the program. It is the step every number downstream inherits its meaning from.

The loop

Three stages, and they repeat rather than finish.

Experimentation. Define good, build a test set that represents the real use case, build metrics, then run experiments until the regression tests pass. Her test set is deliberately three-part: the happy paths that must always work, the edge cases that rarely happen, and adversarial inputs built to test jailbreak resistance and refusal.

Live monitoring. Log traces, which she specified as input, output, tool calls, retrieval steps, latency, and token count. Log user feedback, both thumbs and free text. Log metadata including session, timestamp, model version, and prompt version. Then set real-time alerts on the outliers that matter: latency spikes, cost spikes, and flagged hallucinations. The point of the alert is that the engineering team hears about a broken production system before the customer tells them.

Post-production evaluation. Auto-label the logs to surface candidates, because nobody reads all of them. Look for knowledge gaps where the agent said it did not know, and for negative sentiment where a user got frustrated. Then a human works the shortlist.

Her procedure for that last stage is the part worth copying exactly. Select the problematic traces. Extract the conversation history up to the point where it went wrong. Write evaluators that check whether the conversation passes criteria, either as unit tests or with a model as judge. Then maintain a golden dataset of expected question and answer pairs that must never regress.

That last phrase is the one to bring to your own team. Most enterprises have no such dataset, which means they have no floor.

One score hides the failure

Her metrics slide split into four families, and the split matters more than any individual metric. Retrieval quality asks whether the system found the right source material. Tool and function calling asks whether it acted on the right system with well-formed arguments. Generation quality asks whether the answer itself was any good. Conversation quality asks whether the whole exchange reached its goal, measured by task completion, context retention, and steps to resolution.

A single aggregate score cannot tell you which of those four broke. Teams that report one number spend the next sprint guessing.

Two constraints that bite harder in Europe

Two audience questions surfaced things a consumer-scale team can wave off and a regulated European operator cannot.

Redact before you log, not after. Asked about personal data, her answer was to strip identifiable information at the logging boundary so it never enters the evaluation stack at all. Worth pairing with her own logging list, which includes user and session identifiers. Those should be pseudonymous keys rather than anything that resolves to a person, because an evaluation corpus is a year of conversation history sitting outside your primary data controls. Getting that boundary right on day one costs a design decision. Getting it wrong costs a retrospective purge across every log store you own.

Using a model to judge a model inherits the problem. LLM-as-judge is the standard way to score generation quality at scale, and it puts a non-deterministic evaluator in charge of grading a non-deterministic system. Her answer was that the judge has to be aligned against human-labeled examples before you trust its verdicts, which means building two sets rather than one. Anyone who shows you an evaluation dashboard should be able to say how their judge was calibrated and when it was last re-checked. If they cannot, the dashboard is decoration.

The trap on the other side

The failure mode after buying the tooling is treating the purchase as the finish.

An evaluation suite that never changes measures last quarter’s product. Every real failure found in production is supposed to become a new test case, and the standing routine has to include a random sample of conversations the automated labeling never flagged, because that is the only way to catch what your labels are blind to.

Which brings the whole thing back to the inversion she closed on, borrowed from a practice most engineering leaders already run. First update your test set and your judges. Only then update your prompt. Do it the other way around and, in her words, you were prompting for the sake of prompting. You changed something, the demo felt better, and you have no evidence either way.

What to do on Monday

None of this requires new procurement.

1. Write the quality checklist by hand with the domain experts who own the outcome. One page, plain language.

2. Build the test set in three parts: happy paths, edge cases, and adversarial inputs.

3. Measure four layers rather than one score. Retrieval, tool calling, generation quality, and conversation outcome.

4. Stand up a golden dataset of question and answer pairs that must never regress, and treat a regression as a release blocker.

5. Log with redaction at the boundary, keep identifiers pseudonymous, and alert on latency, cost, and flagged failures.

6. Put a recurring human review on the calendar, including an unfiltered random sample, and name the person who owns it.

The first four are engineering. The last two are operating model, and they are where most programs quietly stop.

Shakurova maintains a public library of evaluation tooling, sorted into no-code tools, Python libraries, safety and moderation guardrails, and voice agent evaluation, which she puts at sixty or more entries. It is email-gated and worth the address. Her worked references on logging and on evaluating multi-turn conversations both point at Langfuse documentation, which is a reasonable starting point whatever platform you land on.

The LLM eval library is published on her own site and also via ParsLabs.

Langfuse log levels, for the trace verbosity and alerting pattern.

Evaluating multi-turn conversations, the cookbook behind the golden dataset method.

Where AtomDigit comes in

Evaluation is not a phase of a build. It is a capability an organization either has or does not, which is why it belongs in the Modernize stage of our ATOM method, Assess, Tailor, Orchestrate, Modernize, and inside the AI Center of Excellence operating model rather than in a project plan that ends at launch.

If you have systems in production and no answer to how you would prove this release is better than the last one, that is the gap worth closing first.