Evaluating AI Output Quality in Production Software

“Looks good” is not a release criterion

AI prototypes are unusually easy to overestimate. The team chooses familiar examples, retries disappointing responses and remembers the impressive outputs. A fluent answer also feels more correct than a hesitant one, even when both contain the same factual error.

Production evaluation begins by describing failure in business terms. A support summariser may omit a customer's stated constraint. An invoice extractor may transpose net and gross values. A retrieval assistant may answer accurately from a document the user was not authorised to see. These are different failures and need different measurements.

Do not start with a generic target such as “90% accurate”. Decide which errors matter, their consequences and the population over which the claim is intended to hold. Ninety-nine per cent field accuracy can still be unacceptable if the remaining one per cent contains bank details, or if every error clusters in one supplier's document format.

Evaluate the application, not the model in isolation

Public model benchmarks say little about whether a particular business workflow is safe or useful. The deployed behaviour is produced by a system: model version, prompt, retrieved context, tool definitions, structured-output schema, validation rules, reference data and user interface. Evaluate that system end to end.

This matters because a better model can make the application worse. It may call tools more readily, produce longer answers that hide omissions or follow a prompt format differently. Likewise, replacing a model may appear to fix retrieval when it merely writes a persuasive answer around poor evidence.

Keep component diagnostics as well as end-to-end measures. For retrieval-augmented generation, measure whether the correct source was retrieved before judging whether the answer used it correctly. For an agent, record tool selection, arguments, authorisation decisions and final outcome. Without that decomposition, the team knows a score fell but not which subsystem to repair.

Build an evaluation set that resembles production

An evaluation set is often called a golden dataset, but “golden” can imply a certainty that does not exist. Expected answers may be incomplete, policies may change and experts may legitimately disagree. Treat the set as a versioned product with ownership, provenance and review-not a folder of permanent truths.

Sample from the workload the feature will actually receive. Preserve the distribution of ordinary cases, then deliberately add rare cases with disproportionate impact: poor scans, conflicting evidence, unusual language, empty results, malicious instructions and policy boundaries. A set containing only edge cases is useful for robustness testing but cannot estimate normal production performance. Keep separate suites for representative performance and targeted challenge tests.

Prevent contamination between development and final assessment. If engineers repeatedly tune prompts against every example, the suite becomes training data for the application and its score becomes optimistic. Maintain a visible development set, a held-out release set and, for important systems, a separately governed audit sample.

Production corrections are valuable additions, but adding only failures distorts the population. Record why each example exists, which cohort it represents and whether it contributes to a population estimate or a must-not-regress suite.

Use metrics that match the task and the harm

Precision and recall encode business choices. If a model routes suspected fraud, false negatives and false positives have different costs. Choose a threshold using those costs rather than maximising a convenient aggregate such as F1.

Measure abstention as a first-class outcome. A system that declines ten ambiguous cases and correctly handles ninety can be safer than one that answers all hundred with five confident errors. Coverage and quality must be reported together or teams will optimise one by silently sacrificing the other.

Human scoring needs a rubric, not intuition

Human review is necessary for qualities such as usefulness, tone and nuanced correctness, but unstructured ratings are noisy. Give reviewers a rubric with observable criteria, anchored examples and a clear distinction between critical and cosmetic defects. Ask separate questions rather than one vague “How good is this?” score.

Blind reviewers to the model or prompt variant when possible. Randomise presentation order and include repeated items to expose inconsistent scoring. Use at least two reviewers on a sample, measure agreement and adjudicate meaningful disagreements. Low agreement often reveals an ambiguous requirement, not careless reviewers.

Subject-matter expertise matters. A developer can check JSON shape but may not recognise a clinically, financially or legally significant omission. Conversely, domain experts need guidance on the technical boundary they are judging. Evaluation ownership is therefore shared between product, engineering, risk and the people who perform the real work.

Do not turn user approval into ground truth. Users may accept a draft because correcting it takes longer, because they trust the interface or because the mistake is discovered downstream. Acceptance rate measures behaviour; it does not automatically measure correctness.

Automate what can be asserted deterministically

Conventional tests remain valuable around AI. Validate JSON schemas, mandatory fields, allowed identifiers, numeric totals, date ranges, citation existence, tenant boundaries, tool arguments and prohibited actions with code. Do not pay a second model to judge something a deterministic assertion can prove.

Property-based checks can catch errors that reference answers miss. A generated refund must never exceed the captured payment. A summary must not contain an identifier absent from its approved sources. A tool call must remain within the authenticated tenant. These invariants survive wording changes and model upgrades.

Exact text matching is appropriate for identifiers and constrained extraction, but poor for open-ended language. Similarity scores are also not factuality scores: two sentences can be semantically close while disagreeing about a negation, amount or date. Select evaluators according to the failure, not because one metric works across the tooling.

An LLM judge is an instrument, not an oracle

Using a model to score another model makes large evaluation sets affordable, but the judge introduces its own error. It may prefer verbose answers, favour its own style, be sensitive to answer order or fail on specialised domain details. A numerical result does not remove that uncertainty.

Calibrate an automated judge against a human-scored sample. Measure agreement separately for important failure categories, not just overall correlation. Give the judge the evidence and a detailed rubric, request structured reasons, and periodically inspect both false passes and false failures. If the judge and production model share a blind spot, agreement can be high while both are wrong.

Use stronger independence for high-impact decisions: deterministic rules, domain reviewers, alternative judge families or targeted adversarial evaluation. LLM judging is excellent for prioritising human attention and detecting broad regressions. It should not be the sole evidence that a consequential system is safe.

Scores need uncertainty and segmentation

A score of 92% on 25 examples is not equivalent to 92% on 10,000. Report sample size and uncertainty, especially when comparing close variants. Small apparent improvements may be noise caused by which examples happened to be selected or by non-deterministic outputs.

Run repeated trials where response variability matters. Fixing temperature to zero can reduce variation but does not guarantee identical behaviour across infrastructure or model revisions. For agents, repeated runs can expose different tool paths even when final success rates look similar.

Segment results by the dimensions that could conceal harm: document type, language, tenant cohort, customer size, input length, risk band and time period. An aggregate improvement is not a release win if quality falls below an acceptable floor for an important group. Define those floors before looking at candidate results.

Beware benchmark accuracy being mistaken for expected production accuracy. That inference is justified only when sampling and deployment conditions support it. An evaluation proves performance on its tested population; broader claims require explicit assumptions.

Make release gates multidimensional

AI changes usually trade one quality for another. A new prompt may improve completeness while increasing latency and unsupported claims. A larger model may reduce extraction errors but make the feature uneconomic. Release gates should therefore contain multiple constraints rather than one weighted score that allows a safety regression to be cancelled out by better writing style.

A practical gate might require:

• no regression on must-not-fail cases;
• minimum per-cohort correctness and evidence-grounding thresholds;
• zero unauthorised retrieval or prohibited tool actions in the security suite;
• latency and cost within an agreed percentile budget;
• statistically credible improvement where a candidate claims to be better;
• manual review of newly introduced failure clusters.

Version the complete evaluated configuration: model and provider version, prompts, tool descriptions, retrieval index and filters, evaluator versions, dataset commit and application build. A score without this lineage cannot support rollback or incident investigation.

Offline evaluation cannot predict all production behaviour

Real users phrase requests differently, upload novel documents and adapt their behaviour to the feature. Source data changes. Permissions change. Providers update infrastructure. Pre-release evaluation is necessary but cannot close the evidence loop.

Roll out progressively and sample production interactions under an approved privacy policy. Track correction, escalation, abstention, task completion and downstream reversal-not merely thumbs-up feedback. Join those signals to model, prompt and retrieval versions so a change point can be investigated.

Monitor input drift as well as output quality. A new document supplier, longer conversations or a shift in language mix can invalidate the original evaluation distribution without any software deployment. Trigger targeted re-evaluation when that happens.

Use delayed labels where outcomes emerge later. A recommended category may look accepted today but be corrected during month-end reconciliation. The production measure closest to business truth is often separated from inference by days or weeks; design the telemetry and identifiers needed to connect them.

Feedback loops can quietly corrupt the evidence

Production feedback is not automatically representative. Highly engaged users report more issues, obvious mistakes are easier to label than omissions, and reviewers see only the cases routed to them. Treat feedback as a biased signal and combine it with random sampling.

Keep evaluation data separate from unrestricted telemetry. It may contain personal information, confidential documents, model responses and human annotations. Define access, retention and deletion, and preserve provenance when examples are redacted or transformed.

Before training or fine-tuning on corrections, check what the correction represents. A user's preferred wording is not necessarily a factual fix. A reviewer may encode an outdated policy. Learning indiscriminately from accepted responses can reinforce automation bias rather than improve quality.

A production evaluation checklist

• Can the team describe critical failures more precisely than “bad output”?
• Does the representative set match the intended production population?
• Are rare, high-impact and adversarial cases tested separately?
• Are retrieval, generation, tool use and final business outcome distinguishable?
• Do human reviewers use an anchored rubric, with disagreement measured?
• Are deterministic invariants tested without an LLM judge?
• Has every automated judge been calibrated against expert review?
• Are uncertainty, cohort results and abstention reported with aggregate scores?
• Can every result be traced to the exact system and dataset version?
• Will production sampling detect drift and failures that appear after release?

If an evaluation cannot answer these questions, it may still help development, but it is weak evidence for a production release.

Related C3 Software guides

• How to Add AI Features to Existing Business Software Safely
• Using AI for Document Processing Without Losing Human Oversight
• Controlling the Cost of AI Features in SaaS Products
• Prompt Injection: What Business Software Teams Need to Know