Testing and observing AI applications in production

An LLM application can return HTTP 200 on every request while its answers get worse. Model providers have shipped such regressions, found them late, and published how. This post maps how LLM applications are tested and observed once they run in production, as of October 2026: why production needs its own layer, which methods catch which failures, the standards and tools, and the problems that remain open. Benchmarks, grader design and evaluation statistics are covered in a companion post, How AI agents are evaluated; this one starts where that one stops. I will update this page as the field moves; the date above says when I last checked it.

Why production needs its own testing

Failures are quiet

Machine learning systems tend to fail silently when their inputs change, rather than raising errors (Rabanser et al., 2019). LLM applications add new ways to fail quietly.

Anthropic’s September 2025 postmortem describes three infrastructure bugs that degraded Claude’s answers over several weeks. The first, introduced on 5 August, misrouted about 0.8% of Sonnet 4 requests to servers configured for a longer context window. A routine load-balancing change on 29 August raised that share, and in the worst hour on 31 August, 16% of Sonnet 4 requests were affected (Anthropic, 2025). Anthropic’s validation already included benchmarks, safety evaluations, performance metrics, spot checks and canary deployments. Its explanation of the delay: “The evaluations we ran simply didn’t capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes.” Early user reports were hard to distinguish from normal variation in feedback (Anthropic, 2025).

The model is not the only moving part

In April 2026 Anthropic traced a second wave of quality complaints in Claude Code to three changes in the product layer around the model; the API was not affected. A lower default reasoning effort; a caching change whose bug, once a session had been idle for an hour, dropped earlier reasoning on every later turn; and an added system-prompt line about verbosity that, together with other prompt changes, hurt coding quality (Anthropic, 2026). The caching bug passed human and automated code review, unit tests, end-to-end tests, automated verification and internal use. The prompt line passed weeks of internal testing with no regressions in the evaluations that were run. Ablations during the investigation, run against a broader set of evaluations, found one evaluation where the line cost 3% for both Opus 4.6 and Opus 4.7, and the prompt change was reverted (Anthropic, 2026).

The prompt is part of the system under test. Sclar and colleagues found that meaning-preserving changes to few-shot prompt formatting moved accuracy by up to 76 points on LLaMA-2-13B (Sclar et al., 2023).

The same input does not give the same output

Hosted models change under the same name. Chen, Zaharia and Zou compared the March and June 2023 versions of GPT-4: accuracy at identifying prime numbers fell from 84% to 51% (Chen et al., 2023). Pinning a version helps. Anthropic, for example, states that each Claude model ID identifies a pinned version, though not the convenience aliases kept for some earlier models, and that the serving infrastructure around the model can change while the weights stay fixed (Anthropic, Model IDs and versioning). It also lists the earliest retirement date for each model (Anthropic, Models overview). Pinning does not protect against serving bugs like the ones above, and retirement dates force a migration on a schedule you do not set.

Even a pinned model at temperature 0 is not deterministic. Across five models, eight tasks and ten runs, Atil and colleagues saw accuracy vary by up to 15% between runs configured to be deterministic (Atil et al., 2024). Thinking Machines sampled 1,000 completions of one prompt at temperature 0 from Qwen3-235B and got 80 different completions. The cause was kernels that are not batch-invariant: a request’s numerics depend on how many other requests share its batch. With batch-invariant kernels, all 1,000 completions were identical. The same post notes that LLM APIs are not deterministic in practice at temperature 0 (He, 2025), so tests have to tolerate variation.

Some failures are loud

Not every failure is subtle. Datadog’s analysis of LLM traces from its customers found that in February 2026, 5% of LLM call spans reported an error, and 60% of those errors came from exceeded rate limits (Datadog, 2026). Capacity, retries and cost belong in the same monitoring as quality.

None of this is new in kind. In 2015 Sculley and colleagues listed hidden feedback loops, data dependencies and changes in the external world as sources of ongoing maintenance cost in ML systems (Sculley et al., 2015). Breck and colleagues turned that into 28 specific tests and monitoring needs for production readiness (Breck et al., 2017). In interviews with 18 ML engineers, Shankar and colleagues found that success in production turned on velocity, validation and versioning (Shankar et al., 2022). LLMs widen the input space and add the prompt, the provider and the judge as new moving parts.

A map of the layers

Testing and observation in production fall into four layers. Each answers a different question and misses something the others catch.

The layers form a loop. Observation finds failures; failures become test cases; tests and gates stop the same failure from shipping twice. Husain describes three levels with different cadences: assertion-style unit tests on every code change, human and model evaluation on a set schedule, and A/B tests after significant product changes, because each level costs more than the one before (Husain, 2024).

Methods and their trade-offs

Assertions and behavioural tests

The cheapest layer is code that checks outputs: the response parses as JSON, the SQL returns the expected number of rows, the answer cites a document that exists. Husain recommends organising these assertions by feature and scenario, generating test inputs synthetically, and adding cases from failures observed in real use (Husain, 2024).

CheckList, from 2020, gives a structure that still fits. A minimum functionality test checks one behaviour with simple examples. An invariance test applies a change that should not alter the output, such as swapping a place name. A directional expectation test applies a change with a known effect on the output. In a user study, practitioners using CheckList wrote twice as many tests and found almost three times as many bugs (Ribeiro et al., 2020). Invariance tests suit LLM applications well because many inputs have no single correct output, but many perturbations have an obvious expected effect.

Non-determinism changes how these tests are written. Atil and colleagues found that no model they tested gave repeatable accuracy across all tasks, let alone identical output strings, and propose separate agreement metrics for raw output and for the parsed answer (Atil et al., 2024). In practice: assert on extracted fields, run important cases several times, and treat a test that passes four times in five as a finding, not as noise.

Regression suites and release gates

A regression suite is a fixed set of inputs, graded the same way on every candidate release. CI tools make it a gate. Promptfoo’s documentation, for example, shows failing a build on any test failure or when the pass rate drops below a threshold (Promptfoo); DeepEval runs test files through a deepeval test run command for the same purpose (DeepEval). How to grade those tasks, and how many you need to detect a real difference, is covered in the companion post.

Anthropic’s April 2026 fixes show what a gate for prompt changes looks like at scale: a broad suite of per-model evaluations for every system-prompt change, line-by-line ablations, soak periods, a broader suite for any change that could trade off against intelligence, and gradual rollouts (Anthropic, 2026).

A passing offline suite still does not prove a change helps users. Schultzberg argues that agreement between an evaluation and human judgement does not tell a team which changes it can ship without an A/B test; that requires evaluation effects that predict online effects across a class of changes. On 183 interventions from the Upworthy Research Archive, the pooled relationship between evaluation and outcome was weak (Schultzberg, 2026). This is a single-author preprint from October 2026, but the question it poses applies to any team that ships on offline scores alone.

Security testing

LLM applications blur data and instructions. Greshake and colleagues showed in 2023 that instructions planted in content an application retrieves can take control of it, which they called indirect prompt injection (Greshake et al., 2023). Prompt injection is first on the OWASP Top 10 for LLM Applications 2025; the list also includes excessive agency and unbounded consumption (OWASP, 2025).

Scanners automate part of this. garak, an open-source scanner hosted under NVIDIA’s GitHub organisation, probes a model or dialogue system for vulnerabilities in a structured way (Derczynski et al., 2024; garak). Promptfoo covers red teaming alongside evaluation and runs in CI (Promptfoo). For agents that act on what they read, the companion post describes AgentDojo-style injection cases.

Staged rollout

Google’s SRE Workbook defines canarying as a partial, time-limited deployment of a change and its evaluation, which decides whether the rollout continues (Google SRE Workbook). For LLM applications the hard part is the evaluation step. Error rates and latency show up in minutes; a drop in answer quality may not show up in any metric the canary reads.

OpenAI’s April 2025 GPT-4o update shows the limit. Offline evaluations “generally looked good”, and A/B tests suggested that the small number of users who tried the model liked it. Some expert testers said the model “felt” slightly off, but there were no deployment evaluations tracking sycophancy. OpenAI launched, watched early usage and user feedback, began a full rollback on the following Monday, and called the launch decision “the wrong call” (OpenAI, 2025). OpenAI’s assessment is that a new reward signal based on thumbs-up and thumbs-down data was among the changes that, together, weakened the influence of the primary reward signal that had been holding sycophancy in check, and that user feedback “can sometimes favor more agreeable responses”.

My read: an A/B test measures what users reward in the short term. It needs explicit checks for the behaviours you do not want users to reward.

Runtime guardrails

Guardrails act on each request. Llama Guard is a Llama 2 7B model fine-tuned to classify prompts and responses against a safety risk taxonomy (Inan et al., 2023). NeMo Guardrails lets developers write programmable rails, independent of the underlying model, with a runtime inspired by dialogue management (Rebedea et al., 2023). Schema validation and allow-lists for tool calls belong here too.

My read: guardrails trade latency and false positives for protection, and a classifier on the request path is one more model to test and monitor. They also catch little of what the postmortems above describe: degraded answers that look like valid output.

Tracing

A trace records one request end to end: the model calls, retrievals, tool calls and their timing. For an agent that takes dozens of steps, the trace is usually the only record of why a run went wrong.

OpenTelemetry’s semantic conventions for generative AI define names for these spans, metrics and events, and now live in their own repository (OpenTelemetry). They cover inference and embedding calls, agent spans for invoking agents, planning and executing tools (OpenTelemetry, agent spans), and conventions for the Model Context Protocol and the Agent2Agent protocol (OpenTelemetry). Metrics include operation duration, time to first token and time per output token (OpenTelemetry, metrics), and token counters that separate cached input tokens and reasoning tokens (OpenTelemetry, token metrics). There is also an event for evaluation results, gen_ai.evaluation.result, which should be attached to the span it evaluates (OpenTelemetry, events). The status of the whole set is still “Development”.

Content capture is off by default. Recording prompts, completions, system instructions, tool arguments and retrieval queries requires an explicit opt-in, set through OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT, with NO_CONTENT as the default value. The specification warns that some of these attributes are likely to contain sensitive information, including personal data (OpenTelemetry, spans). That default has a cost. Anthropic wrote that its privacy controls, which limit when engineers can see user interactions, made it harder to reproduce the 2025 bugs (Anthropic, 2025).

Traces also feed research. A 2024 taxonomy maps which artefacts and data to trace across an agent’s lifecycle (Dong et al., 2024). MAST classifies failures in multi-agent systems into 14 modes in three categories, derived from expert analysis of 150 traces and applied to more than 1,600 annotated traces from seven frameworks (Cemri et al., 2025). A taxonomy like this is a starting point for labelling your own traces.

Online evaluation

Online evaluation grades live traffic, usually a sample of it, with code checks and model graders. Some tools now run model graders on single spans as well as whole traces: Langfuse added evaluation of individual operations, such as one LLM call, retrieval or tool execution, in February 2026 (Langfuse, 2026).

A model grader in production is only as good as its calibration. Husain’s method has one principal domain expert make binary pass or fail judgements with written critiques, then refines the grader’s prompt until it agrees with the expert. In his example that took three iterations to pass 90% agreement. When failures are rare, he advises reporting the grader’s true positive and true negative rates separately instead of raw agreement (Husain, 2024). Shankar and colleagues found that people need criteria to grade outputs, but grading outputs changes their criteria. They call this criteria drift, and it means a grader needs revisiting as the team learns (Shankar et al., 2024).

Three 2026 preprints test model graders as production monitors, each from a single author:

• False success. Agents often claim to have finished tasks they did not finish. On τ²-bench, 45 to 48% of failures in single-control domains were of this kind. No configuration of five judges and five prompt strategies exceeded an AUROC of 0.65 at detecting them, while lightweight TF-IDF detectors reached 0.83, at 3,300 times lower latency (Advani, 2026).
• Judge drift. A grader behind an API can change without notice, so a falling score may mean a worse product or a changed judge. A fixed human-labelled anchor set, re-scored by the judge at intervals, separated the two. A common rolling z-test raised false alarms on 75% of streams with no drift (Li, 2026).
• Inconsistency. Across five models, scores for identical inputs varied between runs even at temperature 0, and different models scored the same answers differently (Lau, 2026).

For hallucination, two research methods need no reference answer. SelfCheckGPT samples several responses and treats disagreement between them as a sign of fabricated facts (Manakul et al., 2023). Semantic entropy samples several answers, clusters them by meaning and measures the uncertainty across clusters to detect confabulations (Farquhar et al., 2024). Both multiply the number of model calls per checked request, which usually limits them to sampled traffic.

Retrieval-augmented applications have their own metrics. ARES grades context relevance, answer faithfulness and answer relevance with fine-tuned lightweight judges, and needs only a few hundred human annotations (Saad-Falcon et al., 2023). Ragas proposes reference-free metrics for the same pipeline (Es et al., 2023).

Drift and user signals

Input drift is the old ML problem in a new form. Rabanser and colleagues found that two-sample tests, using pre-trained classifiers to reduce the dimensionality of the data, performed best at detecting dataset shift (Rabanser et al., 2019). My read: for text inputs, comparing embedding distributions of this week’s prompts with last month’s is the direct analogue, and clusters of new topics are often more useful than a single drift score.

User feedback remains the signal of last resort, and it played a part in finding each of the provider incidents above. In both postmortems, Anthropic credited reports from users, including feedback sent in the product and reproducible examples posted online, with helping it isolate the problems (Anthropic, 2025; Anthropic, 2026). OpenAI’s case shows the other side: feedback is a biased signal, and optimising for it can produce the failure (OpenAI, 2025).

Tools compared

Most platforms now combine tracing, datasets, prompt management and evaluation; they differ in licence, hosting and how closely they follow OpenTelemetry. The table lists the open or widely used ones as of October 2026. It is a starting point, not a ranking.

Ownership has moved. ClickHouse acquired Langfuse in January 2026 and states that the core stays open source under MIT (ClickHouse, 2026). OpenAI announced in March 2026 that it would acquire Promptfoo and keep building the open-source project (OpenAI, 2026; Promptfoo, 2026). The deal has closed: Promptfoo, now part of OpenAI, remains open source under MIT (Promptfoo; PR #8193). My read: these commitments are worth less than the export format. Instrumenting with OpenTelemetry conventions keeps the option to change backends.

Open problems

Production has no answer key

Offline evaluations have reference answers; production traffic does not. Every online quality signal is a proxy: a grader, a heuristic, a user’s thumb. Schultzberg adds a subtler point: once a gate decides which changes ship, the relationship between evaluation and outcome can no longer be observed for the changes it ships or kills, so drift there goes undetected without randomised audit experiments (Schultzberg, 2026).

The grader is another model to monitor

Model graders miss confident false claims of success (Advani, 2026), change when their provider changes them (Li, 2026) and score inconsistently (Lau, 2026). A monitoring stack built on a judge needs a monitor for the judge.

Humans still find what tests miss

In both Anthropic postmortems and in OpenAI’s, people raised the alarm: users reporting problems, or expert testers saying something felt off. A 2026 preprint makes the same observation about one personal-assistant agent runtime. Over eight weeks it documented 22 incidents in a system defended by 4,286 unit tests and 827 governance checks, and about 70% of its silent failures were caught by a person looking at the product as a user, not by tests or audits (Wu, 2026). It is one system and one author, so treat the number as a hypothesis. The pattern matches the larger cases.

Privacy and debuggability pull in opposite directions

The OpenTelemetry default records no content. Anthropic’s 2025 postmortem shows the price of strict privacy controls: bugs that are hard to reproduce without seeing the interactions (Anthropic, 2025). My read: the default is right, and there is no standard answer yet beyond sampling, redaction and access controls.

Standards are still moving

The OpenTelemetry generative AI conventions are marked “Development” across the board (OpenTelemetry). Until they are declared stable, names and attributes can change, so dashboards and queries built on them need maintenance.

Regulation will require some of this

For high-risk systems, the EU AI Act requires that the system technically allow automatic logging of events over its lifetime (Article 12), and that providers run a post-market monitoring system that collects and analyses performance data (Article 72) (Regulation (EU) 2024/1689). The Digital Omnibus on AI of July 2026 moved the date from which the high-risk requirements in Chapter III apply to 2 December 2027 for systems listed in Annex III and to 2 August 2028 for those under Annex I (Regulation (EU) 2026/1744). My read: teams that already trace and monitor will find most of the technical work done.

How to choose and apply

Start here

1. Instrument first. Emit OpenTelemetry spans for every model call, retrieval and tool call, with model, token and latency attributes (OpenTelemetry). Decide deliberately whether to capture content, where it is stored and who can read it.
2. Pin model versions, record them on every trace, and put the provider’s retirement dates in your calendar (Anthropic, Models overview).
3. Write assertions for everything that has a checkable form, and add invariance tests for inputs that do not (Ribeiro et al., 2020).
4. Read traces regularly, and keep reading them. Label failures and turn them into test cases (Husain, 2024).
5. Treat prompts and configuration as code. Run the regression suite on every prompt change, not only on code changes. At Anthropic, one added line, together with other prompt changes, cost 3% on one evaluation (Anthropic, 2026).
6. Roll out in stages, and decide before the rollout which quality signal, beyond errors and latency, will stop it.
7. Add online evaluation on a sample of traffic. Calibrate each model grader against a domain expert, and keep a fixed labelled set to check the grader itself (Husain, 2024; Li, 2026).
8. For agents, check claimed outcomes against the real state of the system. A judge reading the transcript is a weak detector of false claims of success (Advani, 2026).
9. Monitor capacity and cost next to quality: error classes, rate limits, retries and tokens per request (Datadog, 2026).
10. Make feedback cheap to give, and read it. It helped find each of the provider incidents in this post.

Choosing tools

• Data location. If traces may contain personal data, prefer a self-hosted backend or one with a contractual data region. The table shows which tools you can run yourself.
• Licence. MIT and Apache 2.0 allow modification and redistribution. The Elastic License 2.0 forbids offering the software to third parties as a hosted or managed service (Phoenix LICENSE); internal use is unaffected by that clause.
• Standards. Prefer tools that accept OTLP and follow the OpenTelemetry conventions, so that instrumentation survives a change of backend.
• Fit. A test runner in CI (Promptfoo, DeepEval) and a tracing backend (Langfuse, Phoenix, MLflow) solve different problems. Most teams need one of each.