The most common objection I hear to testing LLM-based systems is: the output is different every time, so how do you write a test?
It is a reasonable objection. The testing approaches most engineers have internalised — unit tests with exact output assertions, regression packs that compare strings — do not transfer directly to systems where the model output varies across runs.
But non-determinism does not mean untestable. It means the test strategy needs to change. The properties you test for are different, and some of the tooling is different, but the underlying discipline — define what correct behaviour looks like, verify that the system exhibits it, and prevent regressions — is exactly the same.
Here is what that looks like in practice.
The category error
The reason people struggle to test LLM systems is that they are testing the wrong thing.
A unit test for a deterministic function checks: given input X, does the system return output Y? This works because Y is fixed. You can assert on it directly.
For an LLM system, the equivalent question is not what does it return? — it is does it return something that satisfies the properties we care about?
This is not a looser test. It is a different framing. The properties you care about are precise and enumerable. Asserting on them is no less rigorous than asserting on a specific string. It is often more rigorous, because string comparison tells you nothing about whether the output is actually correct — only whether it matches the expected string exactly.
The testable properties
Every LLM system has a set of properties that define correct behaviour. Identifying them is the first step in building a test strategy.
Here are the categories I use:
Format compliance. Does the output conform to the required structure? If the system is supposed to return JSON, does it? If it is supposed to return a list of items, does it? These are deterministic assertions — the format is either correct or it is not.
Content accuracy. Does the output contain the information it should? For retrieval-augmented systems, does the answer reflect the retrieved context? For classification systems, does the assigned category match human judgement on a reference set? Accuracy tests require a labelled reference dataset, but they are well-defined and repeatable.
Boundary behaviour. What does the system do with inputs it should not process? Does it refuse appropriately? Does it escalate? Does it produce a coherent output that signals uncertainty rather than hallucinating confidence? Boundary tests are among the most important in regulated environments.
Consistency. Given the same input across multiple runs (with temperature > 0), does the system produce outputs that are semantically equivalent? This does not mean identical — it means the key properties hold across runs. A test that runs the same prompt ten times and checks that the format compliance and accuracy properties hold each time is a valuable consistency check.
Latency and reliability. Does the system respond within the required time? Does it handle API errors gracefully? These are not unique to LLM systems, but they are often neglected in the enthusiasm around model quality.
The reference dataset
The single most important piece of infrastructure for testing an LLM system is a labelled reference dataset.
This is a set of inputs, each with a defined expected behaviour (not expected output — expected behaviour). For a document classification system, this means a set of documents with known correct categories. For a question-answering system, this means a set of questions with known correct answers, or known acceptable answer properties.
Building this dataset is tedious and unavoidable. There is no substitute. The model cannot be evaluated without a ground truth to evaluate against.
For regulated environments, the reference dataset also needs to cover protected-characteristic inputs — documents or queries that relate to protected characteristics under equality legislation. The system's behaviour on these inputs needs to be explicitly verified, because it will be explicitly scrutinised.
Regression testing for LLMs
A regression pack for an LLM system checks that the properties defined above continue to hold after any change — model version, prompt change, retrieval configuration, or anything else that touches the system.
The pack runs automatically before every deployment. This is not optional in a regulated environment. If you cannot demonstrate that the system's behaviour has not degraded, you cannot safely deploy.
The key design decisions:
Thresholds, not exact values. The regression pack should not fail because accuracy dropped from 94.3% to 94.2%. It should fail if accuracy drops below a defined threshold (say, 92%) or if any previously-passing boundary test fails. Set thresholds at meaningful values, not at last-run values.
Separate packs for separate properties. A regression failure should be diagnosable. A monolithic test suite that fails without indicating which property regressed is hard to act on. Keep format compliance tests, accuracy tests, and boundary tests in separate groups so a failure is immediately informative.
Version the reference dataset. As the system evolves, the reference dataset should grow. New edge cases discovered in production should be added to the regression pack so they are tested on every subsequent deployment. Version the dataset so you can trace when a particular test was added.
Evaluation beyond the pack
The regression pack tells you whether the system is worse than it was. It does not tell you whether it is good enough for a new use case or a new input distribution.
For that, you need evaluation — a more thorough assessment against the reference dataset, including human review of a sample. Evaluation should run at:
- Initial production sign-off
- Any significant model or prompt change
- After retraining on new data
- Periodically as production drift monitoring
Evaluation is more expensive than regression — it requires human time, not just automated checks. Scoping it carefully (sample size, reviewer criteria, sign-off protocol) is part of the test strategy.
Monitoring in production
Testing before deployment is necessary but not sufficient. Production behaviour can drift as input distributions change, as the model is updated by the provider, or as the retrieval corpus evolves.
A minimum viable monitoring setup for an LLM system includes:
- Confidence distribution tracking (if the system surfaces confidence scores)
- Escalation rate (the proportion of inputs routed to human review)
- User correction rate (if the system has a feedback mechanism)
- Periodic automated re-evaluation against the reference dataset
Thresholds on these metrics should trigger alerts. The question of what action to take when an alert fires — investigation, rollback, retraining — should be defined before launch, not after.
Putting it together
A complete test strategy for an LLM system in a regulated environment includes:
- A labelled reference dataset covering normal inputs, edge cases, and protected-characteristic inputs
- Automated regression pack running pre-deployment: format compliance, accuracy, boundary behaviour, consistency
- A sign-off evaluation protocol using the reference dataset before any production launch
- A production monitoring setup with defined thresholds and escalation paths
None of this is conceptually different from testing any other complex system. The difference is that the assertions are on properties rather than outputs, and the reference dataset replaces the fixed expected values of unit testing.
The discipline is the same. The framing changes.