Testing a system whose outputs are non-deterministic is genuinely different from testing traditional software. The difference is not primarily technical — it is philosophical.
In traditional software testing, a test either passes or fails. The system either returns the right value or it does not. You write the assertion, you run the test, you know. Repeatability is assumed. Any deviation is a bug.
In an AI-native system, the same input can produce different outputs on different runs. Some of those outputs are correct. Some are not. The question is not "does this system always return the right answer?" but "does this system return a good-enough answer, often enough, in the right contexts?" That requires a fundamentally different approach to test design, coverage, and what it means for a test to pass.
The test management discipline has not fully caught up with this shift. Most teams apply traditional QA patterns to AI systems and then wonder why their test coverage does not translate to production confidence.
The core problem: you cannot write deterministic assertions for non-deterministic systems
Classic unit and integration testing relies on assertions: given this input, expect this output. The test framework checks whether the assertion holds. For an AI system, the output of a given input is a distribution, not a value. Writing a deterministic assertion against a distribution is not a test — it is a sampling exercise.
Teams that try to work around this by fixing random seeds, caching model responses, or testing only against specific recorded outputs are not testing their AI system. They are testing a static snapshot of it. Any subsequent model update, prompt change, or shift in the underlying data distribution invalidates those tests without the test suite indicating a failure.
What AI-native testing actually requires
Effective test coverage for AI-native systems requires building an evaluation framework rather than a test suite in the traditional sense. The distinction matters.
An evaluation framework asks: across a representative sample of inputs, does the system's output distribution meet defined quality thresholds? It does not ask: for this specific input, is the output exactly this value.
The components of a minimal evaluation framework for a production AI system:
A curated test dataset. A fixed set of inputs that represents the full distribution of cases the system will encounter in production — including edge cases, adversarial inputs, and the cases most likely to surface failure modes. This dataset needs to be maintained and extended as the system encounters new input patterns.
Defined quality thresholds per output type. For a classification system: acceptable precision and recall bands across categories. For a generative system: human-evaluated rubrics for coherence, accuracy, and appropriate refusal. For an agentic system: correctness of action sequences across a representative set of scenarios. These thresholds should be set before build, not calibrated to whatever the current model achieves.
Regression testing against model and prompt changes. Every change to a model, a prompt, or a data pipeline is a change to the system's behaviour. The evaluation framework should run automatically on each change and surface degradation — including improvements in one area that mask degradation in another.
Latency and reliability testing under load. AI systems have more variable latency than traditional software. Testing what happens to output quality and system behaviour under production load — not just functional correctness on single inputs — is a frequently skipped but critical component of production readiness.
The human evaluation problem
For generative AI systems, some element of human evaluation is unavoidable. There is no automated metric that reliably captures whether a generated document is accurate, appropriately nuanced, and fit for purpose in context. Teams that try to avoid human evaluation typically discover late in the cycle that their automated metrics were measuring something correlated with quality, but not quality itself.
The practical approach is to design human evaluation into the test process from the start — accepting that it is a recurring cost, not a one-time exercise — rather than treating it as a shortcut to be avoided. Small, consistent human evaluation samples, run on a cadence, give better signal than large sporadic exercises.
What this means for test management practice
The test manager role in an AI programme looks different from traditional QA. It requires understanding evaluation methodology — how to design representative test datasets, how to define meaningful quality thresholds, how to interpret degradation signals across model changes. It also requires a willingness to accept that "all tests green" does not mean the system is behaving correctly.
The teams that develop production-grade AI systems are the ones that invest in evaluation infrastructure early — before the build is complete — rather than treating testing as a phase that happens after the system is considered done.
For more on how we approach test management for AI-native systems, see our Test Management services and our AI Services approach.