Everyone talks about foundation models and use cases. Nobody talks about the integration layer in between — the part where most AI projects actually break.
The foundation model is the interesting problem. Which model? What architecture? Fine-tuned or prompted? These are the questions that attract engineers, generate conference talks, and feature in write-ups. They are also, increasingly, the easy part. The capable models are good. The tooling has matured. You can get a working AI system off the ground in weeks.
The use case is also well-understood. Organisations know what they want to automate or augment. They have a business case, a sponsor, and a target outcome. That part is fine.
The middle layer is where projects die quietly. Data pipelines that do not generalise. API contracts that break when the model version changes. Latency assumptions that made sense in the demo and do not survive contact with production traffic. Escalation paths that were designed without thinking about what the AI system does when it is wrong.
What the middle layer actually consists of
The integration layer between a capable foundation model and a reliable production system has several distinct components, each of which introduces risk.
Data pipelines. The model needs data to work with. That data needs to be retrieved, cleaned, chunked, embedded, indexed, and served — often in real time. Each step in that pipeline is a potential failure point. The data that was available in the development environment is often not the same data that runs in production, and the differences matter more than teams expect. Schema changes, missing fields, encoding inconsistencies, and distribution shifts in the underlying data all degrade system performance in ways that are easy to miss until a user reports something wrong.
API and versioning contracts. Foundation models change. Providers update them, deprecate versions, and alter behaviour across API updates. A system that works reliably against a specific model version can degrade without warning when that version is updated or retired — unless API versioning and model pinning are treated as explicit design decisions rather than implementation details. Most early-stage AI projects do not treat them that way.
Latency and load. A demo works at demo scale. Production traffic is different in volume, distribution, and timing. AI inference has variable latency in a way that most traditional software does not. A pipeline that returns results in 800ms on average may return them in 8 seconds under load, with tail latencies that break the user-facing system or trigger cascading failures in downstream processes. Load testing AI pipelines — not just functional testing — is consistently underinvested.
Escalation and fallback paths. What does the system do when it is wrong? Every AI system will be wrong sometimes. The question is whether that wrongness is handled gracefully or whether it propagates. A system with no defined fallback path treats every failure as an uncaught exception — which is fine in a prototype and not acceptable in production. Defining escalation paths (to a human, to a simpler rule-based system, to an explicit "I cannot answer this" response) is part of the production readiness work, not an afterthought.
Observability. You cannot fix what you cannot see. AI systems require a different observability approach from traditional software: logging input distributions, tracking output quality metrics, detecting drift, and monitoring the behaviour of the system over time as underlying data and model behaviour shift. Most teams underinvest in this until something breaks in production and they cannot diagnose why.
Why teams consistently underestimate this layer
The middle layer is not interesting in the way that model selection is interesting. It does not feature in the architecture diagrams that get shared at conferences. It is plumbing — and plumbing does not get the investment it deserves until the pipes start leaking.
There is also a timing problem. By the time a team reaches the integration layer, they have typically spent most of their budget on the model work, the prototype, and the stakeholder presentation. The integration layer is then delivered under time and budget pressure, which is precisely the wrong condition for getting it right.
The teams that reliably ship production AI systems treat the middle layer as a first-class design concern — budgeting for it, specifying it, and testing it — before the model work begins. That reversal of priority feels counterintuitive. It is also the pattern that distinguishes teams that ship from teams that prototype indefinitely.
If your AI programme is stuck between a working prototype and a reliable production system, the problem is almost always in this layer. Our AI Services approach is built around getting this layer right — alongside the business analysis discipline that surfaces data and process assumptions before they become production problems.