DevOps vs MLOps vs LLMOps: how generative AI reached production

The familiar diagram presents three pipelines as successive generations. It is useful orientation, but it hides the decisive point: DevOps, MLOps and LLMOps are neither competing products nor replacements for one another. They are operating responses to artifacts that became progressively harder to define and verify. Software added delivery automation; machine learning added data and statistical behaviour; foundation models added natural-language context, external providers, tools and semantic risk.

This is the documented history of that change and, above all, a guide to what an LLMOps team actually has to operate. There is no universally agreed birthday for either MLOps or LLMOps, so the timeline uses verifiable milestones instead of crediting a single inventor.

Three names, three artifacts to maintain

These are not rigid organisation boundaries. An LLM product still needs CI/CD, secrets, networking, rollbacks and SLOs. If it owns model weights, it also needs data lineage and a model registry. LLMOps extends DevOps and MLOps; it does not erase them.

2009: DevOps and the handoff problem

At Velocity 2009, John Allspaw and Paul Hammond presented 10+ Deploys per Day: Dev and Ops Cooperation at Flickr.^[flickr] The title carried the thesis: frequent delivery did not depend on a hero throwing a package over a wall, but on cooperation and automation. The first DevOpsDays took place in Ghent that same year.^[devopsdays]

A documented anecdote: ten deployments a day sounded provocative while many organisations released quarterly. The lesson was not to worship the number ten. Smaller changes shortened feedback and gave development and operations joint ownership. DORA later formalised delivery metrics and found that speed and stability need not be opposites.^[dora]

DevOps made the pipeline a product: versioned code, automated tests, reproducible artifacts, progressive delivery, telemetry and rollback. Yet a test could still assert, fairly deterministically, whether a function returned the expected value.

2015–2017: when the artifact began to learn

With ML, behaviour no longer came from code alone. It depended on data, transformations, hyperparameters and the world in which predictions were made. Google’s paper on hidden technical debt popularised two useful images: model code is only a small part of the real system, and in an entangled system changing anything can change everything.^[debt]

That is the need now called MLOps: reproduce datasets and features, validate schemas, track experiments and models, prevent training-serving skew, monitor drift and decide when to retrain. The 2017 ML Test Score proposed 28 tests and monitoring needs for production readiness.^[mltest]

A measurable story: the TFX paper describes Google replacing fragile glue scripts with standard components. In one Google Play deployment, models refreshed on new data; the authors reported faster experiment cycles, time to production falling from months to weeks, and a 2% increase in app installs associated with improved data and model analysis.^[tfx] That is a result from one case, not a universal promise.

MLOps added continuous delivery of the pipeline and continuous training to CI/CD. Google Cloud’s architecture guide distinguishes a manual process, automated ML pipelines and full CI/CD pipeline automation.^[mlops]

2020–2022: from an isolated model to a foundation system

Three changes prepared LLMOps. First, RAG combined parametric memory with an external index, giving systems updateable knowledge and provenance; Lewis and colleagues published that work in 2020.^[rag] Second, Stanford’s 2021 report consolidated the term foundation model: a broadly trained model adapted to many tasks, with both leverage and the risk that many products inherit the same defects.^[foundation]

Third, the product was no longer just model weights. InstructGPT showed in 2022 that human-feedback tuning could materially alter usefulness: on the authors’ prompt distribution, human raters preferred a 1.3B model to 175B GPT‑3, though it still made simple mistakes.^[instruct] HELM, also in 2022, argued for multidimensional, transparent evaluation rather than one benchmark score.^[helm]

The LLMOps label spread in this environment. It does not mean “MLOps plus a prompt”, and it does not require training a foundation model. Many applications never train weights: they operate APIs, prompts, retrieval, tools and rules. Their unit of change is the compound LLM system.

What LLMOps actually operates

1. A versioned bundle, not a loose prompt

A deployable version should identify at least: provider and model version; inference parameters; system messages and templates; tool schemas; RAG corpus, parser, chunking, embeddings and index; safety policies; evaluation set; and orchestration code. Two runs with the same visible prompt can be different products when any of those parts changes.

Promotion should be immutable: app@sha + prompt@sha + index@snapshot + eval@version + policy@version + model@id. If that combination cannot be reconstructed, an incident cannot be explained and a rollback cannot be trusted.

2. Evaluation before optimisation

Software tests an expected output. Language admits multiple correct forms and eloquent false answers. LLMOps therefore starts with an evaluation dataset drawn from real tasks, edge cases and foreseeable abuse. Each item needs a rubric, not merely a golden string.

• Task quality: correctness, completeness, format and instruction following.
• RAG: retriever recall, context relevance, answer faithfulness and citation quality. Scoring only the final answer hides whether retrieval or generation failed.
• Safety: direct and indirect injection, data leakage, tool abuse and disallowed content.
• Operations: end-to-end latency, tokens, cost, retries, error rate and quota saturation.

LLM judges help scale review, but they are models too: calibrate them against human judgements, record their version and preserve disagreement. A serious release gate combines deterministic checks, judges, adversarial tests and human sampling.

3. RAG as an online data pipeline

RAG is not “attach a vector database”. It is a chain: document authorisation → extraction → cleaning → chunking → embeddings → indexing → retrieval → reranking → context assembly → generation → citations. Every arrow can regress.

Version corpus and index, measure freshness, stop users retrieving documents they cannot access, retain chunk IDs in traces, and test with questions whose sources are known. Retrieval can improve grounding, but it does not turn output into truth or remove prompt injection; OWASP says so explicitly.^[owasp]

4. Semantic observability and FinOps

An HTTP 200 does not mean the answer helped. A trace should span input, policy, retrieval, reranking, model calls, tools, retries and final output. OpenTelemetry now defines conventions for models, token counts, messages and tool calls; content capture must remain optional because it may expose sensitive data.^[otel]

Useful dashboards join quality and operations: task success and groundedness by version; p50/p95/p99 latency; input/output tokens; cost per completed task rather than per call; cache hit rate; agent depth; tool failures; and human escalations. A model that is cheap per token can be expensive if an agent loops.

Define SLOs from what the user experiences, as SRE practice recommends: availability, latency and correctness.^[sre] An LLMOps SLO might be: “95% of authorised policy questions return within six seconds, with a valid citation and no cross-tenant disclosure.”

5. Security, governance and limits on agency

The model consumes data and instructions through the same channel. That ambiguity makes prompt injection an architectural problem, not a filter added at the end. OWASP maintains an LLM-specific risk catalogue; NIST’s Generative AI Profile structures actions for governing, measuring and managing risk.^{[nist][owasp]}

• Treat retrieved text, web pages and tool results as untrusted input.
• Apply least privilege per tool; separate reading, proposing and executing.
• Validate output before SQL, shell, email, payments or infrastructure changes.
• Require human approval for irreversible or high-impact actions.
• Define retention, PII redaction, residency and access to prompts and traces.
• Ship a kill switch, step and budget limits, timeouts and rollback.

A production LLMOps pipeline, step by step

1. Define task and risk. What decision it supports, what it may do, who is accountable and what must never happen.
2. Create the evaluation set. Normal cases, edges, languages, relevant groups and attacks; freeze a baseline.
3. Select by constraints. Task quality, latency, privacy, region, context, tools and cost — not a general leaderboard.
4. Build context. Use RAG only when it adds knowledge or traceability; preserve ACLs, provenance and freshness.
5. Instrument before launch. Version and correlation IDs, spans, tokens, costs and evaluation outcomes.
6. Run offline gates. Regression, safety, load and budget. An average gain must not hide a critical failure.
7. Deploy shadow or canary. Compare on real traffic without granting the candidate full agency.
8. Sample production. Automated signals, contextual feedback and privacy-aware human review.
9. Promote or revert the whole bundle. Model, prompt, index, tools and policy move together.

Three anecdotes that became operating rules

Flickr, 2009: “ten deploys a day” is what people remember, but “Dev and Ops cooperation” was the durable half of the title. Frequency was the visible effect of shorter feedback and shared responsibility.^[flickr]

TFX, 2017: the team did not solve production with a better algorithm. It replaced ad-hoc glue with a repeatable platform. That is the transition from “we have a model” to “we can operate models”.^[tfx]

Tay, 2016: Tay predated modern LLMs, so it should not be described as an LLM failure. It was an operational warning. Microsoft wrote that a coordinated attack exploited an unforeseen vulnerability during the first 24 hours, and the bot was taken offline.^[tay] The LLMOps rule still applies: laboratory tests, red teaming, live observation and a withdrawal path are part of the product.

Common LLMOps mistakes

• Calling a prompt catalogue a platform. Without evaluation, lineage, permissions, traces and rollback, it is storage.
• Measuring only hallucination rate. Utility, abstention, format, safety, latency and cost also define success.
• Logging every prompt without policy. Observability can become a shadow copy of sensitive data.
• Swapping models directly in production. Even a “better” model changes style, tool calling, length and cost.
• Giving tools the service account’s permissions. An agent should act with the user’s scope or a minimal function identity.
• Turning positive feedback into automatic training. A thumbs-up says little about correctness, safety or representativeness.

The real evolution: widening the object of responsibility

DevOps made software change operable. MLOps added changing data and predictive behaviour. LLMOps adds changing context, intent and action. A mature practice is not more boxes; it is the ability to answer five questions for every output: which version produced it, with what context, how it was evaluated, what it was allowed to do, and what it cost.

If those answers exist, the system can improve without faith. If they do not, the “LLMOps pipeline” is still a demo with a network route to production.

Sources and further reading

Primary papers, official project documentation and first-party incident accounts used for this article.

1. John Allspaw & Paul Hammond — 10+ Deploys per Day: Dev and Ops Cooperation at Flickr
2. DevOpsDays — Ghent 2009, official event archive
3. DORA / Google Cloud — 2019 Accelerate State of DevOps Report
4. Sculley et al. — Hidden Technical Debt in Machine Learning Systems
5. Breck et al. — The ML Test Score
6. Baylor et al. — TFX: A TensorFlow-Based Production-Scale ML Platform
7. Google Cloud Architecture Center — MLOps: Continuous delivery and automation pipelines
8. Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
9. Bommasani et al. — On the Opportunities and Risks of Foundation Models
10. Ouyang et al. — Training language models to follow instructions with human feedback
11. Liang et al. — Holistic Evaluation of Language Models (HELM)
12. Microsoft — Learning from Tay’s introduction
13. NIST AI 600-1 — Generative AI Profile
14. OWASP GenAI Security Project — Top 10 for LLM Applications
15. OpenTelemetry — Inside the LLM Call: GenAI Observability
16. Google SRE Book — Service Level Objectives