Skip to content
Snowfox AISnowfox AI

15 October 2026 · 6 min read

Generative AI in production: what changes after the demo?

A demo shows a model can answer once. Production needs evaluation, retrieval, integration, guardrails, cost control and monitoring around the model.

By the Snowfox AI team

A generative AI demo shows that a model can produce a good answer once, on an input someone chose. A production system has to produce acceptable answers every day, on inputs nobody chose, using company data, inside existing systems, at a known cost, with someone accountable when it is wrong. Most of the work between the two is evaluation, retrieval, integration, guardrails, cost control and monitoring. Very little of it is the model.

What is the difference between a demo and a production system?

Generative AI means models that produce text, code or images. In business settings this is usually a large language model (LLM), a model trained on large amounts of text that generates a response to a prompt.

The model is the same in both. What differs is everything around it.

Dimension Demo Production system
Inputs A few chosen examples Whatever users and upstream systems send
Data Sample files Live company data, with access rights
Quality Judged by eye in a meeting Measured against a fixed evaluation set
Failure Ignored or retried Detected, logged and handed to a person
Cost Not tracked Budgeted per task and monitored

A good demo tells you the task is possible. It does not tell you how often it works, what it costs, or what happens when it fails.

Where is generative AI used in production?

Snowfox’s generative AI page describes four families of use, each with its own condition for success.

  1. Codebase and cloud migration. Moving existing code to a new language, framework or cloud platform. It suits LLMs because there is an objective check: the migrated code has to pass the same tests and reproduce the old behaviour. Where tests do not exist, write them first.
  2. Agents for data analysis. An agent is an LLM that calls tools, such as running a query, in a loop until a task is done. It depends on a governed data layer with agreed metric definitions. Without one, the agent computes a confident number from the wrong table.
  3. Customer support and process automation. Drafting replies, classifying requests, completing steps in a workflow. It works when the scope is narrow and the handoff to a person is explicit.
  4. Internal assistants wired into ERP and CRM. Assistants that answer questions and act in business systems. The hard part is permissions and write access, not conversation.

How do you evaluate a system whose output is free text?

Build an evaluation set: real inputs from the process, each with a description of what a good output must and must not contain. Then score every version of the system against it.

  1. Collect real cases, including the awkward ones.
  2. Write grading criteria a colleague could apply without asking you.
  3. Automate the checks that can be automated (required fields, citations, format) and have people review a sample of the rest.
  4. Rerun the full set on every prompt change, retrieval change and model version change.

What not to do: approve a system because five questions went well in a steering meeting.

Why does retrieval matter more than the model?

Most business questions are about company information the model was never trained on. Retrieval-augmented generation (RAG), the technique behind enterprise search assistants, handles this. The system searches company documents, passes the relevant passages to the model and asks it to answer from them.

When a RAG system answers wrongly, the cause is often retrieval. The search returned an outdated policy, a duplicate, or a fragment of a table that lost its headers. A stronger model does not fix that.

  • Document hygiene. Remove superseded versions and decide which source is authoritative.
  • Permissions. Retrieval must respect the access rights of the person asking.
  • Separate measurement. Evaluate whether the right passage was retrieved separately from whether the answer was right.

This is data engineering work, and often the larger part of the project.

What does integration with ERP and CRM involve?

Reading from a system and writing to it carry different levels of risk. Start read-only and add write actions one at a time, each with a defined scope.

For every action the assistant can take, settle whose identity it acts under, what its limits are, whether repeating it by mistake is safe, where it is logged, and whether it needs human approval before it runs.

What guardrails does a production system need?

Guardrails are the controls placed around the model, in three places.

  • On input. Treat retrieved documents and incoming messages as data, not instructions. Prompt injection, where text inside a document or email tries to redirect the model, is a risk for any system that reads external content.
  • On output. Validate structure, require citations to source passages, and check for personal data that should not leave.
  • On actions. Keep an allowlist of tools, limits on amounts and volume, and approval steps for anything hard to undo.

The system also needs a way to decline and hand over to a person.

How do you keep cost and quality under control after launch?

Cost is driven by tokens (the units of text a model reads and writes), the size of the model and the number of calls per task. Agents multiply calls, so a task that looks cheap in a demo can be expensive at volume. The levers are smaller models for simple tasks, less context per call, caching and a budget per task.

Monitoring means logging inputs, retrieved passages, outputs, tool calls, latency and cost, then reviewing a sample on a regular schedule. Rerun the evaluation set whenever the model version changes, because providers update and retire models and behaviour shifts with them. This is MLOps, the practice of operating machine-learning models in production, applied to LLMs.

When is generative AI the wrong tool?

Often. AI does not only mean generative AI. Snowfox lists machine learning, data engineering, workflow automation, system integrations, analytics, custom software, enterprise search and AI agents as approaches, and many solutions combine several.

If the problem is The usual fit is
Predicting or optimizing a number (demand, price, risk) Machine learning
A fixed sequence of steps between systems Workflow automation and integrations
Understanding what happened and why Data engineering and analytics
Reading, writing or searching language and documents Generative AI, NLP and search

Three published Snowfox cases make the point. In retail pricing, the published case reports a 6% margin increase compared with matched comparison stores, using a causal model with demand and cross-elasticity forecasts. For a consumer-goods company, the published case reports a 64% click-through improvement in the first month, using clustering and reinforcement learning. For a media portal with more than 50 million monthly page views, the published case reports more than 50 analytical views within three months, built on a data mart and ETL jobs. None is described as a generative AI project.

Where to start

Start with the problem, not the technology. Pick one process with a clear owner, a measurable outcome and real examples you can turn into an evaluation set. Check the data and systems it touches before choosing a model.

Questions and answers

Why do generative AI demos fail to become production systems?

A demo proves a model can answer well on chosen inputs. Production requires measured quality on real inputs, access to company data with permissions, integration with existing systems, guardrails, cost control and monitoring, and most of that work is outside the model.

What is RAG and why does it matter for enterprise generative AI?

Retrieval-augmented generation (RAG) searches company documents and gives the relevant passages to a language model so it answers from them. It matters because most wrong answers come from retrieving the wrong or outdated document, which a stronger model does not fix.

When should a company not use generative AI?

When the problem is predicting or optimizing a number, running a fixed sequence of steps, or understanding what happened in the business. Those are better served by machine learning, workflow automation and integrations, or data engineering and analytics.

Start with the problem, not the technology

Tell us what is slow, expensive or invisible in your operation. If AI or better data is the right answer, we will show you where. If it is not, we will tell you that too.

Not ready to scope a project?

Start with the AI Readiness Assessment. Five dimensions, reviewed together, and an honest read on where you actually are.

Take the assessment