Why AI projects stall on data, and what a data foundation requires
AI projects stall when data is not available or not trusted. A data foundation needs repeatable pipelines, a warehouse with BI, and clear ownership.
By the Snowfox AI team
AI projects stall on data because the data a model needs is either not available when it is needed or not trusted by the people who would act on the result. A data foundation is the minimum set of pipelines, storage, definitions and rules that makes data available and trustworthy at every stage, from the source system to the decision. It does not have to be complete before a first model. It has to be sufficient for one use case.
Why do AI projects stall on data?
The model is usually the smallest part of the work. The common sequence goes like this: a team receives a one-off extract, builds a promising prototype, and then finds that the extract cannot be reproduced on a schedule, the numbers do not match the finance report, or a key field is typed in by hand.
The recurring causes are few:
- Data sits in several systems that do not share identifiers.
- Extracts are produced manually, so nothing can run every day.
- Departments define the same metric in different ways.
- History is overwritten instead of kept.
- Nobody is sure who owns the data or whether it may be used for this purpose.
This applies to every kind of AI, not only predictive models. An assistant built with retrieval-augmented generation (RAG), where a language model answers using documents retrieved from your own content, will give confident answers from outdated documents if nobody maintains the document store.
What does “available and trustworthy at every stage” mean?
Available means the data arrives where it is needed, at the frequency it is needed, without a person moving it. Trustworthy means the same question gives the same answer wherever it is asked, and when a number is wrong, someone can trace why.
Some terms first. ETL stands for extract, transform, load: the jobs that pull data from source systems, clean and reshape it, and load it into storage. A pipeline is the chain of those jobs. A data warehouse is a central database organized for analysis instead of transactions, and a data mart is a smaller version focused on one subject. BI, or business intelligence, is the reporting and dashboard layer.
| Stage | What goes wrong | What to check |
|---|---|---|
| Source systems | Fields entered by hand or changed without notice | Who owns each source and how changes are communicated |
| Ingestion | Manual exports, jobs that fail silently | Whether every load is scheduled and monitored |
| Transformation | Business rules buried in scripts one person understands | Whether rules are documented and tested |
| Storage | Several competing copies of the truth | Whether each metric has one agreed home |
| Consumption | Dashboards nobody opens, models fed by a separate path | Whether reports and models read from the same tables |
Where do pipelines and ETL become the bottleneck?
Usually in four places.
- No orchestration. Orchestration is the tooling that runs jobs in order, retries failures and raises alerts. Without it, one failed job breaks everything downstream and nobody notices until a report is wrong.
- Full reloads. Reprocessing everything on every run when only new records changed makes the pipeline slow and expensive.
- Duplicated logic. The same rule is implemented in three scripts and drifts apart.
- No data tests. Nothing checks row counts, missing values or duplicates before the data is used.
What not to do: start by rewriting everything on a new platform. Diagnose the existing pipelines first and fix the bottlenecks that cost performance and reliability.
Why build a warehouse and BI before a model?
Because reporting is the cheapest test of whether the data is trusted. If leaders look at a revenue dashboard and do not believe it, they will not believe a forecast built on the same tables.
There are two more reasons. Reporting is valuable on its own, before any model exists. And the cleaned, agreed tables that feed dashboards are the same tables a model will learn from.
What not to do: build dashboards nobody asked for.
What does governance add?
Data governance defines who owns each dataset and each definition, who can access it, how long it is kept and how personal data is protected. For AI it also has to answer one more question: may this data be used for this purpose?
A workable minimum is short: an owner for each critical dataset, a written definition for each key metric, access rules, and retention and privacy rules you can defend.
What not to do: set up a governance committee before there are datasets to govern.
What does this look like in practice?
Snowfox’s media data pipeline case is a useful worked example because the published version describes no model at all.
A major media portal needed a scalable, economical data foundation that could combine many systems and support timely analysis of revenue and audience behavior. The published case reports that the portal handled more than 50 million page views each month.
The approach was a data mart feeding Power BI views through Python ETL jobs orchestrated with AWS Step Functions. The published case reports that within three months leaders had visibility across more than 50 analytical views, while infrastructure costs were kept low.
Four things are worth noting:
- The scope was set by business questions, revenue and audience behavior, not by an ambition to centralize all data.
- The storage layer is a data mart, not an enterprise-wide warehouse.
- The components were plain: Python jobs, a managed orchestrator and a BI tool.
- The output reached leaders, the people making the decisions, which is the test that matters.
A foundation like this is what a first model would stand on.
How much foundation is enough before a first model?
Enough for one use case. Check these six points:
- The data the use case needs is loaded by a scheduled, monitored pipeline, not by hand.
- There is enough history at the right level of detail, and it is kept, not overwritten.
- The key entities, such as customer, product or store, have consistent identifiers across the sources involved.
- The target metric has one agreed definition, and the business side accepts the numbers in a report.
- Someone owns the data, and it is clear that it may be used for this purpose.
- There is a route for the model’s output to reach the system or person who acts on it.
If these hold for one use case, start. If they do not, the first project is a data project, and that is a legitimate result.
Two mistakes sit at opposite ends. One is waiting for a complete enterprise data platform before trying anything. The other is skipping the foundation and hoping the model compensates. A model does not correct data it cannot trust. It repeats the same errors faster.
Where to start
- Data engineering covers pipeline and ETL diagnosis, warehouses and BI.
- Data strategy connects business priorities to architecture and governance.
- For high-volume or real-time processing, see big data consulting.
- Read the media data pipeline case.
- If you are not sure where the gaps are, the AI Readiness Assessment reviews data alongside four other dimensions. Or tell us the problem.