When text becomes warehouse data

Most of what an organisation knows is written down as text: emails, contracts, claims, call transcripts, customer comments, expert reports. For decades the data warehouse held only the structured part, the amounts, dates and codes. The text stayed outside, read one document at a time by whoever happened to open it.

Bill Inmon, who defined the data warehouse, argued for years that text belongs in it too, and developed what he called textual ETL to get it there. He set out the idea in Tapping into Unstructured Data (2007, with Anthony Nesavich) and later in Turning Text into Gold (2017). For a long time that stayed a niche idea, mostly because the extraction was brittle and expensive. Language models have changed that. They can extract entities, amounts, dates, reasons and sentiment, classify documents, and map the results to the enterprise data model, at a cost that puts many more text flows within reach than before.

Within reach doesn’t mean worth doing everywhere. Processing cost, the review effort needed to trust the output, and the consequences of a wrong extraction vary a lot from one flow to another. A wrong sentiment tag on a customer comment is a nuisance. A wrong amount extracted from a contract can end up in a regulatory report. I’d make the case flow by flow, starting where the value is clear and errors are cheap to catch.

An early attempt

I built a modest version of this a few years ago: an NLP pipeline that turned customer comments into pain points that managers could follow over time. The models were far weaker than today’s, and most of the effort went into making the categories stable enough to compare one quarter with the next.

That last part is what I remember most. Extracting something from a sentence was the easy part. The hard part was making the extraction trustworthy as data: comparable over time, joinable with everything else, and explainable when someone asked why a number had moved.

Once it lands in the warehouse, people will treat it as a fact

The moment model extractions land in the warehouse, finance, risk and management will query them, aggregate them and put them in reports. Most of those users won’t know, or care, that a model produced the number.

That is exactly why the distinction matters. A model extraction is an assertion: this model, with this prompt, read this passage and concluded this value. It becomes something close to a fact only once it has been checked, by reconciliation with structured data, by expert review, or by measured accuracy on a golden set. Until then it should be stored and labelled as what it is: versioned, evidence-backed, and possibly wrong.

Whether verified or not, it also has to respect what Inmon’s definition asks of any warehouse data.

Subject-oriented and integrated. Extracted fields must link to the real keys of the corporate model, the customer, the contract, the product. A parallel “AI schema” that nobody else can join to is a dead end.

Time-variant. Each extracted value carries when it was extracted, and by which model and prompt version. Re-processing the same text with a newer model creates a new version instead of silently overwriting the old one.

Non-volatile. History is kept. If last year’s figures change because the model changed, that has to be a visible, explained restatement. Nobody should discover it by accident in a board pack.

What model-produced data needs on top

The general rules for AI-produced data, such as provenance, lineage and quality checks, are in why I’m still a Data & AI leader. Text extraction adds a few guarantees of its own:

• a link from every value back to the exact passage it came from,
• a confidence score, but only one that has been calibrated against reviewed samples and tied to a decision, for example “below this threshold, a person checks it”. A raw model score on its own is not a reason to trust a value,
• expert review on samples, so quality is measured rather than assumed,
• a golden set to measure extraction quality before each new model version goes live,
• a clear status on each value, such as extracted, reviewed or reconciled, so reports can tell assertions from verified data,
• reconciliation with the structured data wherever both exist. When the contract system says one thing and the extracted text says another, you want to know.

None of this is exotic. It’s the same discipline a good data team already applies to any source, adapted to a source that happens to be a model.

Telling users what they’re looking at

All of that metadata is useless if it stays in the pipeline. The analyst building a dashboard and the manager reading it need to know, at the moment they use a number, that a model produced it, and how much it has been checked.

This is where a data intelligence platform earns its place: the catalogue, lineage and governance layer that sits across the warehouse and the tools people actually use. For AI-produced data, I’d expect it to do a few specific things:

• Mark AI-derived data visibly. Tables, columns and individual values produced or transformed by a model carry a clear label in the catalogue, and that label follows them into BI tools and reports. Nobody should have to read pipeline code to find out.
• Show lineage end to end. From a figure in a report, a user can trace back to the extraction run, the model and prompt version, and the source passage.
• Expose status and quality. Whether a value is extracted, reviewed or reconciled, and the latest measured accuracy for that extraction, sit next to the data, not in a separate report.
• Make usage rules explicit. Some AI-derived fields are fine for trend analysis but not for regulatory reporting or individual decisions. The platform says so, and access policies can enforce it.
• Notify downstream users when a new model version re-processes text and changes historical values.

Without this layer, transparency depends on every analyst remembering which columns came from a model. With it, AI-produced data becomes something people can use knowingly instead of by accident.

Who owns it

This is where data and AI teams meet, and where they most often talk past each other.

The warehouse team owns the model, the keys and the history. The AI team owns the extraction, its quality and how it evolves. The data governance team owns the platform that makes all of it visible to users. Left to work separately, they tend to produce one of two outcomes: a warehouse that refuses AI-extracted data altogether, or a warehouse quietly filled with values nobody can explain, used by people who don’t know where they came from.

Neither is a good place to be. It’s one of the clearest examples of why I think data and AI belong under one accountability, with the two teams designing the extraction and the data model together from the start.

Done well, and where it pays off, textual ETL may be one of the biggest gains AI brings to an organisation: decades of written knowledge, finally available to the same analysis as the numbers. Done carelessly, it becomes a new and very fluent form of data maltreatment.