A machine learning model was trained to predict risk of death in patients with pneumonia to help determine whether they should be admitted to hospital or treated as an outpatient.
When the researchers evaluated the model’s predictions, they found something surprising. The model predicted that asthmatic patients were at a lower risk of dying to pneumonia.
Statistically, the model was correct. In the historical data, asthmatic patients with pneumonia did have better outcomes. However, from a medical perspective, this conclusion made little sense.
Doctors already knew that pneumonia patients with asthma were vulnerable, so they tended to treat them more aggressively, admitting these patients to the ICU. The model saw the better outcomes, but it did not understand the clinical decisions that had produced them. The model therefore incorrectly concluded that asthma was protective against pneumonia.
Left unchecked, the model would have recommended less intensive treatment for the patients who needed it most.
The AI Black Box
In the previous example it is relatively simple to see where the model went wrong. It identified a correlation in the data but lacked the context needed to understand what was causing it.
The decisions that modern AI systems arrive at are however often less intelligible. Large language models can contain billions of parameters, and it is often difficult to explain precisely why a particular input leads to a particular output.
That creates a problem for organisations. An AI system can produce an answer that sounds convincing while relying on incomplete information, outdated information, or relationships that do not mean what we assume they mean.
As AI systems become more complex it becomes increasingly difficult for us to see exactly how that system arrived at a particular answer.
More Data Is Not More Context
The obvious response is to give AI access to more organisational data. Surely with more data the AI will have more understanding and make better decisions right?
However, access does not automatically lead to understanding.
Organisations are full of competing definitions, undocumented exceptions, legacy processes and local workarounds. A model might have access to five different definitions of “customer”, but it does not inherently know which one Finance relies on, which one is outdated, or which one applies to a particular decision.
AI can read what is documented, but that does not mean it understands which system is authoritative or what a “normal” business process looks like in practice. Without that context, having more information can simply create more ways for the AI to produce something wrong.
Better Data Management, Better AI
This is where established data management practices become critical to AI.
Metadata tells us what data means. Lineage tells us where it came from and how it has changed. Data quality helps establish whether it is complete, accurate and current.
The pneumonia example shows why good data management is imperative to the success of AI systems. The model did not fail because it lacked data. It failed because the data, on its own, did not capture enough of the clinical context behind the outcome.
The same risk exists inside organisations. If data is poorly defined, inconsistently managed or stripped of its context, AI can learn the wrong lesson from it with complete confidence.
If an organisation cannot confidently explain what its own data means, where it came from and whether it should be trusted, an AI system will not magically solve that problem.
Context is the next challenge
We may never be able to fully open every AI black box. But we can make the environment around it much less opaque.
Organisations should be able to explain what information an AI system could access, where that information came from, how it was governed and whether it was fit for the decision being made.
AI can find patterns in your data. Good data management helps make sure they are the right ones.
James Wilson
Graduate
