If you think about AI in terms of the material it was trained on, there is a fairly simple way to work out where it will be useful and where that usefulness starts to break down.
Ask it something basic. What flavour of ice cream is nice?
Billions of people eat ice cream and millions have written about which flavours they like, so there is an enormous amount of material on the subject in the training data. There is also a pull towards the average position in that data, so you will get a safe answer. Vanilla. Chocolate. Something a lot of people like.
That single example contains two facts, not one. The model is on very firm ground, and it will still drift towards the middle of that ground. A large amount of data gets you the most typical answer rather than the best one.
Making the question harder
Ask about software development instead. There are obviously fewer software developers than there are people who like ice cream, but there are still millions of them, along with vast quantities of documentation, tutorials, forum posts, books and argument. Ask how to make a piece of software do a particular thing and there is a good chance the relevant knowledge is in there somewhere.
Now start combining things. Say I want to bring software development together with economics. There are millions of developers. There are plenty of economists. The number of people who genuinely understand both is much smaller, and that raises a second question: how much of what those people know has ever been written down anywhere it could end up in training data?
People know far more than they publish. You can spend twenty years in an industry and know things that seem so obvious to you that you have never written an article about them. Nobody has.
So as knowledge becomes more specialised, and particularly as you begin combining specialisms, there is less and less relevant material sitting behind the answer.
I think that matters for the way we talk about AI being creative. These systems are very good at pattern matching, and at combining things out of the patterns they have learned. But ask for a connection between two specialist areas that hardly anyone has written about, and there may be very little in the data connecting them. There is no separate reservoir of creativity that fills in the missing knowledge.
Then add local knowledge
Say I want to build software that models the economy of Ballymoney.
An AI could have plenty of economics behind it, plenty of software development, and some factual information about Ballymoney. What it is unlikely to have is much material combining all three usefully, and it may still not understand how they fit together locally.
Working purely from what is written down, you might assume Ballymena is more economically relevant to Ballymoney than Coleraine is. Ballymoney and Ballymena are both associated with North Antrim. Coleraine is in East Derry. On paper, that sounds reasonable.
Anyone living around Ballymoney knows Coleraine is hugely relevant. People work there, shop there and trade there. They travel in that direction. That is local knowledge, and very little written material captures it properly, which means very little of it reaches the training data either.
So I am now asking for economics, software development and specific local knowledge about Ballymoney, all at once. If I want a genuinely useful result, I am going to have to bring most of that myself. I provide the data. I provide the local context. I say which assumptions hold and which do not. The application, and most of the creativity, comes from me.
That is usually what the work looks like in practice. Not hunting for a cleverer model, but giving a capable one the things it has no way of knowing: your figures, your documents, your previous decisions, a connection to the system where the real data actually lives, and somebody who can tell it which of its assumptions do not apply here. A lot of disappointing AI projects are not really model failures. The model was never given the knowledge that mattered.
The answer arrives regardless
Here is the danger. The model still answers. It does not stop when the relevant knowledge is missing. It gives you a good-looking answer, a confident answer, and it can be absolute bollocks.
I see a version of this when I am consulting. Someone asks me a question and I give them the best answer I have, but where something is uncertain, I say so. The evidence suggests this. I think this is probably the better option. I would give this an eighty per cent chance of working. Sometimes there is no hundred per cent answer.
People do not always like that. They want certainty. So you get the strange situation where somebody asks an expert, receives a careful answer with some uncertainty in it, then puts the same question to an AI and is told yes, absolutely, this is the answer. The AI sounds more convincing. Not because it knows more, but because it sounds more certain.
If you do not know enough about the subject yourself, you have no way of telling the difference. You do not know what you do not know, and you do not know what the AI does not know.
The average is not necessarily right
Come back to vanilla, because that pull towards the average causes a second problem.
We have already seen it in AI inheriting biases around gender, race and disability. If those biases are present in the material, they become part of the patterns learned from it. Where most of the information about something comes from one group of people, that group’s experience starts to look like the default. Where a group is poorly represented, misunderstood, or mostly discussed through stereotypes, that becomes part of the pattern too.
The same mechanism turns up in far more ordinary places, and small businesses run into it constantly. Large organisations are much better documented than small ones. American practice dominates advice that then gets applied in the UK. Urban assumptions get carried into rural areas. Ask about marketing and you can be handed guidance written for a marketing department, when your marketing department is one person who also answers the phone.
The most common view in the training data is not necessarily the correct view. The most represented experience is not everybody’s experience. What appears most often is not automatically the most useful or accurate answer, and it may not be the answer for your business.
AI companies can train systems away from some of those behaviours and put safeguards in place. That rather illustrates the point. If we have to deliberately correct patterns learned from the data, we have already accepted that the average pattern is not automatically the answer we want.
Why this keeps mattering more
We are all going to be under more pressure to use AI over the next decade. Businesses will use more of it. Governments will use more of it. Workers will be expected to use more of it. That is not completely illegitimate, because these tools can be genuinely useful.
But AI literacy cannot just mean knowing how to write a prompt. It means:
- Having some sense of what is likely to sit behind the model.
- Knowing when you are asking about something common and when you are asking about something specialised.
- Recognising when you are combining fields that have rarely been combined before.
- Knowing when local knowledge is doing the real work.
- Recognising when the important information may never have been written down in the first place.
Mistakes scale as easily as good work
The danger with AI is not simply that it can be wrong. People are wrong all the time. The difference is that AI lets us produce, repeat and act on answers at enormous scale, so a bad assumption underneath an answer scales exactly as fast as a good one.
Which is why this still comes down to judgement. The model can generate an answer, help you analyse something, help you write software, organise information or explore an idea. You are still the person deciding whether the answer makes sense. You are responsible for knowing when to question it, for supplying the context it does not have, and for what you eventually do with what it gives you.
That responsibility does not disappear because the AI sounds confident. If anything, the easier these systems make it to generate and scale an answer, the more the judgement matters.
AI can help you work faster and at far greater scale. Whether what you are doing is right still depends on how well you understand the tool, the problem, and the answer in front of you.
At Dynumo, I help businesses work out where AI and automation genuinely fit. Sometimes that means connecting AI to your own data and context, sometimes automating the workflow around it, and sometimes training a team to judge when an answer can be trusted. Often it means deciding that a simpler approach would do the job more reliably.
If that is a conversation worth having, get in touch or book an introduction.
