ΑΙhub.org
What happens when AI runs out of pictures?
By Khawar Islam
A hospital may only ever collect a few dozen scans of a rare condition – for example, an unusual tumour. The radiology department wants software to flag this on a scan – not to replace the specialist, but so a hospital without one still gets their scan checked the same way.
The clinicians know what they’re looking for. The problem is arithmetic.
Over a decade, the hospital might gather 40 confirmed cases. Enough for an expert human to recognise a pattern, but the software needs thousands. This is the bottleneck behind a lot of artificial intelligence (AI) in the real world.
Text is everywhere and costs almost nothing to gather. Images are the opposite.
A lot of people are excited about learning by example, and they need enormous numbers of examples that someone qualified has already checked and labelled.
In medicine, that someone is a specialist whose time is scarce. On a factory line, the defect you most want to catch is the one that almost never happens. For a farmer, the disease worth detecting early is the outbreak that has not arrived yet.
Our new method, GenMix, aims to solve this issue – and to maximise its impact, we made the code free for anyone to access.
It keeps the same technology on a much shorter leash – using AI to alter existing images just enough that we can turn dozens into hundreds, all while maintaining rigorous accuracy in detection and recognition.
The trouble with computer vision
Researchers in computer vision spend a lot of effort squeezing more value out of existing photos to teach AI systems how to better detect things.
The standard tricks are simple: flip the photo, crop it, brighten it, cut a rectangle out of one image and paste it into another.
That last approach is popular, but it produces something that could not exist. Half a dog stitched to half a cat teaches the system something, though not always something true or worth teaching.
There is a second problem, and it’s worse.
If we let AI generate pictures to train computer vision models, it can produce things like birds that don’t exist. Picture: Nano Banana.
Or objects can be put in places they would never actually be. Picture: Nano Banana.
AI models can struggle to alter images without drifting too far from the original. Picture: ClaudeAI.
Models tend to do well on pictures that resemble the ones they trained on, then stumble when conditions change. Different camera, different light, different season.
The real world is full of exactly those mismatches.
The obvious modern answer is to let generative AI produce the extra pictures. In practice, it has disappointed.
Ask AI for a thousand new photos of a bird, and it will happily oblige, but the birds drift. The small details that separate one species from another get smoothed away or invented outright.
Models trained on that material sometimes do worse than models given nothing extra at all.
The GenMix fix
This is what GenMix hopes to solve.
Instead of asking the AI to invent a bird, it gets a real photograph and selects any of the nine prompts embedded in the system: make this snowy, make this an aurora, make this a watercolour or a crayon sketch.
The subject stays put. Only the world around it changes.
We then blend the edited version back with the untouched original, so every training picture keeps a foot in reality.
A filtering step compares each edited image against the original it came from and discards anything whose meaning has drifted too far, so the system never trains on the failures.
The GenMix method selects from nine prompts to alter the original image. Picture supplied by author.
A filtering step ensures the system never trains on images that have drifted too far. Picture supplied by author.
We tested it on several public image collections across a range of tasks: everyday object recognition, fine-grained recognition (where differences are subtle, like telling 200 species of birds apart), learning from very small datasets, and situations where the training and target images come from visibly different sources.
It improved results in all of them.
It also held up better when we deliberately tried to fool the trained models with a scatter of tiny pixel changes, invisible to a human but enough to make a classifier confidently name the wrong thing.
Models trained with GenMix were fooled about 30 per cent less often than standard models. That matters more than it sounds.
A system that can be tricked by a few altered pixels is not one you want reading medical scans.
We’re not asking the model to dream up new data; we’re asking it to show us the pictures we already have under conditions we could not go out and photograph.
This is just the beginning
While GenMix is an exciting step forward in improving AI use in detecting medical anomalies, technical defects and other hard to spot issues, there are still limitations.
Our experiments run on public photo collections, not patient data, and any clinical use would need its own validation and approvals.
What we have built is a tool for the situation where labelled examples are scarce and expensive. That situation turns up in healthcare imaging, transport, agriculture, security and education, and it’s the reason the work has drawn interest beyond computer vision researchers.
The code is freely available, which is deliberate.
The groups working on rare diseases or threatened species are rarely the groups with budgets for data collection, and they are the ones this should help most.
Next, we want to make the approach cheaper and faster to run, and test it properly in the specialised areas where data is hardest to gather: medical imaging, environmental monitoring and safety-critical systems – the places where being wrong is expensive and can impact lives.
This work builds on the original paper published in Expert Systems With Applications, written with colleagues at the University of Melbourne, Mohamed bin Zayed University of Artificial Intelligence and Information Technology University in Lahore, and extends work we first presented at the Computer Vision and Pattern Recognition conference in 2024.
AUAI is supported by:
