We Pointed Off-the-Shelf AI at Real Inspection Photos. It Detected Almost Nothing on Four of Five Fault Types.
Before we built custom models, we tried the shortcut: ask a powerful general-purpose vision AI to find faults with no training. Here is what it found, what it missed, and why "train on your own data" wins for field inspection.
The tempting shortcut
If you run field inspections, you have probably seen the pitch: point a powerful, ready-made AI at your photos, type in what you are looking for, and let it flag the faults. No training. No labeling. No data scientists. Just describe the defect in plain language and the model does the rest.
It is an appealing promise, and we took it seriously. Before committing to custom models, we ran the experiment ourselves. We took a real field-inspection dataset, actual photos captured by crews in the field, and pointed the strongest off-the-shelf, zero- and few-shot, text-guided detectors we could configure at it. The goal was simple: find the faults with no training, and see how far the shortcut could take us.
No training. No labeling. No data scientists. It is an appealing promise, so we tested it.
What we actually found
The best off-the-shelf configuration, with no custom training, reached only about a 40% overall hit rate on real inspection photos. A more elaborate combined-pipeline approach was projected to reach roughly 50%, but that figure is a projection, the best result we actually measured was around 40%.
More telling than the headline number was the breakdown. Of the five defect types we tested, four came in at near-zero detection, and two of those were literally at 0%. In other words, for most of the faults that matter, the general-purpose AI simply did not see them.
We also ran a self-reference check: we asked the method to match a defect against a near-identical crop of the very same image. On those perfectly matched examples, it preserved the ranking and worked. But the moment it had to generalize across different photos, different lighting, different angles, different units of the same equipment, performance collapsed. That is the gap between a clever demo and a tool you can put in front of a crew.
Four of five defect types came in at near-zero detection. Two were literally at 0%.
Why real field photos break generic models
The reasons are instructive, and they are not the model's fault; they are simply what real inspection work looks like.
First, the faults are small. A high-resolution phone photo gets downscaled before a general model can process it, and by the time it does, a missing screw or a loose clamp occupies only a tiny patch of the image. The detail that defines the defect is mostly gone before the AI ever looks at it.
Second, field photos are messy in ways studio images are not: odd shooting angles, glare, partial obstructions, and lots of repetition. A single enclosure can contain rows of near-identical parts. A generic model trained on the open internet has no reason to know which one is wrong.
Third, and this is the hardest case, many of the most important defects are about absence or state. 'A cover that should be closed is open.' 'A screw that should be there is missing.' These are not objects you can point to; they are conditions. A general detector, handed a few text prompts and no training, cannot reliably encode 'what should be here but isn't' from a handful of examples.
- Tiny parts: faults shrink to a few pixels once a phone photo is downscaled to the model's input size
- Odd angles and glare: real crews shoot in the field, not a studio
- Repetition: rows of near-identical components confuse a model that was never taught your equipment
- Absence and state: 'missing' and 'open' are conditions, not objects, and the hardest thing for a generic model to grasp
The lesson: a model has to be taught what your equipment looks like
None of this means AI vision doesn't work for inspection. It means the shortcut doesn't. A general-purpose model knows a great deal about the world in general and almost nothing about your equipment in particular, your enclosures, your fittings, the specific ways your installations pass and fail.
The documented conclusion from our evaluation was unambiguous: to get reliable detection on real faults, you have to move to supervised custom-model training. You teach the model what a correct installation looks like and what each failure looks like, using your own photos. That is the difference between a model that recognizes 'objects' and one that recognizes your work.
Why this is the foundation of how Forge works
This experiment is exactly why Forge is built around training on your own faults. Instead of hoping a generic model happens to understand your equipment, your team labels real examples of the defects you care about, missing screws, loose clamps, open covers, whatever matters on your sites, and Forge trains a purpose-built model on them.
You don't need data scientists or outside consultants. Your inspection and QA teams already know what good and bad look like; Forge turns that knowledge into a working model and deploys it to your crews' phones in days, not months. The shortcut tops out around 40% on most fault types. A model taught on your own data is built to do the job the generic one can't.
If you have a fault your current tools keep missing, that is the place to start. Bring us your defect types and we'll show you what a model trained on your own data can do.
How we measured this. These figures come from a focused evaluation on a single real field-inspection dataset, using off-the-shelf zero- and few-shot, text-guided detectors with no custom training. The ~40% figure is the best measured overall hit rate; the ~50% figure is a projection for a combined-pipeline approach and was not directly measured. Results on a different equipment set or fault mix would differ, but the pattern, generic detectors struggling with tiny parts, odd angles, repetition, and absence-based defects, is consistent with what these models are designed to do.