An article on why AI fault detection models are not 100% accurate, and why that's not important.
“Soo.. how precise is that fault detection model of yours? Does it catch everything?”
A very common question in our line of work. It is also a relevant question, as you often see AI fault detection marketed that way; as 100% accurate. Unfortunately, the reality is not quite as simple.
This article will cover the complex topic of AI accuracy, in an easily understandable way, and let you know just what you can and should expect in terms of AI asset fault detection.
To understand what level of accuracy is reachable and realistic for fault detection models, you first need to understand what makes such models accurate.
First off, it’s important to understand that visual data is a minefield of potential issues, that can impact the accuracy of the analysis results. That applies to situations where humans complete the analysis as well as ones where AI models are doing the heavy lifting. Below are the most common issues:
Since data quality can be negatively impacted, no matter who is analysing the images, the first two potential issues remain relevant for AI- as well as human fault detection.
The issue of fatigue or distraction is completely eliminated when using AI for asset fault detection, as those are human feelings.
If we look at the issue of experience, things start to get more interesting. AI, per default, has zero experience but instead learns everything from the data it is given. Let’s look a little deeper into that.
“Practically speaking, no human is 100% accurate and since humans are our training data source, AI is limited to what the humans train it with.”
– Thomas Laskowski, Head of AI, Scopito
A model for detecting faults in inspection images has to be trained on relevant, labeled data, in order to learn what to look for at what to ignore. You can read about this process in more detail in our AI Explained article.
If you want to reach a high level of accuracy, it is important to train on a diverse data set, and to make sure, that the crew in charge of identifying issues in the training data, do as good a job as possible. Since the model is learning from them, the level of AI accuracy will match theirs.
If you only train on images captured in broad daylight, the model will never learn to identify issues on images taken at dusk. Similarly, if the crew annotating training data overlooks a certain type of issue, the model will never learn to look for that issue either.
Once a training process is complete, you can expect the AI model to match the accuracy of a human, given the context (background, lighting) of the image is fairly consistent. A human will better identify anomalies where the context is unique/extreme such as extreme angle of view, shadows, or a busy background.
If you keep running the model for a year, you can expect the accuracy to at least match a human independent of the context complexity, and out-perform a human with a consistent context.
This improvement comes from the additional training data that becomes available over time from the humans highlighting the missed anomalies. This training data is used to retrain the model giving it better detection capabilities.
What we usually understand as 100% accuracy, is the ability of AI models to correctly identify and tag all faults in all images. With that definition, AI asset fault detection will never be 100% accurate, as no human (or groups of humans) will ever be 100% accurate. However, over time, it can get very close.
“While “accuracy” is tossed around as a metric for AI, the real metric for success is labor savings. An AI accuracy, that is at least as good as human accuracy, is the threshold for success. The AI model will build its accuracy based on what the “humans” agree on which ideally results in a higher accuracy than any one human.”
Even though it is possible to for an artificially intelligent model to outperform a human expert in terms of accuracy, the real value of implementing AI into a workflow, is not the higher levels of accuracy.
The real return on investment comes from the speed at which an AI model can perform analysis. Having results faster saves resources in several ways:
1. AI models can completely or almost completely eliminate the need for human personnel in asset fault detection. As this is a (time)costly operation, eliminating it brings a lot of value.
2. Having analysis results faster, means maintenance crews are able to act faster, avoiding additional costs related to late repairs.
3. Annotated data can be used for predictive maintenance and help avoid failures on assets by predicting them. Getting access to the type and amount of annotated data needed for predictive maintenance, is hard without access to good AI.
So just how fast is asset fault detection using AI, compared to human analysis?
A standard AI model processes one image per second but can be sped up by adding more processing power. A trained profession averages about 30 seconds per image, when the data is simple. That’s a 30x improvement (or savings in labor costs).
Example
Let’s assume a utility has 800.000 structures they need analysed yearly. This utility captures a (modest) 2 images per structure, totaling at 1.6 million images early.
In this example, standard Scopito AI would be able to analyse and annotate all images in 66 days. In comparison, that same operation would take it’s human counterpart 380 days, provided that he/she could work non-stop for 24 hours/day 7 days a week.
In summary, success is augmenting or replacing human labor with artificial intelligence within an acceptable tolerance.
Augmenting human analysis using AI is another method of increasing the accuracy of fault detection results. Some companies choose to let AI sort through their images first and follow up with human manpower to zoom in on more complex issues. In this case, AI is still a huge time-saver.
If we are able to identify faults on kilometers of power structures in minutes with a 90% confidence, that results in immediate labor savings in analysis alone.
The secondary savings comes in accelerating the time to take corrective action on an already deteriorating issue where time is the enemy.
In time, after immediate issues are corrected, then forecasting of deterioration becomes available. For example, we can potentially predict when light rust will become heavy rust and plan accordingly well in advance of emergency action.
We see a future where AI models are part of any successful maintenance operation. That is why we have been working intensely on bringing the best solutions to Scopito and why we continue to do so today.
At the core of every Scopito AI-project, is the belief that standardized models are unsuccessful. Models must be trained on customer-specific data, to be effective. We do this two ways:
1. Offer a train-yourself solution, where users can take advantage of their already annotated data and train a custom AI model on the basis of it. You can preview it in our webinar here.
2. On larger projects, our AI specialists complete the training and any modifications needed in the platform view. We are currently doing several of these projects, and you can read about one here.
Contacts us if you’re considering implementing Artificial Intelligence in your business and would like a quote.
Enter your details below to continue
I have read the Terms of Use and declare that I agree
Please send me occasional e-mails about the product.
14 Days free trial starts automatically when you sign up
