Large Language Models (LLMs) are rapidly advancing, offering promising capabilities for visual inspection and quality control. But when working with visual data, the more accurate term is Large Multimodal Models (LMMs), which extend LLMs by processing additional modalities such as vision or audio. For simplicity, we will refer to them as LLMs throughout this article, as the term is more widely recognized and commonly used.
Many companies are eager to explore LLMs for their own visual quality inspection solutions. However, the key question remains: Can LLMs match the performance of tailored approaches?
To find out, we conducted a series of experiments evaluating LLMs in real-world visual inspection tasks. The first involved detecting defects on PCB boards, a complex challenge due to the nature of the dataset. The second focused on industrial inspection, where conditions are unpredictable, and defects vary significantly. These case studies help illustrate where LLMs succeed, where they fall short, and what this means for businesses considering AI-driven quality control.
Case Study 1: PCB Board Defect Detection
Our first experiment focuses on detecting defects in PCB boards using the VISA dataset (PCB4) from AWS Open Data. This dataset presents significant challenges, including a limited number of defect images, class imbalance, and the complexity of multilabel classification. Some defect classes are well represented, while others are scarcely present, making it difficult for any model to generalize across all possible defects.
To evaluate GPT-4o’s performance, we presented images to the model along with a description of potential defect types. The model was then asked to classify the defects or confirm that the board was defect-free.
Experiment Results
We tested 40 images under two conditions. In the first scenario, only defective images were used. The model achieved an F1 score of 0.37, performing well for certain defect types but missing others entirely. The F1 score represents the quality of a model when dealing with varying sample distributions across classes. A score closer to 1 indicates better performance.
In the second scenario, both defective and non-defective images were included, leading to an improved F1 score of 0.59 – mainly because defect-free images were correctly classified.
To improve accuracy, we introduced one-shot learning, providing the model with a reference image showcasing possible defects before testing. This significantly improved performance, raising the F1 scores to 0.51 and 0.68 for the respective scenarios. However, even with this enhancement, the results fell short of traditional anomaly detection methods, where a comparable AU PRC score of 0.98 has been reported in prior studies1.
From PCB Boards to Industrial Inspection
While PCB defect detection provided valuable insights into LLM performance, the conditions in this experiment were still relatively structured compared to real-world industrial inspections. In many industries, inspections take place in far less predictable environments, with varying lighting, object orientations, and visual complexities that make defect detection even more challenging. To further test the capabilities of LLMs in such conditions, we turned to a second case study involving industrial inspection images and videos from a client’s dataset.
Case Study 2: Industrial Inspection Defect Detection
The second case study involved detecting defects in industrial inspection images and videos. Unlike PCB inspections, which often occur in controlled environments, industrial inspections are much more unpredictable. The dataset introduced additional complexities: visually similar defect types, a multilabel structure, and sparse labeling. Lighting conditions, object orientations, and image resolutions varied, making the task even more challenging.
Experiment Results
We evaluated two different models, which currently top Chatbot Arena Leaderboard for vision tasks: ChatGPT-4o and Gemini-2.0-pro-exp-02-05. We tested two different approaches. In the first scenario, the models were required to return the most likely and second most likely defects. This method improved metrics but did not yet meet the accuracy required for a production-grade alarm system. One-shot learning, where defect examples were shown before testing, helped improve predictions but not as much as to rule out manual verification by human operator of the system.
The second approach simplified classification into a binary decision – defect vs. no defect. This method improved accuracy from 0.61 in the initial test to 0.73 for ChatGPT-4o and 0.63 to 0.66 for Gemini-2.0-pro-exp-02-05. Despite this improvement, these results remain insufficient for real-world industrial systems that demand near-flawless reliability.
Key Challenges of LLMs for Industrial Visual Inspection
While LLMs demonstrate potential for quality inspection, several limitations must be considered before deploying them in real-world industrial settings:
Processing Time: LLMs accessed via APIs introduce latency, as each image must be processed individually. Depending on the complexity of the request and the required one-shot learning examples, processing times can range from 1 to 10 seconds per image. This delay makes real-time quality control impractical in fast-paced industrial environments.
Hardware and Scalability: Deploying smaller, locally hosted models (such as Qwen or Molmo) can help reduce reliance on cloud-based APIs, but this requires substantial hardware investment. Additionally, scaling such a solution across multiple inspection locations introduces logistical and infrastructure challenges, further complicating adoption.
Read more about why tailored LLMs are a smart solution for businesses: Here
Can LLMs Replace Traditional Quality Control Systems?
LLMs continue to improve, but they are not yet a standalone solution for industrial visual inspection. While they can assist with certain tasks, they require human oversight and additional infrastructure.
For businesses considering LLM-based quality control, trade-offs must be carefully evaluated. Traditional machine learning models still outperform general-purpose LLMs in controlled environments. However, for companies operating outside standard production lines – where conditions vary, imagery differs, and mobile data acquisition is required – LLMs may serve as a useful complementary tool rather than a full replacement.
Work with theBlue.ai
Many businesses see the potential of LLMs/LMMs for quality control, but real-world deployment presents significant challenges. While these models can be useful for experimentation, their results are often inconsistent when subjected to rigorous testing. Traditional approaches may still provide better accuracy and reliability in many cases.
Our role is to help businesses navigate these complexities. We analyze each case individually, evaluate different AI-based and traditional solutions, and conduct thorough testing under real-world conditions. This ensures that businesses do not rely on a one-size-fits-all model but instead implement the most effective and dependable solution for their needs.
Our role is to:
- Evaluate AI models for practical deployment rather than just theoretical performance.
- Test and compare different methodologies, ensuring that businesses use the most effective approach available.
- Provide expert consultation on integrating AI solutions, minimizing risk and maximizing efficiency.
- Ensure reliability and scalability, so businesses can confidently implement AI-driven inspection at scale.
With our expertise, we provide consulting on selecting the right approach, assessing model performance, and ensuring practical integration into existing workflows. Many companies may not have the specialized knowledge required to conduct these evaluations themselves, and that’s where we come in. We bridge the gap between cutting-edge AI technology and real-world quality control needs, ensuring accuracy, efficiency, and scalability.
If your company is exploring AI-powered quality control, we can help identify the best approach -wether LLM-based, traditional, or a hybrid model. Get in touch to discuss how we can optimize your inspection processes and improve defect detection reliability.
Sources:
1Zou, Yang, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. “SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation.” arXiv, 2022, https://arxiv.org/pdf/2207.14315.
Frequently Asked Questions (FAQs)
Can LLMs completely replace traditional visual quality control methods?
No, while LLMs can assist in detecting defects and anomalies, they do not yet match the reliability and accuracy of traditional machine learning models or rule-based systems, especially in industrial applications where consistency is critical.
What are the biggest challenges when using LLMs for visual inspection?
The main challenges include processing time, the need for cloud-based APIs, variability in image conditions, and the requirement for human validation due to inconsistent results.
Can an LLM-based quality inspection system work in real-time?
Currently, LLMs face latency issues due to API processing times, making them impractical for real-time applications. Locally hosted models can reduce this lag but come with scalability and hardware investment challenges.
How can businesses determine if LLMs are the right choice for their quality control needs?
The best approach is to conduct structured experiments and compare results with traditional methods. We help businesses evaluate different AI models and select the most effective solution based on their specific requirements.
Are there cases where LLMs outperform traditional methods in defect detection?
LLMs can be useful in exploratory analysis, situations with highly variable conditions, or when labeled data is scarce. However, in well-structured industrial environments, traditional models still provide better accuracy and reliability.
How can our company get started with AI-driven visual quality control?
We offer consultation and testing services to assess the feasibility of AI-based quality inspection in your specific application. Contact us to discuss how we can tailor a solution to your needs.
