How to plan a basic vision-based quality control system

thijs · September 2, 2026, 12:18am

Based on contributions by anarueda.

Before writing any detection code, it helps to plan a vision-based quality control (QC) system as a sequence of well-defined stages. This how-to is a workflow overview, not a specific implementation: use it to structure a project that decides, from a camera image, whether a part is OK or defective.

What you need

A camera and a way to capture consistent images of the parts you want to inspect.
A rough idea of what “correct” and “defective” look like for your process.
Depending on which stage you implement: OpenCV (How to zoom into an image or video frame with OpenCV using array slicing, How to detect and generate fiducial markers, barcodes, QR codes and data matrix codes with OpenCV), an HSV color filter (How to detect objects by colour using HSV filtering in OpenCV), or a segmentation model such as SAM 3 (How to segment objects with SAM 3 using text prompts in Python).

Steps

1. Define the inspection goal

Before writing any code, be clear about what your model actually needs to answer. In most projects it’s the same underlying question: is the result of my process correct, or does it have a flaw that prevents it from functioning properly?

Decide what the final output should be, for example PASS/FAIL or OK/DEFECTIVE.
Focus only on the features that actually matter for the inspection task, ignore everything else.
Keep the first version simple. Solve one problem well before adding more complexity.

2. Collect and label example images

The most important ingredient for a good model is good data. A smaller, well-organized dataset that’s actually representative of your goal will outperform a larger, messy one.

Capture images that represent both correct and defective cases.
Make sure labels are consistent and unambiguous.
Split the data into training and test sets so the system can be evaluated properly, rather than just checked against images it was trained on.

3. Preprocess the images

Preprocessing makes the images easier for a model (or a simpler classical algorithm) to work with. Depending on your setup this can mean cropping to the relevant area, using fiducial markers to normalize the view (see How to detect and generate fiducial markers, barcodes, QR codes and data matrix codes with OpenCV), or dividing the image into smaller regions to inspect separately.

Remove unnecessary background information wherever possible.
Keep the position and scale of the object as consistent as possible between captures, for example with a fixture or an alignment marker.
Check intermediate outputs manually. Don’t assume preprocessing is actually helping, verify it.

4. Train (or choose) a simple classifier

Once images are prepared, pick the approach best suited to the problem. Prefer a simpler, more reliable method over a complex one, especially if you’re new to computer vision or machine learning: a well-tuned classical technique (HSV color filtering, contour/shape checks, a reference-object technique like How to detect transparent glass using a reference object and OpenCV) is often enough, and is easier to debug than a full model.

Start with a standard model or a simple baseline before reaching for something complex.
Train it on the labeled images and monitor performance on validation/test data, not just training data.
Save the best-performing version of the model, not just the last one you trained.

5. Turn predictions into a practical decision

A prediction is only useful once it leads to an action. In a QC system, this usually means converting a raw model output into a clear, actionable result the rest of the process can use.

Define a simple decision rule based on the model’s output (for example, a confidence threshold).
Make the final result easy to interpret — a clear PASS or FAIL, not a raw probability.
Test the full pipeline end-to-end on new, real images, not only on training data.