How to do high-precision robot-camera calibration with ArUco markers

đź’ˇ How-to
, , , ,
1

Based on contributions by BramHeerschap.

When you’re handling small parts, small camera-to-robot misalignment causes real positioning errors. This method splits the calibration into two independent stages — camera→board and board→robot — and deliberately over-determines each one with more points than the mathematical minimum, so detection noise gets averaged out instead of directly corrupting the result.

This targets OpenCV ≥ 4.7 (ArucoDetector, getPredefinedDictionary). It also assumes you already have a camera intrinsic calibration (camera_matrix, dist_coeffs) — see How to calibrate a camera and map pixels to world coordinates with a ChArUco board or How to calibrate a camera and convert a pixel distance to a real-world distance if you don’t have one yet.

What you need

  • A camera with a prior lens calibration (camera_matrix, dist_coeffs).
  • Four ArUco markers placed at accurately known, fixed positions on a flat surface (measure them carefully — any error here propagates through both calibration stages).
  • A robot you can jog to touch physical points on the board.
  • OpenCV ≥ 4.7, Python, numpy.

Steps

1. Define the board layout

Place four ArUco markers on the flat work surface and record their center positions in millimeters, keyed by marker id:

WORLD_POINTS_MM = {
    2: (0.0, 0.0),
    0: (100.0, 0.0),
    1: (0.0, 80.0),
    3: (100.0, 80.0),
}

2. Detect the markers and build point correspondences

Instead of using only each marker’s center (4 points total), use its four corners and its center (5 points per marker, 20 points total) — noise in any single point matters less when it’s averaged against 19 others.

:warning: Check: the original description of this method didn’t state the physical marker size or show the code that turns WORLD_POINTS_MM into per-corner object points — the block below reconstructs it from the description (“4 corners + 1 center per marker”). Measure your own printed marker and set MARKER_SIZE_MM before running this.

import cv2
import numpy as np

MARKER_SIZE_MM = 40.0  # measure your printed marker and set this

def marker_object_points(center_mm, size_mm):
    """
    Corner order must match cv2.aruco's detected corner order:
    top-left, top-right, bottom-right, bottom-left, then the center.
    """
    cx, cy = center_mm
    h = size_mm / 2
    return [
        (cx - h, cy + h, 0.0),
        (cx + h, cy + h, 0.0),
        (cx + h, cy - h, 0.0),
        (cx - h, cy - h, 0.0),
        (cx, cy, 0.0),
    ]

aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_5X5_1000)  # ⚠️ Check: use the dictionary your markers were generated with
parameters = cv2.aruco.DetectorParameters()
detector = cv2.aruco.ArucoDetector(aruco_dict, parameters)

# gray: a grayscale frame from your camera (cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY))
corners, ids, _ = detector.detectMarkers(gray)
if ids is None:
    raise RuntimeError("No ArUco markers detected.")

object_points = []
image_points = []
for marker_corners, marker_id in zip(corners, ids.flatten()):
    if marker_id not in WORLD_POINTS_MM:
        continue
    obj_pts = marker_object_points(WORLD_POINTS_MM[marker_id], MARKER_SIZE_MM)
    img_pts = list(marker_corners[0]) + [marker_corners[0].mean(axis=0)]
    object_points.extend(obj_pts)
    image_points.extend(img_pts)

object_points = np.array(object_points, dtype=np.float32)
image_points = np.array(image_points, dtype=np.float32)

3. Estimate the camera pose and compute a board homography

solvePnP gives you the camera’s 3D pose relative to the board (useful if you need the camera height, or want to sanity-check the setup):

ok, rvec, tvec = cv2.solvePnP(object_points, image_points, camera_matrix, dist_coeffs)

Because the board is planar, the pixel→board mapping itself is simplest as a direct homography fit through the same correspondences (drop the z=0 column):

board_xy = object_points[:, :2]
H, mask = cv2.findHomography(image_points, board_xy, cv2.RANSAC)

def pixel_to_board(x, y, H):
    pt = np.array([[[x, y]]], dtype=np.float32)
    out = cv2.perspectiveTransform(pt, H)
    return out[0, 0]

Using 20 points instead of the minimum 4 significantly improves accuracy: detection noise averages out, individual point errors matter less, and the fit is more stable overall.

4. Board to robot calibration

Touch physical points on the board with the robot and pair them with their known board-plane coordinates. Using more than the minimum (3) points again improves precision — for example, touch each marker twice (once on a vertical edge, once on a horizontal edge) for 8 calibration points spread across the workspace:

M, _ = cv2.estimateAffine2D(board_pts, robot_pts, method=cv2.RANSAC)

def board_to_robot(x_mm, y_mm, M):
    pt = np.array([[[x_mm, y_mm]]], dtype=np.float32)
    out = cv2.transform(pt, M)
    return out[0, 0]

board_pts and robot_pts are Nx2 numpy arrays of matching board-plane and robot coordinates, built the same way as any of the point-collection approaches in How to calibrate a camera and map pixels to world coordinates with a ChArUco board or How to map camera coordinates to robot coordinates with a least-squares fit.

5. Run the full pipeline

x_mm, y_mm = pixel_to_board(px, py, H)
rx, ry = board_to_robot(x_mm, y_mm, M)

This lets the robot move to any point detected by the camera. Test it by picking points in the image and verifying the robot reaches the expected location; if accuracy isn’t good enough, add more calibration points or improve marker detection (better lighting, sharper focus, bigger markers).

Common mistakes

  • Inaccurate marker positions or size. A 1 mm error in WORLD_POINTS_MM or MARKER_SIZE_MM shifts every downstream coordinate.
  • Skipping RANSAC. Both cv2.findHomography and cv2.estimateAffine2D accept a method=cv2.RANSAC argument, which automatically rejects a bad touch point or a misdetected corner instead of letting it skew the whole fit.
  • No prior lens calibration. solvePnP and marker detection accuracy both depend on a good camera_matrix/dist_coeffs — don’t skip that step (How to calibrate a camera and map pixels to world coordinates with a ChArUco board).
  • Mixing up which points belong to which stage. Keep the camera→board correspondences (image pixels ↔ board mm) and the board→robot correspondences (board mm ↔ robot mm) as separate point sets; they use different fitting functions (findHomography vs. estimateAffine2D).

Related

Rewritten and consolidated (Sept 2026) from the original student how-to’s: High precision robot-to-camera calibration.

How to map camera coordinates to robot coordinates with a least-squares fit
How to calibrate a camera and map pixels to world coordinates with a ChArUco board
Choosing a method to convert camera pixels to robot coordinates
How to convert a RealSense camera pixel to a 3D point and robot coordinates
Start here: index of all how-to articles
How to calibrate a fixed camera to a robot with a single ArUco marker