Choosing a method to convert camera pixels to robot coordinates

đź’ˇ How-to
, , , ,
1

Based on contributions by Bastiaan.

Every vision-guided robot project eventually needs to answer the same question: the camera sees an object at pixel (u, v) — where does the robot need to move? This article explains the building blocks behind that conversion and points you to the concrete how-to for each method, so you don’t have to read all of them to pick one.

What you need

Before choosing a method, know the answers to these:

  • Is your camera fixed above the workspace, or mounted on the robot (eye-in-hand)? All the methods below assume a fixed camera looking down at a flat or near-flat workspace, which is the most common student setup.
  • Do you need X/Y only, or also Z (height)? Picking from a single known height only needs 2D. Picking objects of varying height needs a real 3D method or a height-correction trick.
  • How much accuracy do you need? A few millimetres is fine for most gripper pick-ups; sub-millimetre accuracy for small parts needs more calibration points and a proper camera calibration.

Steps

1. Decide what you’re actually converting

2. Check what hardware you have

3. For a 2D camera, pick a plane-based method

All of these assume the object lies on (or a known fixed height above) one reference plane, and turn a pixel into an X/Y coordinate on that plane. They differ in how much setup effort they need and how well they handle rotation between the camera and robot axes.

Method Handles rotation between camera & robot axes? Needs a lens calibration first? Setup effort Article
Manual offset (axes assumed parallel) No — only mirrored axes No Lowest see step 4 below
Single ArUco marker (scale + offset) No No Low How to calibrate a fixed camera to a robot with a single ArUco marker
Least-squares affine fit (≥3, ideally 7+ point pairs) Yes No Medium How to map camera coordinates to robot coordinates with a least-squares fit
ChArUco board + homography (many points, corrects lens distortion) Yes Yes Medium-high How to calibrate a camera and map pixels to world coordinates with a ChArUco board
Two-stage ArUco + homography + affine (built for high precision) Yes Yes High How to do high-precision robot-camera calibration with ArUco markers

As a rule of thumb: start with the least-squares affine fit (How to map camera coordinates to robot coordinates with a least-squares fit) unless you already know you need lens distortion correction (wide-angle lens, or accuracy requirements under a millimetre), in which case go straight to the ChArUco method (How to calibrate a camera and map pixels to world coordinates with a ChArUco board).

4. The manual offset method (no code, good for understanding the problem)

If your camera looks straight down and its X/Y axes are exactly parallel (or exactly mirrored) to the robot’s — no rotation at all — you don’t need a fitted transform. You only need a pixel-to-mm scale and a fixed offset:

  1. Build (or reuse) a camera matrix and get a pixel-to-mm conversion factor: measure the width of the camera image in millimetres and divide it by the image width in pixels.
    x_mm = pixel_x * conversion_factor
y_mm = pixel_y * conversion_factor
  2. Pick a reference point that both the camera and the robot can see, and call it StartPos. Measure how far StartPos is from the robot’s own origin — this distance is the offset.
  3. Check whether the camera’s and robot’s axes point the same way or are mirrored, and write your conversion accordingly, for example:
    pickupX = deltaX + offsetX + array1[0]
pickupY = deltaY + offsetY + array1[1]
    where deltaX/deltaY are the object’s coordinates in mm relative to StartPos, offsetX/offsetY is the offset from step 2, and array1 is the robot’s current position relative to its own starting position (needed if the robot itself moves between captures, e.g. a wrist-mounted camera).
  4. Add the robot’s current position (if it isn’t stationary) to get the final pick coordinate:

Formula of pickup coordinates

Here pickupX/pickupY are the final robot coordinates, deltaX/deltaY are the object’s coordinates in mm (step 1), offsetX/offsetY is the offset from step 2, array1[0]/array1[1] is the robot’s current position relative to its own starting position, and firstposX/firstposY are the StartPos coordinates from step 2.

:warning: Check: this method only works if there is zero rotation between the camera frame and the robot frame — it can only handle mirrored axes, not an arbitrary angle. If your camera is mounted even slightly rotated relative to the robot base, use the least-squares fit (How to map camera coordinates to robot coordinates with a least-squares fit) or a homography-based method instead — both fit rotation, scale and translation together instead of assuming a fixed axis relationship.

Related

Rewritten and consolidated (Sept 2026) from the original student how-to’s: How to go from Camera Pixels to Real-World Coordinates (calibration).

Start here: index of all how-to articles