Based on contributions by Jawara.
If you don’t want to run a full lens/camera calibration, you can fit a 2D transform (rotation, scale and translation combined) directly from a handful of camera-coordinate ↔ robot-coordinate point pairs, using least squares. This is the quickest way to get a robust camera-to-robot mapping for a flat work plane.
What you need
- A stationary camera and vision pipeline that reports a consistent
(x, y)per detected point. - A robot you can jog to a position and read the TCP pose from.
- At least 3 point pairs (the transform has 6 free parameters); 7 or more, spread across the workspace, is recommended for a reliable fit.
- Python with
numpyandmatplotlib.
Steps
1. Set up a fixed camera and a robot capture position
This method assumes a stationary camera. If the camera is mounted on the robot arm instead, define a fixed capture position the robot returns to before every image.
2. Collect matching point pairs
For each of several positions (aim for 7):
- Move an item into view and read its camera coordinates.
- Manually jog the robot to where it should be to pick up the item at that position, keeping any tool offset consistent. Get the robot’s TCP as close as possible to the item for the best accuracy.
- Record the pair (camera coordinates, robot coordinates).
Repeat with the item in different positions spread across the workspace:
3. Fit the transform with least squares
The transform has the form:
xd = a * xp + b * yp + c
yd = d * xp + e * yp + f
where xp, yp are camera coordinates and xd, yd are robot coordinates. Fill in your own point pairs in the A and B arrays below (the example values are from a real run and can be replaced):
import numpy as np
import matplotlib.pyplot as plt
# Camera-space coordinates, homogeneous form [x, y, 1] -- add more rows for a better fit
A = np.array([
[269, 241, 1],
[435, 350, 1],
[269, 110, 1],
[270, 372, 1],
])
# Matching robot coordinates [x, y], same order as A
B = np.array([
[-0.45392306201597526, 0.008666534496233558],
[-0.6332256232813833, 0.12730990848339344],
[-0.4661438800601707, -0.12857826313800147],
[-0.45911760805987534, 0.14637253602441908],
])
# Solve xd = a*xp + b*yp + c , yd = d*xp + e*yp + f
coefficients, _, _, _ = np.linalg.lstsq(A, B, rcond=None)
a, b, c = coefficients[:, 0]
d, e, f = coefficients[:, 1]
print("Coefficients:")
print(f"a = {a}\nb = {b}\nc = {c}")
print(f"d = {d}\ne = {e}\nf = {f}")
# Check the fit against the input data
transformed_coords = A @ coefficients
print("\nTransformed coordinates vs. actual robot coordinates:")
for i in range(len(B)):
print(f"Transformed: {transformed_coords[i]} vs. actual: {B[i]}")
mse = np.mean((B - transformed_coords) ** 2)
distances = np.linalg.norm(B - transformed_coords, axis=1)
print(f"\nMean squared error: {mse}")
print(f"Average Euclidean distance (error, same units as B): {distances.mean():.4f}")
# Plot: circles (actual) and triangles (transformed) should overlap closely
fig, ax = plt.subplots()
ax.scatter(B[:, 0], B[:, 1], c='r', marker='o', label='Actual robot coords')
ax.scatter(transformed_coords[:, 0], transformed_coords[:, 1], c='b', marker='^', label='Transformed coords')
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.legend()
plt.show()
Example output for the four points above:
Coefficients:
a = -0.0010677615453140213
b = 3.094561948991097e-05
c = -0.17959680557618776
d = 2.482562688915765e-05
e = 0.0010493343791252749
f = -0.2507558317896495
Mean squared error: 5.571609920914277e-06
Average Euclidean distance (error, same units as B): 0.0027
In the plot, you want the circles and triangles to overlap closely. Judge the fit by the average Euclidean distance, not a raw MSE-derived percentage — it’s directly in the same units as your robot coordinates (metres or mm) and tells you the typical positioning error you can expect.
4. Use the fitted coefficients in your pick routine
Once you’re happy with the fit, hardcode the six coefficients and reuse them without recomputing:
def transform_coordinates(xp, yp,
a=-0.0010677615453140213, b=3.094561948991097e-05, c=-0.17959680557618776,
d=2.482562688915765e-05, e=0.0010493343791252749, f=-0.2507558317896495):
"""Apply a previously fitted affine transform to a new camera-space point (xp, yp)."""
xd = a * xp + b * yp + c
yd = d * xp + e * yp + f
return xd, yd
Common mistakes
- Too few points. With only 3 points the fit is exact but not “least squares” in any meaningful sense — any detection noise goes straight into the coefficients. Use 7+ points spread across the whole workspace, not clustered in one corner or colinear.
- Inconsistent tool offset. If the robot’s active TCP/tool offset changes between captures, the recorded robot coordinates won’t be consistent and the fit will be poor no matter how many points you add.
- Camera moved between captures. Any camera movement after starting to collect points invalidates all previous points.
- This fits a single flat plane (2D only). For pick points at varying heights, use the ChArUco method’s height correction (How to calibrate a camera and map pixels to world coordinates with a ChArUco board) or a depth camera (How to convert a RealSense camera pixel to a 3D point and robot coordinates).
Related
- Choosing a method to convert camera pixels to robot coordinates — choosing a calibration method
- How to calibrate a fixed camera to a robot with a single ArUco marker — single ArUco marker quick calibration
- How to calibrate a camera and map pixels to world coordinates with a ChArUco board — ChArUco board + homography (handles lens distortion and multiple heights)
- How to do high-precision robot-camera calibration with ArUco markers — high-precision two-stage calibration
Rewritten and consolidated (Sept 2026) from the original student how-to’s: How to translate Robot TCP from Camera Coords using the Least Squared Method.