5 Camera Models and Calibration
In Chapter 2 we met the pinhole camera, where a point at depth \(Z\) lands at \(x = fX/Z\). That model assumed the camera sat at the origin, looked straight down its own axis, and measured positions in millimeters on the sensor. A robot needs more. Its camera is mounted somewhere on the robot, points in some direction, reports positions in pixels, and looks through a lens that bends straight lines.
This chapter builds the complete camera model that perception systems use every day, and shows how to measure its parameters for a real camera. By the end you will be able to take any 3D point in the world and say exactly which pixel it lands on.
In this chapter. Coordinate frames · 3D rotations, Euler angles and gimbal lock · extrinsics \([R \mid t]\) · intrinsics \(K\) · the projection matrix \(P = K[R \mid t]\) · lens distortion · calibration with a checkerboard · practical tips for robots.
5.1 Four coordinate frames
Every camera model is a chain of coordinate changes. Four frames appear again and again:
- World frame. A fixed frame attached to the environment, such as the corner of a room or the robot’s starting position. In robotics it usually has \(x\) forward, \(y\) left and \(z\) up.
- Camera frame. Attached to the camera, with its origin at the center of projection. By the convention used in OpenCV and most vision software, \(x\) points right, \(y\) points down and \(z\) points forward along the optical axis.
- Image plane (normalized coordinates). The plane \(z = 1\) in front of the camera. A camera-frame point \((X_c, Y_c, Z_c)\) lands at \((x, y) = (X_c / Z_c,\; Y_c / Z_c)\), the pinhole projection with \(f = 1\).
- Pixel frame. Integer pixel positions \((u, v)\), with the origin at the top-left corner of the image, \(u\) to the right and \(v\) down.
The full camera model is the journey through these four frames:
\[ \underbrace{\mathbf{X}_w}_{\text{world}} \;\xrightarrow{\;R,\,t\;}\; \underbrace{\mathbf{X}_c}_{\text{camera}} \;\xrightarrow{\;\div Z_c\;}\; \underbrace{(x, y)}_{\text{image plane}} \;\xrightarrow{\;K\;}\; \underbrace{(u, v)}_{\text{pixels}}. \]
Step through it below with a real camera, one transformation at a time. Move the point with the sliders and watch every number update.
- 1 · World frame
- 2 · Camera frame (extrinsics)
- 3 · Image plane (divide by depth)
- 4 · Pixels (intrinsics)
The camera here sits 1.2 m above the world origin, turned 10° to the left and tilted 5° down, with \(f_x = f_y = 600\) pixels and the principal point at the image center. The next sections explain each step.
5.2 Rotations in 3D
A rigid motion in 3D is a rotation followed by a translation. Rotations are the harder half.
5.2.1 Rotation matrices
A rotation is a \(3 \times 3\) matrix \(R\) whose columns are orthonormal and whose determinant is \(+1\):
\[ R^\top R = I, \qquad \det R = 1. \]
The columns of \(R\) are the axes of the rotated frame written in the original frame. Because \(R^{-1} = R^\top\), undoing a rotation costs nothing. Rotations about single axes are the building blocks:
\[ R_x(\phi) = \begin{bmatrix} 1 & 0 & 0 \\ 0 & \cos\phi & -\sin\phi \\ 0 & \sin\phi & \cos\phi \end{bmatrix},\quad R_y(\theta) = \begin{bmatrix} \cos\theta & 0 & \sin\theta \\ 0 & 1 & 0 \\ -\sin\theta & 0 & \cos\theta \end{bmatrix},\quad R_z(\psi) = \begin{bmatrix} \cos\psi & -\sin\psi & 0 \\ \sin\psi & \cos\psi & 0 \\ 0 & 0 & 1 \end{bmatrix}. \]
A rotation matrix has nine numbers but only three degrees of freedom. Several smaller parameterizations exist, each with strengths and pitfalls.
5.2.2 Euler angles and gimbal lock
The most intuitive description chains three single-axis rotations. Aircraft and drones use yaw \(\psi\) about \(z\), pitch \(\theta\) about \(y\) and roll \(\phi\) about \(x\):
\[ R = R_z(\psi)\, R_y(\theta)\, R_x(\phi). \]
Euler angles are easy to read but have a famous flaw. When the pitch reaches \(\pm 90°\), the first and last rotations act about the same axis, and the three angles can no longer describe every orientation. This is gimbal lock. Try it: press Pitch to 90°, then Add 20° to yaw and roll.
Gimbal lock is not a curiosity. A flight controller that integrates Euler angles can misbehave during aggressive maneuvers, and an estimator near the singularity becomes numerically unstable. That is why the internals of most robot software use one of the representations below.
5.2.3 Axis-angle and Rodrigues’ formula
Euler’s rotation theorem says that every rotation is a single turn by an angle \(\theta\) about some unit axis \(\mathbf{k}\). Packing the two into one vector \(\boldsymbol{\omega} = \theta \mathbf{k}\) gives a compact, singularity-free description for small and moderate rotations. Rodrigues’ formula converts it to a matrix:
\[ R = I + \sin\theta\, [\mathbf{k}]_\times + (1 - \cos\theta)\, [\mathbf{k}]_\times^2, \qquad [\mathbf{k}]_\times = \begin{bmatrix} 0 & -k_z & k_y \\ k_z & 0 & -k_x \\ -k_y & k_x & 0 \end{bmatrix}. \]
OpenCV uses this form for camera poses: the rvec returned by calibration and pose estimation is exactly \(\boldsymbol{\omega}\), and cv2.Rodrigues converts between the two. The figure above shows the axis-angle equivalent of every orientation you set.
5.2.4 Quaternions
A unit quaternion \(q = (w, x, y, z)\) with \(w = \cos(\theta/2)\) and \((x, y, z) = \sin(\theta/2)\,\mathbf{k}\) stores the same axis and angle in four numbers. Quaternions never suffer gimbal lock, compose with a cheap multiplication, and interpolate smoothly, which is why state estimators, flight controllers and game engines use them. Their one quirk is that \(q\) and \(-q\) describe the same rotation.
5.3 Extrinsics: where the camera is
The extrinsic parameters describe the camera’s pose: a rotation \(R\) and a translation \(\mathbf{t}\) that carry a point from world coordinates into camera coordinates,
\[ \mathbf{X}_c = R\, \mathbf{X}_w + \mathbf{t}. \]
It is easy to get the direction wrong. \(R\) and \(\mathbf{t}\) map world to camera. If the camera’s center sits at \(\mathbf{C}\) in the world, then \(\mathbf{t} = -R\,\mathbf{C}\), and the camera’s own axes, written in the world frame, are the rows of \(R\). In homogeneous coordinates (Chapter 4) the extrinsics become a single \(3 \times 4\) matrix \([R \mid \mathbf{t}]\) acting on \((X, Y, Z, 1)\).
5.4 Intrinsics: how the camera sees
The intrinsic parameters map normalized image coordinates \((x, y)\) to pixels. They collect into the camera matrix
\[ K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}, \qquad \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}. \]
- \(f_x\) and \(f_y\) are the focal length measured in pixels. If the lens has focal length \(f\) in millimeters and each pixel is \(p\) millimeters wide, \(f_x = f / p\). A 4 mm lens on a sensor with 2 µm pixels has \(f_x = 2000\).
- \((c_x, c_y)\) is the principal point, where the optical axis meets the sensor. It is close to the image center but rarely exactly at it.
- Different values of \(f_x\) and \(f_y\) mean non-square pixels, and the skew \(s\) models axes that are not perpendicular. On modern sensors \(f_x \approx f_y\) and \(s \approx 0\).
Change each parameter and watch what it does to the image of a checkerboard held one meter from the camera.
5.5 The projection matrix
Putting the pieces together gives the complete pinhole camera model in one line:
\[ \lambda \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K \,[\,R \mid \mathbf{t}\,] \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix} = P \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}, \]
where \(\lambda = Z_c\) is the depth of the point and \(P = K[R \mid \mathbf{t}]\) is the \(3 \times 4\) projection matrix. It has 11 degrees of freedom: 5 intrinsic and 6 extrinsic (it is only defined up to scale). Compare with Chapter 2: the pinhole formula \(x = fX/Z\) is this equation with \(R = I\), \(\mathbf{t} = 0\), no principal point offset and units of millimeters.
Below, the left panel shows the world from outside: three boxes, a floor grid, the world axes and the camera with its viewing frustum. The right panel shows what the camera sees, computed with \(P = K[R \mid \mathbf{t}]\). Move the camera, turn it and change its intrinsics. Drag the left panel to look around.
- Raise the camera and pitch it down. Which entries of \(R\) change, and which of \(\mathbf{t}\)?
- Increase the focal length and watch the frustum narrow in the left panel while the boxes grow in the right one.
- Move the principal point. The frustum tilts sideways even though the camera does not rotate: shifting \(c_x\) is like shifting the sensor behind the lens.
5.6 Lens distortion
Real lenses bend light in ways the pinhole model ignores. The most important effect is radial distortion: points are pushed toward or away from the image center by an amount that grows with their distance \(r\) from it. Lenses can also be slightly tilted relative to the sensor, which adds tangential distortion. The standard model, used by OpenCV, applies to normalized coordinates \((x, y)\) with \(r^2 = x^2 + y^2\):
\[ \begin{aligned} x_d &= x\,(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + 2 p_1 x y + p_2 (r^2 + 2x^2), \\ y_d &= y\,(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + p_1 (r^2 + 2y^2) + 2 p_2 x y. \end{aligned} \]
Negative \(k_1\) gives barrel distortion, common in the wide-angle lenses robots favor; positive \(k_1\) gives pincushion distortion. The distorted coordinates then pass through \(K\) as before. Very wide fisheye lenses need a different model altogether (OpenCV’s cv2.fisheye module).
Bend the drone scene from Chapter 2 with your own lens below. The gold grid shows where straight lines in the world end up. Then press Undistort: if you know the parameters, the distortion can be removed exactly.
The distortion model is easy to apply but has no closed-form inverse, so undistortion solves for \((x, y)\) given \((x_d, y_d)\) numerically, here by a few rounds of fixed-point iteration. In practice OpenCV precomputes the mapping once with cv2.initUndistortRectifyMap and then undistorts every frame with a fast lookup, cv2.remap.
5.7 Calibration
Calibration measures \(K\) and the distortion coefficients for a real camera. The standard approach, introduced by Zhengyou Zhang in 2000, needs nothing more than a printed checkerboard.
5.7.1 Zhang’s method
The idea is to photograph a flat pattern whose geometry is known exactly, from several viewpoints.
- Place the board at \(Z = 0\). In the board’s own frame every corner has \(Z_w = 0\), so the projection equation loses a column and becomes a \(3 \times 3\) homography between the board and the image: \[ \lambda \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K\,[\,\mathbf{r}_1 \;\; \mathbf{r}_2 \;\; \mathbf{t}\,] \begin{bmatrix} X \\ Y \\ 1 \end{bmatrix} = H \begin{bmatrix} X \\ Y \\ 1 \end{bmatrix}. \]
- Estimate one homography per view. \(H\) has 8 degrees of freedom and each detected corner gives two equations, so four corners suffice; a real board gives dozens, solved by least squares as in Chapter 4.
- Use the rotation’s structure. Because \(\mathbf{r}_1\) and \(\mathbf{r}_2\) are orthonormal, each homography gives two linear constraints on the symmetric matrix \(B = K^{-\top} K^{-1}\). With three or more views the constraints determine \(B\), and a Cholesky factorization recovers \(K\).
- Refine everything. With \(K\) known, each view’s \(R\) and \(\mathbf{t}\) follow from its homography. A nonlinear least-squares optimization then adjusts all parameters, including distortion, to minimize the reprojection error.
5.7.2 Reprojection error
The quality of a calibration is measured by the reprojection error: project the known board corners through the estimated model and measure, in pixels, how far they land from the detected corners,
\[ E = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \big\lVert \mathbf{u}_i - \pi(K, \mathbf{d}, R_j, \mathbf{t}_j, \mathbf{X}_i) \big\rVert^2 }. \]
A good calibration of a typical camera reaches well under one pixel. A low error is necessary but not sufficient: if every image shows the board in the middle of the frame, the error can look excellent while the distortion near the edges is poorly constrained.
5.7.3 Choosing the views
Which views you collect matters as much as how many. Capture board poses below and watch what the calibration needs.
5.7.4 Calibrate a camera in your browser
The cell below runs a complete calibration with OpenCV. It simulates a camera with a known \(K\) and barrel distortion, photographs a 9 × 6 checkerboard from fifteen random poses, adds a little detection noise, and then hands only the 2D corners to cv2.calibrateCamera. Compare what it recovers with the truth. Press Run Code; the first run takes a few seconds while OpenCV loads.
Change the loop so it keeps only views with a small tilt, for example rvec = rng.uniform(-0.05, 0.05, 3). What happens to the estimated focal length and to the reprojection error? Why can a near-perfect reprojection error still hide a bad calibration?
On a real camera the only change is where the 2D corners come from. You photograph a printed board and detect its corners with cv2.findChessboardCorners, refine them to sub-pixel accuracy with cv2.cornerSubPix, and pass them to the same function.
5.8 Calibration in practice
A few lessons that save robotics projects:
- Calibrate at the settings you will use. Focus, zoom and even some aperture changes alter the intrinsics. A camera that refocuses (autofocus) or is bumped needs recalibration; many teams lock focus with a drop of glue.
- Cover the whole image and tilt the board. Corners near the edges constrain distortion, and tilted views separate focal length from distance, as the coverage figure above showed.
- Keep the board flat and the images sharp. A warped board or motion blur biases the result. Good lighting lets you use a short shutter time (Chapter 2).
- Mind the shutter. Rolling-shutter cameras distort moving boards. Hold the board still, or use a calibration tool that models rolling shutter.
- Calibrate the extrinsics to other sensors too. A robot rarely uses a camera alone. The rigid transform between the camera and an IMU, a lidar or a second camera must also be measured, with tools such as Kalibr for camera and IMU pairs. Even event cameras, which have no frames at all, are calibrated with the same model by first reconstructing images from their events.
5.9 Summary
- A camera model is a chain of frames: world, camera, image plane and pixels.
- Rotations have three degrees of freedom. Euler angles are intuitive but suffer gimbal lock; axis-angle vectors and quaternions avoid it.
- Extrinsics \([R \mid \mathbf{t}]\) say where the camera is; intrinsics \(K\) say how it turns directions into pixels. Together, \(P = K[R \mid \mathbf{t}]\).
- Lens distortion bends straight lines. It is modeled with radial and tangential coefficients and removed by undistortion.
- Zhang’s method calibrates a camera from several views of a planar checkerboard, and the reprojection error measures how well the model fits.