2 Image Formation
Every perception system starts with a measurement. For most robots that measurement is an image: a grid of numbers produced when light from the world passes through a lens and lands on a sensor. To understand what an image can and cannot tell a robot, we first need to understand how it is made.
This chapter follows light on that journey, from the scene, through the camera, to the numbers in memory. Most sections have a figure you can play with.
In this chapter. The pinhole camera and perspective projection · focal length and field of view · lenses, aperture and depth of field · exposure, noise and dynamic range · how sensors record color · shutters and lens artifacts · the image as data.
2.1 Cameras as artificial eyes
Eyes appeared in the animal kingdom more than 500 million years ago, and nature has since tried many designs: simple light-sensitive patches, pinhole eyes, compound eyes and lens eyes like our own. Cameras follow the same physics. A camera gathers light, focuses it, and converts it into a signal.
Cameras have become the most common robot sensor for good reasons:
- They are cheap and small. A capable camera module costs a few dollars and weighs a few grams.
- They are dense. A single frame contains millions of measurements, far more than a laser rangefinder or an ultrasonic sensor.
- They are passive. A camera only receives light, so it uses little power and does not interfere with other robots.
- They are close to how we see. Much of the world, from road signs to object colors, is designed for eyes.
The price of this density is that a camera does not measure geometry directly. Depth, size and motion must all be inferred, and that inference starts with a model of how the image was formed.
2.2 The pinhole camera
The simplest camera is a dark box with a tiny hole in one side. Light from each point in the scene passes through the hole in a straight line and lands at exactly one point on the back wall. The result is a sharp, upside-down image.
Let a point in the scene be at height \(H\) above the optical axis and distance \(Z\) in front of the pinhole, and let the back wall be at distance \(f\), the focal length. Similar triangles give the height of its image:
\[ h = f \, \frac{H}{Z}. \]
In three dimensions, a point \((X, Y, Z)\) in the camera’s coordinate frame lands at image coordinates
\[ x = f \, \frac{X}{Z}, \qquad y = f \, \frac{Y}{Z}. \]
This is perspective projection, and the division by \(Z\) is its most important feature. Distant objects appear smaller, parallel lines meet at vanishing points, and, crucially, depth is lost.
Press Double size and distance. The image does not change. A person 1.8 m tall standing 6 m away and a statue 3.6 m tall standing 12 m away produce exactly the same picture. This is the ambiguity from Chapter 1 made precise: a single image fixes the ratio \(H/Z\), never \(H\) and \(Z\) separately.
A drone’s camera has a focal length of 4 mm. A gate 1 m wide appears 0.5 mm wide on the sensor. How far away is the gate? What extra information did you need to answer?
2.3 Focal length and field of view
Focal length sets how much of the world the camera sees. For a sensor of width \(w\), the horizontal field of view is
\[ \theta = 2 \arctan\!\left(\frac{w}{2f}\right). \]
A short focal length gives a wide view with small-looking objects. A long focal length narrows the view and magnifies. The scene below was rendered with a 50 mm lens. Increase the focal length to zoom in.
Robots choose focal length by task. A drone avoiding obstacles wants a wide view to see threats from the side; a camera reading distant signs wants a narrow one.
2.4 Why cameras need lenses
A real pinhole faces a hard trade-off. A tiny hole gives a sharp image but lets in very little light. A larger hole lets in more light, but each scene point now spreads over a disc on the sensor, and the image blurs. (Make the hole too small and diffraction blurs the image again.)
A lens escapes this trade-off. It gathers light over a large opening and bends all the rays from one scene point so that they meet again at a single image point. For a thin lens, a point at distance \(Z\) comes into focus at distance \(z'\) behind the lens, where
\[ \frac{1}{f} = \frac{1}{Z} + \frac{1}{z'}. \]
The catch is that only one distance can be in perfect focus at a time. Points nearer or farther focus in front of or behind the sensor and appear as small discs, called circles of confusion.
2.5 Aperture and depth of field
The opening of the lens is the aperture. Its size is given as an f-number,
\[ N = \frac{f}{D}, \]
where \(D\) is the aperture diameter. Larger f-numbers mean smaller openings: f/1.4 is wide open, f/22 is a pinhole-like squint. Each full stop (\(1.4, 2, 2.8, 4, 5.6, \dots\)) halves the light.
The aperture also controls depth of field, the range of distances that look acceptably sharp. A wide aperture gives a shallow depth of field and soft, blurred backgrounds (photographers call this bokeh). A narrow aperture keeps nearly everything sharp.
The scene below was rendered in Blender with a physically based 50 mm camera. Three drones stand 0.6 m, 1.5 m and 4 m away. Choose where to focus, then open and close the aperture.
Depth of field is not only a matter of style. Blur depends on distance, so a camera can use it as a depth cue; later chapters on computational imaging exploit exactly this. For a robot that needs everything sharp, a small aperture helps, but it costs light, which brings us to exposure.
2.6 Exposure
How bright an image looks depends on three settings, often called the exposure triangle:
- Shutter time: how long the sensor collects light. Longer is brighter, but moving things smear into motion blur.
- Aperture: how wide the lens opens. Wider is brighter, but depth of field shrinks.
- ISO (gain): how much the sensor’s signal is amplified. Higher is brighter, but noise is amplified too.
Each of these doubles or halves the light per stop, so different combinations can give the same overall brightness with very different side effects. Try it below. Here the camera is panning sideways, as on a moving robot, so long shutter times show motion blur.
Robots face these trade-offs constantly. A fast-flying drone needs short shutter times to avoid motion blur, so in dim light it must either open the aperture or raise the gain and accept noise.
2.6.1 Dynamic range
A sensor can only record brightness within a limited range. Anything brighter than its maximum saturates to pure white; anything darker than its noise floor is lost in black. The ratio between the brightest and darkest values it can record is its dynamic range. Scenes such as a tunnel exit on a sunny day exceed the range of most cameras. High dynamic range (HDR) imaging captures several exposures and merges them, and some sensors do this within a single frame.
2.7 From light to numbers: the sensor
A digital camera sensor is a grid of millions of light-sensitive sites called photosites. Turning light into an image takes several steps:
- Capture. Photons strike each photosite during the exposure.
- Conversion. Each photosite converts the light into electrical charge, roughly in proportion to the number of photons.
- Color filtering. A filter over each photosite lets through only red, green or blue light.
- Amplification. The signal is amplified according to the ISO setting.
- Digitization. An analog-to-digital converter turns the signal into a number, typically 10 to 14 bits.
- Processing. The camera’s image signal processor (ISP) fills in missing colors, reduces noise, corrects white balance and compresses the result. Every manufacturer does this differently.
Even with no light at all, a sensor reports small random values. Photons also arrive randomly, so even a perfectly uniform scene produces a slightly grainy image. This shot noise grows with the square root of the signal, which is why dim images look noisier.
2.7.1 How sensors see color
Each photosite measures only one color. The most common arrangement, the Bayer pattern, repeats a 2 × 2 tile of one red, two green and one blue filter. Green gets twice as many sites because human vision is most sensitive to green light, which carries most of the detail we perceive.
The camera reconstructs a full-color image by estimating the two missing colors at each pixel from its neighbors, a step called demosaicing. Explore a small patch of the render below.
Look closely at the Demosaiced view near sharp edges. Small colored fringes appear where the estimate guesses wrong. Better demosaicing algorithms, including learned ones, exist precisely to reduce these artifacts.
2.7.2 Try it in code
The cell below builds a small color image, records it through a simulated Bayer filter, and asks OpenCV to demosaic it. Press Run Code. The first run takes a few seconds while OpenCV loads in your browser.
Where in the demosaiced image is the error largest, and why? Try cv2.COLOR_BayerBG2RGB_EA (edge-aware demosaicing) and compare the largest error.
2.8 Shutters and motion
Many low-cost sensors do not capture the whole frame at once. A rolling shutter reads the rows one after another from top to bottom, so each row sees the scene at a slightly different moment. A global shutter exposes every row at the same instant.
For a still scene the difference is invisible. For a spinning propeller, or a camera on a vibrating drone, it is dramatic. On the left is the propeller as it really moves (slowed down 30 times); on the right is what a camera captures. Speed it up, then switch on the global shutter.
Rolling-shutter distortion bends straight lines and breaks geometric algorithms that assume one moment per image. Robots that move fast, or that rely on precise geometry, often pay more for global-shutter cameras.
2.9 Lens imperfections
Real lenses are not ideal. Two imperfections matter often:
- Geometric distortion. Straight lines bow outward (barrel distortion, common in wide-angle lenses) or inward (pincushion). It can be measured and corrected through calibration, which a later chapter covers.
- Chromatic aberration. Glass bends different wavelengths by different amounts, so a lens cannot focus red, green and blue at exactly the same place. Edges, especially near the image corners, show colored fringes.
2.10 The image as data
After all of this, what reaches the robot’s software is an array of numbers. A grayscale image of height \(H\) and width \(W\) is a matrix \(I \in \{0, \dots, 255\}^{H \times W}\). A color image stacks three such matrices, one per channel. Turn on your camera to see your own image split into its red, green and blue channels. Hold up something brightly colored and watch which channel lights up.
Video is processed in your browser and never uploaded.
The cell below builds an image from scratch and shows its channels. Change the shapes or colors and run it again.
Red, green and blue are not the only way to describe color. Other color spaces separate brightness from color, which is often more useful for perception: HSV describes hue, saturation and value, and Lab is designed so that distances match perceived differences. OpenCV converts between them with cv2.cvtColor.
Convert the image above to grayscale with cv2.cvtColor(img, cv2.COLOR_RGB2GRAY). Which shape looks brighter, and why? Then convert it to HSV and display the hue channel.
2.11 Summary
- A camera projects the 3D world onto a 2D sensor. Perspective projection, \(x = fX/Z\), divides by depth, so a single image cannot separate size from distance.
- Focal length sets the field of view. Lenses gather light without the blur of a large pinhole, but focus only one distance perfectly.
- Aperture, shutter time and gain together set exposure, each with a side effect: shallow depth of field, motion blur or noise.
- Sensors measure one color per photosite through a Bayer filter and reconstruct full color by demosaicing.
- Rolling shutters, lens distortion and chromatic aberration all bend the simple pinhole model, and robots must account for them.