9 Learning Depth
A single image cannot fix the size and distance of what it shows: Chapter 2 proved it with similar triangles, and Chapter 7 showed how far geometry alone can go. Yet people judge depth from one photograph all the time. We use experience: floors recede, doors have a familiar height, distant things look hazy and small. A neural network can learn the same kind of experience from data.
This chapter builds the ideas needed to train a network that predicts depth from images: how networks learn, why convolutional encoder-decoders suit dense prediction, how to measure success, and why a camera that deliberately codes its blur can make the network’s job far easier.
In this chapter. Depth as a learning problem · neurons, layers and training · gradient descent · convolutional networks and receptive fields · encoder-decoders and U-Net · losses and metrics for depth · coded apertures and simulated training data · limits of learned depth.
9.1 Depth as a learning problem
The task is easy to state. Given an image \(I\) of size \(H \times W\), predict a depth map \(D\) of the same size, with one distance in metres for every pixel:
\[ \hat{D} = f_\theta(I), \qquad f_\theta : \mathbb{R}^{H \times W \times 3} \rightarrow \mathbb{R}^{H \times W}. \]
Here \(f_\theta\) is a neural network and \(\theta\) are its parameters, often millions of numbers. We do not write \(f_\theta\) by hand. We show it many pairs of images and true depth maps and let an optimizer adjust \(\theta\) until predictions match. This is supervised learning.
Two warnings come with the approach. First, a monocular network can only be as good as the cues in its training data: it learns that cars are about 1.5 m tall, not why. Second, scale is fragile. A network trained indoors may predict the right shape of an outdoor scene but the wrong overall size, the same scale ambiguity we met in Chapter 2. We will return to both.
9.2 How a network learns
9.2.1 Neurons and layers
A neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear activation function \(\sigma\):
\[ a = \sigma\!\left(\sum_i w_i x_i + b\right). \]
Common activations are the rectified linear unit \(\text{ReLU}(z) = \max(0, z)\) and the hyperbolic tangent. A layer applies many neurons to the same inputs, and a network stacks layers so that each one builds on the features computed by the last. Without the nonlinearity, any stack of layers would collapse into a single linear map; with it, a network can approximate almost any function.
9.2.2 Loss and gradient descent
Training needs a number that says how wrong the network is: the loss. For regression, a simple choice is the mean squared error over \(N\) examples,
\[ \mathcal{L}(\theta) = \frac{1}{N} \sum_{n=1}^{N} \big(f_\theta(x_n) - y_n\big)^2 . \]
Gradient descent lowers the loss by repeatedly stepping against its gradient,
\[ \theta \leftarrow \theta - \eta \, \nabla_\theta \mathcal{L}, \]
where \(\eta\) is the learning rate. Backpropagation computes \(\nabla_\theta \mathcal{L}\) efficiently with the chain rule, passing error signals backwards from the output to every weight. In practice the gradient is estimated on small random mini-batches, and improved optimizers such as Adam adapt the step size for each parameter.
Train a small network yourself. It has one input, one hidden layer of tanh neurons and one output, and it learns from the gold training points. The hollow points are held out for validation: the network never trains on them, so they reveal whether it has learned the curve or memorized the noise.
training pointsvalidation pointsnetwork predictiontrue function
Things to try: with one or two neurons, the wave cannot be fitted however long you train (underfitting). With 32 neurons and high label noise, training loss keeps falling while validation loss turns upward (overfitting). A learning rate that is too high makes the loss jump around instead of settling.
9.2.3 The shape of the loss
The loss of a real network lives in millions of dimensions, but a two-dimensional surface already shows the main behaviours. Drag the gold point to choose a starting value for the two parameters, set the learning rate, and descend.
In the narrow valley, a learning rate small enough to stay stable across the steep direction crawls along the shallow one, while a larger rate zigzags and then diverges. Momentum keeps a running average of past steps, which damps the zigzag and speeds the crawl. In the two valleys, where you start decides which minimum you reach: a reminder that training finds a good solution, not necessarily the best one.
9.3 Convolutional networks
Connecting every pixel to every neuron would need billions of weights for even a small image, and it would ignore what we know about images: nearby pixels belong together, and a pattern means the same thing wherever it appears. A convolutional layer builds both facts in. It slides a small learned kernel across the image, exactly like the filters of Chapter 3, and reuses the same weights at every position. A layer with 64 kernels of size \(3 \times 3\) over 32 input channels has only \(64 \times 3 \times 3 \times 32 \approx 18{,}000\) weights, whatever the image size.
Stacking convolutions lets later layers see more of the image. The region of the input that can influence one output value is its receptive field. Depth needs context: to know that a patch of floor is far away, a unit must see the room, not one tile. Downsampling (pooling or strided convolution) is what makes the receptive field grow quickly.
With five \(3 \times 3\) layers and no downsampling, each output sees only an \(11 \times 11\) patch. Halve the resolution after every layer and the same five layers see \(63 \times 63\).
9.4 Encoder-decoders and U-Net
Depth estimation is a dense prediction task: the output is an image, not a single label. The standard architecture has two halves:
- The encoder repeatedly convolves and downsamples, trading spatial detail for context. At the bottom, the bottleneck is a small, deep summary of what is in the scene and roughly where.
- The decoder reverses the process, upsampling step by step back to full resolution to produce one depth value per pixel.
The bottleneck has lost the fine detail needed for sharp object boundaries. U-Net, first developed for biomedical image segmentation, fixes this with skip connections: each decoder level also receives the encoder’s feature map from the same resolution, so it can combine what (from the bottleneck) with exactly where (from the encoder). Hover over the blocks, then turn the skip connections off.
The depth maps on the right are illustrative rather than outputs of a trained network: the “without skips” view is the true depth reduced 16 times and smoothly enlarged, which is roughly all the bottleneck can describe. Real networks without skips show exactly this smearing at depth edges.
9.5 Measuring depth predictions
9.5.1 Losses for training
The L1 loss \(\frac{1}{N}\sum_i |\hat{d}_i - d_i|\) and the L2 loss \(\frac{1}{N}\sum_i (\hat{d}_i - d_i)^2\) compare predicted and true depth pixel by pixel. L1 is less sensitive to a few large errors, which are common at object boundaries, and is a frequent default.
Because a monocular network struggles most with the overall scale, Eigen and colleagues proposed a scale-invariant loss on log depth. With \(g_i = \log \hat{d}_i - \log d_i\),
\[ \mathcal{L}_{\text{SI}} = \frac{1}{N}\sum_i g_i^2 \; - \; \frac{\lambda}{N^2}\Big(\sum_i g_i\Big)^2 . \]
With \(\lambda = 1\), multiplying every prediction by the same factor adds a constant to every \(g_i\) and leaves the loss unchanged: the network is judged only on the shape of the scene. Values of \(\lambda\) between 0 and 1 blend scale-invariant and absolute error.
9.5.2 Metrics for evaluation
Papers report a standard set of numbers, each over all valid pixels:
| Metric | Definition | Better |
|---|---|---|
| AbsRel | \(\frac{1}{N}\sum_i \lvert \hat{d}_i - d_i \rvert / d_i\) | lower |
| RMSE | \(\sqrt{\frac{1}{N}\sum_i (\hat{d}_i - d_i)^2}\) (metres) | lower |
| \(\delta < 1.25\) | fraction of pixels with \(\max(\hat{d}_i/d_i,\, d_i/\hat{d}_i) < 1.25\) | higher |
AbsRel weighs errors relative to distance, so a 10 cm error matters more at 1 m than at 10 m. RMSE is in metres and dominated by the largest errors. \(\delta < 1.25\) asks a simple question: how often is the prediction within 25 % of the truth?
The scene below was rendered in Blender together with its exact depth map, the kind of perfect ground truth that only simulation provides. The prediction is simulated: use the sliders to give it a wrong overall scale, noise, or smeared boundaries, and watch how each metric responds. Move over either image to read depths in metres.
Move over an image to read the depth.
Set the scale error to 1.3. AbsRel and RMSE grow sharply, but the scale-invariant loss does not move. Now switch on median scaling, which rescales the prediction so its median ratio to the truth is 1, as many papers do for methods that cannot know absolute scale. The errors fall back. Median scaling is fair for comparing shapes, but a robot that needs to stop 30 cm from a wall needs metric depth, with no rescaling allowed.
9.5.3 Try it in code
The cell below computes the metrics and both losses for a small depth map, then confirms that the scale-invariant loss ignores a global scale. It uses only NumPy.
Change lam to 0.5 and run the cell again. Why does the scale-invariant loss now change when the prediction is scaled? Then make the box prediction wrong (for example pred[20:36, 18:34] = 3.0) and see which metric reacts most.
9.6 Coding the aperture
A monocular network has to infer depth from context alone. Depth from defocus offers a physical cue: as Chapter 2 showed, blur grows with distance from the focal plane. A network can learn to read that blur. But an ordinary circular aperture leaves two problems:
- Sign ambiguity. A point slightly in front of the focal plane and one slightly behind it produce the same disc of blur.
- Weak information. A disc is a low-pass filter, so heavy blur erases the very texture that would reveal how blurred the image is.
A coded aperture places a patterned mask in the lens, so the blur of every point takes the shape of the pattern, scaled by its defocus. An asymmetric pattern flips as a point passes through the focal plane, which removes the sign ambiguity, and a well-chosen pattern preserves more image detail at each scale. Approaches such as PhaseCam3D go further and learn the optical element jointly with the network.
The figure below simulates both apertures from the rendered colour image and depth map. Choose the focus distance and compare the blur patterns (the point-spread functions, or PSFs) at four depths.
Blur pattern at each depth
9.6.1 Simulating training data
Real coded cameras are rare, but coded images are easy to simulate, and that is how such networks are usually trained. The recipe has three steps:
- Compute the PSF of the coded lens at a range of depths, from the mask pattern and the lens optics.
- Render a coded image from an all-in-focus (AiF) colour image and its depth map: blur each depth layer with its PSF and combine the layers.
- Train a supervised network to map coded images back to metric depth.
The second step needs care. The simplest linear model blurs each depth layer and adds the results. It is fast, but it ignores occlusion: at the edge of a near object, background colour leaks over the foreground. A layered (non-linear) model composites the blurred layers from far to near, letting nearer layers cover what lies behind them, much like alpha compositing in graphics. Switch the image model in the figure above and look closely at the edges of the nearest drone.
Datasets matter just as much. Indoor benchmarks such as NYU Depth V2, captured with a Microsoft Kinect, provide real images with measured depth. Synthetic datasets rendered from 3D scenes (see Chapter 6) provide perfect depth and unlimited variety, and they make it possible to simulate any lens, including ones that do not exist yet. Results from the author’s group show that a network trained on coded images from only about 2,000 examples, mixing real and simulated data, can estimate metric depth in scenes it has never seen, because it learns the geometry of the lens rather than the appearance of particular rooms.
9.7 A taste of training, in code
Real depth networks are trained with deep-learning frameworks on GPUs. To see the mechanics without any of that, the cell below trains a one-hidden-layer network with plain NumPy and hand-written backpropagation, on the same kind of toy problem as the playground.
Set lr = 1.5 and run again. What happens to the loss curve, and how does it match what you saw on the loss surface earlier? Then try H = 2.
9.8 Limits of learned depth
Learned depth is powerful and widely used, but a robot designer should know where it breaks:
- Domain shift. A network trained indoors may fail outdoors, under unusual lighting, or with a different camera. Its errors are often confident.
- Scale. Monocular networks learn scale from object sizes in training data. Change the camera’s focal length or field of view and the scale can drift. Physical cues such as coded blur, stereo (Chapter 10) or motion tie depth to the sensor rather than to habits learned from data.
- Thin and reflective things. Wires, glass, mirrors and dark surfaces are underrepresented in data and hard for every depth sensor.
- No guarantees. Unlike triangulation, a network gives no error bound unless it is trained to estimate its own uncertainty.
The best systems combine learned priors with geometry and optics, which is the theme of the chapters ahead.
9.9 Summary
- Depth from a single image is ambiguous in principle, but networks can learn the cues that make it solvable in practice.
- Networks are stacks of weighted sums and nonlinearities, trained by gradient descent on a loss, with gradients from backpropagation.
- Convolutions share weights across the image; downsampling grows the receptive field so units see context.
- Encoder-decoders produce dense outputs; U-Net’s skip connections restore sharp boundaries.
- L1 and scale-invariant losses train depth networks; AbsRel, RMSE and \(\delta < 1.25\) evaluate them, and median scaling hides scale errors that robots cannot ignore.
- Coded apertures add a physical, sign-resolving depth cue; simulated coded images from RGB and depth make training practical, and a layered image model handles occlusion edges correctly.