1 Introduction

A robot that cannot perceive cannot act with purpose. Before it can grasp an object, cross a room or fly through a gap, it has to work out what is around it, where things are and how they are moving. This book is about how robots do that: how they turn light and other signals into an understanding of the world that is good enough to act on.

This chapter sets the stage. It answers four questions: what perception is, why it is hard, who it matters to, and why the subject is changing so quickly in the current era of artificial intelligence.

1.1 What is perception?

Philosophers have debated the word for centuries. For engineers, a working definition is enough:

Perception is the organization, identification and interpretation of sensory information in order to represent and understand the environment.

Three parts of this definition matter for robots:

Sensory information. The raw input is a measurement, not a description. A camera records how much light reached each pixel. It does not record that there is a chair in the room.
Representation. Perception produces something the rest of the system can use: a depth map, a set of detected objects, a 3D model, an estimate of the robot’s own motion.
Understanding for a purpose. A robot does not perceive for its own sake. It perceives so that it can decide and act.

That last point separates robot perception from the wider field of computer vision. Computer vision asks what an image contains. Robot perception asks what the robot needs to know, right now, to do its job safely. The two share most of their tools, and this book draws on both.

Figure 1.1: Perception closes the loop between sensing and acting. What the robot does changes what it senses next.

1.2 Why is perception hard?

People see effortlessly, so it is easy to underestimate the problem. A five-year-old recognizes a cup from any angle, in any light, half hidden behind a book. For decades this was far beyond any computer, even as computers became superhuman at chess and arithmetic. Hans Moravec observed this pattern in the 1980s: the abilities that feel easiest to us are often the hardest to build.

Several reasons stand out.

A camera loses information. An image is a two-dimensional projection of a three-dimensional world. Depth is discarded at the moment of capture, and many different scenes produce exactly the same image. A small object nearby and a large object far away can look identical. Recovering the world from images is therefore an inverse problem without a unique answer, and every perception system relies on assumptions, priors or extra measurements to resolve the ambiguity.

Try it: find the object

There is an object in the picture below. Can you find it? Then press Play motion and watch it again.

A surface covered in newspaper clippings, with an object hidden somewhere on it — A camouflaged object is almost invisible in a single image. The moment it moves, it stands out. Motion carries information that one picture cannot.

A camouflaged object is almost invisible in a single image. The moment it moves, it stands out. Motion carries information that one picture cannot.

The same thing never looks the same twice. Lighting, viewpoint, occlusion, motion blur, sensor noise, reflections and weather all change the measurement while the underlying object stays the same. A useful perception system must be invariant to the changes that do not matter and sensitive to the ones that do.

There is too much data and too little time. Cameras produce millions of measurements per frame, many times per second. Biological vision copes by being highly selective: the eye samples finely only at the center of gaze, and the brain processes only a small fraction of what reaches the retina in detail. Robots face the same constraint, sharpened by limits on power, weight and onboard computation.

Mistakes have consequences. An error in a photo-tagging application is an inconvenience. An error in a robot’s perception can mean a collision. Robot perception must work in real time, report how confident it is, and fail safely.

1.3 Lessons from nature

Nature offers proof that capable perception does not require enormous resources. A honeybee’s brain is smaller than a sesame seed and contains about a million neurons, yet bees navigate kilometers to food sources, avoid obstacles in cluttered environments and return home reliably.

Birds and insects solve the same problem, flying through a narrow gap, in very different ways. Many birds have sharp, fast eyes and enough neural processing to judge a gap at a glance and fly straight through at speed. An insect has neither. Its compound eyes have low resolution, and its brain is tiny. Instead it moves to see: it sweeps from side to side in front of the gap, uses the resulting image motion to tell near from far, and then steers toward the opening.

A hummingbird in flight — **See, then fly** **Eyes** Sharp and fast **Brain** Large **Strategy** Judge the gap at a glance

A honeybee in flight carrying pollen — **Move, then see** **Eyes** Low resolution **Brain** About a million neurons **Strategy** Sweep in front, then steer

Honeybee photo: Bogdan Suditu, CC BY 2.0, cropped.

Robots can borrow the insect’s trick. In GapFlyt, a small quadrotor finds a gap of unknown shape and size by moving in front of it and watching how the scene slides past, then flies through it using only its onboard camera and computer.

GapFlyt: active perception lets a quadrotor fly through an unknown gap (shown at quarter speed).

The insect’s strategy carries two ideas that run through this book:

Perception can be active. A robot can choose how to move, where to look and what to sense in order to make perception easier. This idea, formalized as active vision and active perception in the late 1980s, turns some hard problems into much simpler ones.
Perception can be minimal. A robot rarely needs a complete model of the world. It needs the information that matters for its task. Designing perception around the task, rather than around a general-purpose reconstruction, is what makes small, low-power robots possible.

Minimal perception in practice: tiny drones that fly indoors and through forests, custom sensors and lenses, and perception that runs on small, low-power hardware. Examples from the author's research.

1.4 A brief history

Computer vision is younger than most engineering disciplines, and it has been wonderfully overconfident from the start. Here is the story in ten stops.

1963 The blocks world Larry Roberts gets a computer to recover the 3D shape of toy blocks from a photograph. The world is simple, the lighting is perfect, and the field is born.
1966 A summer project MIT's Summer Vision Project sets out to connect a camera to a computer and have it describe what it sees, in one summer. Sixty years later, the summer is still going.
1969 The first textbook Azriel Rosenfeld publishes Picture Processing by Computer and later founds the field's first journal. Every image library you will use, OpenCV included, carries his ideas.
1982 Vision as computation David Marr's Vision argues that seeing is a sequence of representations, from pixels to edges to surfaces to objects. The framework still shapes how we teach the subject.
1988 Move to see Active vision and active perception show that a camera that moves with purpose can solve problems a still camera cannot. Insects knew this all along.
1999 Features that travel SIFT finds the same points in different photos despite changes in scale and rotation. A decade later, researchers rebuild Rome in 3D from tourists' photos in about a day.
2012 The deep learning moment A convolutional network trained on two gaming graphics cards wins the ImageNet challenge by a wide margin. Learned features start replacing hand-designed ones almost overnight.
2020 Photos become worlds Neural radiance fields turn a few dozen photos into a 3D scene you can fly through from new viewpoints. Rendering and vision start to merge.
2023 See anything, in real time Foundation models segment objects they were never trained on, and Gaussian splatting renders reconstructed scenes in real time. The tools reach everyday users.
Now Your turn Robots that perceive with a few watts, explain what they see and know when they are unsure. The next stop on this line is yours to write.

Each era kept what worked from the last. Geometry did not disappear when deep learning arrived; it became a source of structure, supervision and guarantees for learned systems.

1.5 Perception in the era of AI

Perception is changing faster now than at any point in its history. Models trained on very large datasets can segment almost any object, estimate depth from a single image, describe a scene in language and answer questions about it. Vision-language models allow robots to take instructions in plain words. These advances have moved tasks that were research problems a few years ago into everyday tools.

They have also sharpened the questions that matter for robots:

Can it run on the robot? The largest models need data-center hardware. Many robots, from drones to wearable devices, must perceive with a few watts and no network connection.
Can we trust it? A model that is right most of the time can still fail in ways that are hard to predict. Robots need calibrated confidence, physical consistency and a clear account of what the system does not know.
Does it respect geometry and physics? Cameras, lenses and motion obey precise laws. Methods that build these laws in tend to need less data, generalize better and fail more gracefully.

The approach of this book follows from these questions. Learning-based methods are covered in depth, but always alongside the geometry and physics that explain why they work and when they fail. A practitioner who understands both can choose the right tool, combine them, and recognize when a result is too good to be true.

1.6 Who this book is for

This book is written for advanced undergraduate and graduate students in robotics, mechanical, electrical and aerospace engineering and computer science, and for engineers who want a solid foundation in robot perception.

No prior knowledge of computer vision is assumed. You will get the most from it with:

Linear algebra: vectors, matrices, eigenvalues and least squares.
Calculus and probability: derivatives, gradients, distributions and expectation.
Programming: working knowledge of Python. Familiarity with NumPy helps.

1.7 How to use this book

Each chapter combines explanation, mathematics and practice:

Live demos run on your own webcam in the browser. Video is processed on your device and never uploaded.
Editable code cells contain real Python and OpenCV. Change the code and press Run Code to see the result, with nothing to install.
Exercises at the end of sections test understanding and suggest experiments.

The chapters build on each other, starting from how images are formed and represented, moving through features and geometry, and arriving at 3D reconstruction, motion and learned models. Readers who already know the basics can use the sidebar to go straight to the topic they need.

Before you continue

Look around the room you are in and pick one object. List everything you would need to know about it to pick it up with a robot arm: where it is, how big it is, what it is made of, how it might move. Then ask which of those a single photograph could tell you, and which it could not. Keep the list. Much of this book is about filling in the gaps.