Self driving vehicles combine perception, reasoning, and control to move safely through an unstructured world. The problem is hard because the environment is only partially observable, other agents are adversarial or unpredictable, and the cost of a mistake can be a human life. This note surveys the autonomy stack, the modular versus end-to-end debate, the SAE levels, the sensor suite, and the safety and long-tail challenges that dominate real deployments.
The autonomy stack
Most production systems decompose driving into a pipeline of subsystems, each consuming the output of the previous one.
| Stage | Question it answers | Related note |
|---|---|---|
| Perception | What is around me? | Object detection, R-CNN |
| Localization | Where am I? | SLAM, Visual Odometry |
| Prediction | What will others do? | Motion Prediction for Autonomous Vehicles |
| Planning | What should I do? | Motion Planning |
| Control | How do I execute it? | Path Tracking, Control Architectures |
- Perception fuses raw sensor streams into a set of tracked objects, drivable surface, lane geometry, and traffic light states.
- Localization places the vehicle in a prior map (often HD, centimeter accurate) using GPS, IMU, wheel odometry, and sensor matching against the map.
- Prediction forecasts the future trajectories of surrounding agents, usually as multimodal distributions.
- Planning produces a collision free, comfortable, rule compliant trajectory, typically split into route planning (Dijkstra’s Algorithm, A*), behavioral planning, and local trajectory optimization.
- Control tracks the planned trajectory with actuators (steering, throttle, brake), often via PID Control or MPC.
Modular vs end-to-end
The stack above is the modular paradigm: interpretable, testable per module, and easy to inject priors into. Its weakness is that errors compound across interfaces, and hand-designed representations may discard information useful downstream.
End-to-end learning instead maps sensors directly to controls with a single network, trained by Imitation Learning (behavior cloning) or deep reinforcement learning. It can exploit information the modular interfaces throw away, but is data hungry, hard to interpret, and suffers from covariate shift. Direct Perception and Conditional Imitation Learning sit between the two extremes, learning a compact intermediate representation (affordances) that a classical controller consumes.
Compounding error
In a modular stack the end-to-end reliability is roughly the product of per-stage reliabilities. Five stages each at 99 percent yield only about , so small per-module gains matter enormously.
SAE levels of autonomy
| Level | Name | Who drives | Who monitors |
|---|---|---|---|
| 0 | No automation | Human | Human |
| 1 | Driver assistance | Human + one aid | Human |
| 2 | Partial automation | System (steer + speed) | Human always |
| 3 | Conditional automation | System in its domain | Human on request |
| 4 | High automation | System in its ODD | System (no human needed in ODD) |
| 5 | Full automation | System everywhere | System |
The Operational Design Domain (ODD) is the set of conditions (geography, weather, speed, time of day) under which the system is validated. The jump from Level 2 to Level 3 is legally and technically sharp because responsibility for monitoring shifts from human to machine.
Sensor suite and fusion
- Camera: dense, cheap, rich color and texture, good for classification and lane markings. Poor absolute depth, degrades in low light, glare, and fog.
- LiDAR: direct, accurate 3D geometry via time of flight, strong for object shape and free space. Expensive, sparse at range, degrades in heavy precipitation.
- Radar: measures range and radial velocity via Doppler, robust to weather, long range. Low angular resolution and sparse.
is the radar Doppler relation, where is the frequency shift, the carrier frequency, and the speed of light, giving instantaneous relative speed that camera and LiDAR must infer over time.
Fusion combines the streams so strengths cover weaknesses. Early fusion concatenates raw or low level features; late fusion combines per-sensor detections; mid-level fusion (increasingly common) fuses learned feature maps in a shared bird’s eye view space. A simple probabilistic late-fusion update treats each sensor as a Gaussian estimate and combines them by inverse-variance weighting:
Why redundancy matters
A camera may see a white truck against a bright sky as empty space, but radar reports a strong return with near-zero relative velocity and LiDAR reports a solid surface. Fusing the three prevents the perception system from declaring the lane clear.
Safety and the long-tail problem
The core difficulty is not the common case but the distribution’s tail: rare events (a mattress on the highway, a pedestrian in a costume, an occluded cyclist) that are individually unlikely but collectively frequent given billions of miles.
The validation gap
To statistically demonstrate a fatality rate better than human drivers (roughly one per miles) you would need to drive hundreds of millions of miles crash free. This is infeasible by road testing alone, which is why simulation, scenario mining, and formal methods are essential.
Mitigations include:
- Simulation and scenario generation to synthesize rare cases cheaply.
- Redundant, diverse sensing and compute so no single failure is catastrophic.
- Runtime safety monitors and minimal-risk-condition fallbacks (safe stop).
- Formal frameworks like Responsibility Sensitive Safety (RSS) that define provably safe following distances and right-of-way rules.
is the RSS safe longitudinal distance, where is response time, and are rear and front speeds, encoding the intuition that a following vehicle must be able to brake to avoid a collision even under worst-case front braking.
Distribution shift after deployment
A model validated on last year’s data can silently degrade as roads, signage, and driving cultures change. Continuous data collection, monitoring, and retraining are part of the system, not an afterthought.