Object Detection: Problem Framing & Requirements
Explore how to frame object detection system requirements for autonomous vehicles, focusing on latency budgets, hardware constraints, safety-critical failure analysis, and rare-event data challenges. Understand the importance of detailed requirements scoping and how it impacts design decisions across model architecture, deployment, and system integration for robust, scalable ML solutions.
We'll cover the following...
Unlike visual search, where a query can tolerate hundreds of milliseconds of latency behind a load balancer, an autonomous vehicle hurtling down a highway at 70 mph has no such luxury. A single missed frame, a pedestrian stepping off a curb, a cyclist swerving into the lane, can mean the difference between a safe stop and a catastrophe. This is the deployment regime we enter now, and it changes every design decision from model architecture to how we label training data.
This lesson opens the object detection case study, anchored to real-world systems at Waymo, Tesla Autopilot, and Amazon Robotics. The core interview prompt is deceptively simple: “Design an object detection system for an autonomous vehicle.” A weak answer jumps straight to model selection. A strong answer recognizes this as a full system design challenge spanning latency budgets, edge hardware constraints, safety certification, and failure mode analysis. The requirements scoping step is what separates an L4 answer from a Staff+ answer, and that is exactly what we will work through here. Along the way, we will ground our discussion in key terms such as
Hard real-time inference constraints
The single most important number in this system is 50 milliseconds. That is the maximum allowable end-to-end latency from the moment a camera frame is captured to the moment the control system receives detection results. This number is not arbitrary. At highway speeds of roughly 30 meters per second, every 100ms of latency means the vehicle travels approximately 3 meters without updated perception. A 50ms budget keeps that distance to about 1.5 meters, which is the minimum margin the planning stack needs to initiate emergency braking.
Decomposing the latency budget
The 50ms budget is not entirely consumed by the neural network. It is shared across the full perception pipeline, and understanding this decomposition is critical for making architecture decisions.
The pipeline breaks down into five stages, each with a tight allocation.
Sensor capture consumes roughly 5ms for reading a frame from the camera sensor and transferring it to the processing unit’s memory.
Preprocessing and resize takes another 5ms to normalize pixel values, resize the image to the detector’s input resolution, and ...