How Object Detection Works in Computer Vision

Image classification answers one question about a picture: what is it mostly showing? Object detection asks a harder set of questions. Where is each object, what class does it belong to, and how confident is the system? A detector returns a list of boxes, each with a label and a score, for every object it finds in the image.

This post walks through the building blocks of detection, the two main design families, and the metrics used to judge them.

What detection adds beyond classification

A classifier outputs one label, or a probability for each label, for the whole image. A detector must handle any number of objects, including zero, and each object can have a different size, position and class. A street photo might contain three cars, two people and a traffic light, and the output must list all six with separate boxes.

This changes the shape of the output. Instead of a fixed vector of class scores, a detector outputs a variable-length list. Training also changes, because each object in the training data needs a box and a label, which takes more annotation effort than a single tag per image.

Bounding boxes

The standard representation for an object location is an axis-aligned bounding box. It is usually stored as four numbers, either the top-left and bottom-right corners or a center point plus width and height. Some datasets use slightly different conventions, so practical code must confirm which format a model expects. Mixing formats is one of the most common bugs in detection pipelines.

Many detectors do not predict absolute pixel coordinates directly. They predict offsets relative to a set of reference boxes, called anchors or priors, placed across the feature map. Predicting small adjustments is easier for the network to learn than predicting a box from scratch.

Two-stage detectors

The R-CNN family, introduced starting in 2014, works in two steps. First, a region proposal method suggests a few thousand candidate regions that might contain objects. Second, a CNN extracts features from each region and a classifier labels it while a regressor refines its box.

Fast R-CNN in 2015 shared the convolution work across all proposals, and Faster R-CNN later replaced the external proposal method with a small network that learns to propose regions. Two-stage detectors tend to be accurate, especially on small or crowded objects, but they are slower because the second stage runs for every proposal.

One-stage detectors

One-stage detectors skip the proposal step and predict boxes and classes in a single pass over the image. YOLO, first published in 2015, divides the image into a grid and has each cell predict a few boxes with class scores. SSD, released in 2016, does something similar with feature maps at several scales so that large and small objects are handled at different resolutions.

These models are fast enough for real-time video on modest hardware. Early versions lagged behind two-stage methods in accuracy on small objects, and later versions closed much of that gap through better architectures and training techniques.

Comparing the two families

Property	Two-stage (R-CNN family)	One-stage (YOLO, SSD)
Number of passes	Two, with a proposal step	One
Typical strength	Accuracy, small objects	Speed, real-time use
Typical weakness	Latency per image	Harder for tiny or crowded objects
Common use	Offline analysis, research benchmarks	Video, cameras, robotics

The gap in this table has narrowed over time, so the right choice depends on the speed budget and the size of the objects in the target data.

Intersection over union

To decide whether a predicted box matches a ground-truth box, detection uses intersection over union, or IoU. It divides the area where two boxes overlap by the area covered by either box. Two identical boxes have an IoU of 1. Boxes that touch only at an edge have an IoU of 0. A common rule counts a prediction as correct when its IoU with a ground-truth box is at least 0.5.

IoU is also the core of training. The loss function penalizes boxes that have low overlap with their targets, and variants such as GIoU and DIoU add terms that still give useful signal when two boxes do not overlap at all.

Non-maximum suppression

A detector usually produces many overlapping boxes for the same object. Non-maximum suppression, or NMS, removes the duplicates. It sorts all boxes of one class by confidence and keeps the highest-scoring one. Then it removes every other box of that class whose IoU with the kept box is above a threshold, often 0.5. It repeats this with the next remaining box until nothing is left.

NMS is simple, but the threshold involves a real tradeoff. A low threshold removes too much and can delete true neighbors in a crowd. A high threshold keeps too many duplicates. Some newer detectors avoid NMS by training the model to produce one box per object, but many production systems still use it.

Measuring detection quality

Detection is judged with precision and recall at various confidence thresholds. Precision asks what fraction of predicted boxes are correct. Recall asks what fraction of real objects were found. Plotting precision against recall for every threshold gives a curve, and the area under that curve is the average precision for one class.

The standard headline number is mean average precision, or mAP, which averages the per-class scores. Benchmarks such as COCO also average over several IoU thresholds from 0.5 to 0.95, which rewards boxes that are tightly aligned and not just roughly in the right place.

A simple way to think about a detector

A detector can be read as three cooperating jobs. The backbone turns pixels into feature maps. The head proposes boxes and class scores from those features. The post-processing step, usually NMS, cleans the output into a final list. Errors can come from any of the three, so debugging starts by checking which stage produces the wrong answer.

Why small objects are harder

Small objects are hard for two reasons. First, after several rounds of pooling or strided convolution, a tiny object may cover only a fraction of one cell in the final feature map, so its signal is weak or mixed with background. Second, the IoU of a small box is very sensitive to a few pixels of error. Shifting a 10 by 10 pixel box by 3 pixels in each direction drops its IoU with the original to about 0.32, which counts as a miss at the usual 0.5 threshold. A 200 by 200 pixel box shifted by the same 3 pixels still scores about 0.94. This is why detectors often score far lower on small-object subsets than on large-object subsets of the same benchmark.

Feature pyramids are a common remedy. They combine high-resolution early features, which keep fine detail, with deeper features, which carry stronger meaning. The detector then predicts on several levels and assigns each object to the level that matches its size. Higher input resolution helps as well, but it raises compute cost quickly because the work grows with the number of pixels.

Key takeaways

Object detection extends classification by returning a variable list of boxes, labels and scores. Two-stage detectors trade speed for accuracy, while one-stage detectors favor speed and are common in real-time systems. IoU measures box overlap for both training and evaluation, and NMS removes duplicate predictions. Mean average precision summarizes quality across classes and thresholds.

Good results depend as much on the training data as on the architecture. Accurate boxes need consistent annotation, and the chosen IoU threshold should match how strict the application needs to be.