Clinic 20

IoU Threshold Or NMS Tune

AP jumped from 0.41 to 0.47 after a teammate lowered the NMS IoU threshold from 0.50 to 0.30. That looks like a win. Before the PR merges, decide whether you moved the model or moved the ruler.

Back To Clinics Open Object Detection Open Detection & Segmentation

Situation

AP Jumped 6 Points

A one-line change to the NMS IoU threshold lifted validation AP. The model weights did not change. Someone called it a win.

Your Job

Is It A Real Win?

Decide whether the AP lift reflects better detection or easier scoring. Name the inspection that decides.

Bad Habit To Avoid

A Tempting Free Win

The headline number moved without retraining. Decide whether that gain transfers to the deployment slice — or only to the validation grader.

Situation

You are shipping a fine-tuned object detector (Faster R-CNN class) for retail shelf product recognition. The evaluation harness computes COCO-style mAP @ IoU[0.50:0.95] plus the single-number AP@0.50.

Last week's numbers:

• mAP @ IoU[0.50:0.95]: 0.38
• AP@0.50: 0.61

Today's numbers after a teammate changed the NMS IoU threshold from 0.50 to 0.30 at inference time:

• mAP @ IoU[0.50:0.95]: 0.41 (+0.03)
• AP@0.50: 0.67 (+0.06)

The teammate has opened a PR titled "Lower NMS IoU: +6pt AP@0.50". The PR comment says: "no retraining, one config line, ship it before the demo."

What changed:

• the NMS threshold controls how many overlapping boxes survive post-processing
• lowering from 0.50 → 0.30 means more aggressive suppression — boxes overlapping at IoU ≥ 0.30 get suppressed (before, only IoU ≥ 0.50 suppressed)
• fewer predicted boxes overall; fewer duplicates; apparent precision up

Two things to inspect before the merge:

• on the retail domain, products sit close together on shelves. Aggressive NMS can delete real detections that happen to overlap a neighboring product's box.
• AP is averaged over recall. If NMS lowers recall in dense scenes, AP can still rise in the average because the precision lift is stronger than the recall drop — but deployment performance on dense-shelf images specifically may be worse.

Artifact Packet

What NMS controls (roughly):

IoU threshold	behavior	typical cost
high (e.g., 0.70)	permissive — many overlapping boxes survive	low precision; duplicated detections
0.50 (default)	balanced	standard trade-off
low (e.g., 0.30)	aggressive — small overlaps suppressed	fewer duplicates but real adjacent objects suppressed

COCO AP recap — why a single number can shift without the model changing:

• AP integrates precision over recall thresholds
• NMS threshold changes the set of boxes scored, not the score per box
• the IoU threshold that AP uses (the 0.50 in "AP@0.50") is the matching threshold for calling a prediction correct; it is different from the NMS threshold

Four slice numbers from your eval set, before and after the NMS change:

slice	n images	AP@0.50 before	AP@0.50 after	delta
sparse shelves (< 5 products)	600	0.78	0.80	+0.02
medium density (5–15)	900	0.63	0.70	+0.07
dense shelves (> 15)	500	0.45	0.42	−0.03
edge cases (occlusion, labels turned)	200	0.41	0.44	+0.03

The overall AP@0.50 went up. The dense-shelf slice — the slice the retail customer cares about most — went down.

Decision Prompt

Write a six-sentence defense that answers:

1. Is the AP lift a real model improvement or a harness movement? What is the one-sentence explanation you would give the teammate?
2. What slice number would make you block the PR even with overall AP up?
3. Is there a separate inspection that shows whether recall on dense images changed?
4. If you must ship something today, what compromise IoU threshold would you set (0.30, 0.40, 0.50) and why?
5. When would lowering NMS IoU actually be the right model-level fix?
6. What retraining change would be the "real" fix if NMS tuning only moves the trade-off?

Strong Reasoning Looks Like

• names the distinction explicitly: NMS IoU is a post-processing knob; AP@0.50 is an evaluation ruler; changing NMS does not change the model, only the predictions the model emits
• blocks the merge because the dense-shelf slice regressed by 3 points — that is the customer-visible slice and the overall lift hides it
• asks for per-slice recall@fixed-precision in addition to AP; a 5-point recall drop on dense shelves is the kind of thing AP averaging can camouflage
• proposes a compromise: 0.40 NMS may capture most of the precision lift without the full dense-shelf recall collapse — but only after checking the slice numbers at 0.40
• recognizes the real fix: train with dense-shelf augmentation or switch to an architecture with better small-object handling (e.g., FPN on a higher-resolution input), not just tune NMS
• treats the PR title "+6pt AP@0.50" as the red flag: an improvement that requires no training and moves a single global number deserves extra slice inspection

Common Wrong Moves

• merging because the overall AP went up — retail shelves are exactly the slice where the lift hides the loss
• tuning NMS on the full validation set and reporting the global delta — missing the slice movement entirely
• claiming a model improvement in the PR title without retraining
• assuming AP@[0.50:0.95] agrees with AP@0.50 about NMS — they can move in opposite directions; the wider range is more sensitive to NMS changes
• lowering NMS without re-calibrating the confidence threshold used at deployment; the two together determine what the user actually sees
• forgetting that the labeling of the ground-truth boxes also has an IoU assumption — if annotators drew loose boxes, aggressive NMS is punishing the model for the annotator style

Run The Clinic In Browser

Use the runner to sweep NMS IoU across {0.30, 0.40, 0.50, 0.60, 0.70} and compute per-slice AP and recall; the slice plot is the decider.

Reference Reveal

Open only after you write the defense

The reference call is **block the 0.30 NMS PR; hold at 0.50 (or compromise at 0.45) until retraining addresses dense-shelf cases**. Why: - the 3-point regression on the dense-shelf slice is a customer-visible regression. The customer's retail stores sell products packed on shelves — the "dense" slice is the *point* of the system. - AP@0.50 global going up by 6 points while the key slice goes down 3 points is the textbook case of **Simpson's-paradox-for-detectors**: the average moved because the large-volume sparse and medium slices moved. - NMS is a post-processing knob, not a training signal. A PR that changes a knob has no information about *why* the model missed dense detections; it just stops the model from reporting them. The right next cycle: - retrain with dense-shelf **augmentation**: CutMix-style paste of adjacent products, small-object-focused mosaics, higher input resolution on dense crops - if a one-shot NMS tune *is* acceptable for an immediate demo, choose **0.45** after confirming the slice sweep shows dense-shelf regression is < 1 point. Document it as a temporary workaround. - do the comparison under both AP@0.50 and AP@[0.50:0.95]; if they disagree on direction, the harness is the thing to pick apart, not the model What "real" improvements in this area look like: - a +2 AP from a larger input resolution that costs 20ms/image — that is a model improvement with a latency trade-off - a +1.5 AP from training with dense-shelf augmentation — a real improvement because it changes the model's behavior on the slice you care about - a +3 AP from correcting a label-style mismatch (annotators drew boxes 5% looser than the detector expects) — not the model, but a *harness* fix that is honest about the cause Inspection ladder for any AP-changing PR: 1. is the weight checkpoint different? If no, the change is a harness/post-processing change — label it correctly 2. per-slice AP and per-slice recall@precision=0.9 — the aggregate hides the slice 3. visual inspection of 20 cases from the regressed slice — the eye catches suppression failures the metric averages away 4. latency check — a "free" improvement that doubles inference time is not free The practical lesson: **post-hoc harness tweaks are free in compute and expensive in trust**. A number lift that requires no training and moves no model weights is a movement of the ruler, not of the cloth being measured. The PR title has to reflect that.

What To Do Next

1. open Object Detection Basics for the NMS-and-matching-IoU distinction in depth
2. open Detection and Segmentation for the architectural knobs that move dense-slice performance
3. open Reliability Slices — the discipline this clinic depends on
4. sweep NMS IoU on your own eval set and plot per-slice AP; one chart answers most of this clinic