Multimodal Small-Object Detection

Overview

Finding Waldo: Multimodal Approaches to Small Object Detection in Cluttered Scenes (with Nancy Patel) treats "Where's Waldo?" as a controlled benchmark for small-object detection under maximum visual clutter. Waldo is small (32–256px), surrounded by hundreds of visually similar distractor characters, and defined by a precise conjunction of visual attributes (red-and-white stripes, glasses, blue jeans) rather than coarse category membership — making evaluation unambiguous while stress-testing localization.

Method

A two-stage, three-part pipeline:

• Stage 1 — Region proposal: A Faster R-CNN with a ResNet-50-FPN backbone and custom small-object anchors ({8,16,32,64,128}px) generates region proposals. An auxiliary attribute loss, computed via a frozen CLIP ViT-B/32 image encoder against the target's text embedding, is added to the training objective (L = L_cls + L_box + λ_attr·L_attr) to push the backbone toward features aligned with Waldo's full attribute description.
• Stage 2 — Semantic reranking: Each of the top-K proposal crops is re-encoded by the same frozen CLIP image encoder and scored by cosine similarity against the text query (18 phrasings were swept, e.g. "Waldo wearing red and white striped shirt").
• Stage 3 — Score fusion: Final score = detector confidence + 0.1 × CLIP similarity, with the fusion weight chosen by grid search on the validation set.

Results

Method	Mean IoU	Top-1@0.5	IoU=0 (failure) rate
WaldoNet baseline (YOLOv9)	—	0.541	—
Faster R-CNN (ours, no CLIP)	0.556	0.680	0.24
+ CLIP rerank (best config)	0.567	0.680	0.16

The best configuration improves Top-1@0.5 over the published WaldoNet baseline (0.541) and substantially reduces the complete-failure rate. Ablations over query phrasing show the model is sensitive to how precisely the text names Waldo's attributes: explicit mention of the stripe pattern and Waldo's name both help, since CLIP's pretraining encodes that association. Learning rate (best: 1e-4) had the single largest effect, and the attribute loss weight (λ_attr = 0.1) gave a consistent but secondary boost over no attribute loss (0.522 IoU).

Failure analysis

~60% of failures are confusion with background characters whose clothing partially matches the query (e.g., red stripes without glasses); ~40% are complete mislocalization when Waldo is very small and densely surrounded by distractors, where both detection and CLIP similarity signals collapse toward noise. The identified limiting factor is the post-hoc reranking architecture — CLIP scores crops after detection rather than shaping proposal generation, unlike end-to-end grounded detectors such as MDETR or OWLv2.

Code

github.com/EunbiYoon/waldo · github.com/nancypatel12/Finding-Waldo-Multimodal-Detection

Tech stack

PyTorch, Faster R-CNN (ResNet-50-FPN), CLIP ViT-B/32, COCO-style annotations