GAZE framework enables medical vision-language models to iteratively inspect brain MRI images and retrieve literature, reaching 58.2 mAP for lesion localization on rare neurological conditions

GAZE (Grounded Agentic Zero-shot Evaluation) lets medical vision-language models call viewer-level tools (zoom, windowing, contrast, edge detection) and retrieval tools backed by PubMed and Open-i, with structured outputs validated against a schema and full tool-call traces recorded for auditability.

On NOVA, a benchmark of 906 brain MRI cases covering 281 rare neurological conditions, GAZE reaches 58.2 mean average precision (mAP) at intersection-over-union (IoU) 0.3 for lesion localization and 34.9% Top-1 diagnostic accuracy under a joint protocol scoring captioning, diagnosis, and localization, without task-specific fine-tuning.

Tool use helps rare pathologies disproportionately: the fraction of cases with IoU > 0.3 rises from 17% to 58% for diagnoses with three or fewer examples versus 25% to 68% for common conditions (≥10 cases), with retrieval ablations revealing a model-dependent trade-off in which gains in diagnosis can coincide with losses in localization.