Property Finder cut AI image-review cost 86% after fixing biased labels

The root problem Property Finder hit is a common shortcut in AI procurement: using a production model's own outputs as the answer key for comparing candidates. Because 96.8% of labels came from that model, alternatives were judged on how closely they mimicked it rather than on correctness, and hidden errors — hallucinated watermark text, single labels forced onto rooms that genuinely had two — went unnoticed. Rebuilding ground truth required a domain expert to annotate images from scratch and settle ambiguous cases, plus a switch to multi-label classification so a hallway showing a bedroom through a doorway could be scored fairly.

With a trustworthy benchmark in place, the team ran prompt optimization as a loop from v0 to v27, scoring each version against the 256-image development set and feeding errors back into the next prompt. A regional rule for decorative Arabic text overlays removed a category of false positives, and instructions to classify by the room occupying most of the frame addressed doorway misreads. Llama 4 Maverick 17B Instruct rose from a 0.75 Micro F1 with the shared starting prompt to 0.81 after tuning, versus 0.55 for the incumbent on the development set.

The broader lesson is about sequencing. Model choice and prompt tuning only pay off after measurement is fixed; as the team puts it, you cannot improve what you cannot measure correctly. Property Finder's production monitoring also showed a 40–60% reduction in inference latency versus its previous setup, suggesting the gain was not accuracy alone. The team notes that its reported accuracy figures are optimistic because prompts were tuned on images included in the evaluation set, so independent replication on a held-out split would be the next test.