New book: Zillow's $500 million failure shows AI test scores lie

The book builds its argument from case studies of real incidents. Zillow's iBuying failure is presented as an example of raising a system's importance without raising its governance, after the company removed expert review and later took over $500 million in write-downs and laid off about 2,000 employees. A separate XGBoost example shows how an acceptable overall AUC of 0.78 can conceal a segment where the false positive rate is 1.0. The book sets these against a regulatory backdrop that includes the EU AI Act's 2024 effective date and phased enforcement from August 2026, and it deliberately declines to cover generative AI because the authors believe failure modes and controls are not yet well established there. A red-teaming exercise reports that a constrained model was easier to extract than an unconstrained one, illustrating that defenses can create new attack surfaces. The outcome likely hinges on whether organizations treat documentation, independent testing, and team diversity as core engineering requirements rather than afterthoughts.