Google's Android Bench 2.0: top pass rate falls to about 28%

Google's earlier Android Bench and early AI coding benchmarks concentrated on small, local work: small changes, bug fixes and feature additions. Android Bench 2.0 reorganises the evaluation around long-horizon tasks taken from a developer's actual time spent with an AI, including upgrades of dependencies like libraries, additions of new features, application designs from scratch, cross-platform application porting to Android, and code conversion between frameworks.

The body also ties the benchmark to evaluation frameworks. Google aligned it with Harbor, a framework for LLM evaluation, and moved beyond judging models alone toward evaluating AI agents combined with development environments. To produce more meaningful metrics, Google computes a Completion Rate instead of a binary score, combining functionality, UI quality and regression prevention, and applies objective deductions and penalties when rules or design principles are violated.

The dataset shows the split by task type. AI was better at refactoring existing code than at creating new code, and in conversions such as Java to Kotlin, Retrofit to Ktor and migration to ViewModel, it applied known patterns across code bases exceeding 8000 lines in more than 125 files. On tasks needing runtime verification, breaking framework changes or unknown gaps, performance fell sharply. Cross-platform application porting stayed difficult, with a top score of 80% and a pass rate short of 100%. The stakes hinge on whether Google's planned expansion of model and agent-tool combinations, and its first steps on agent evaluation, actually close that gap.