Amazon memo: token counts, lines of code are flawed AI metrics

Amazon's memo arrives after companies spent considerable effort pushing employees to adopt AI, and it reflects a shift in emphasis from sheer usage to judgment. Terkowitz wrote that counting tokens and measuring lines of code are 'deeply flawed' because they don't necessarily reflect the quality of work, and he warned that turning those metrics into targets could drive up costs and add unnecessary complexity, undermining Amazon's 'invent and simplify' principle.

The memo also acknowledges that Amazon has already felt the downside of encouraging AI use for its own sake. Employees reportedly ran AI agents on meaningless tasks to inflate their token count and climb an unofficial internal leaderboard called Kirorank, which Amazon nixed in May after the practice raised concerns about unnecessary computing costs. The memo does not abandon measurement entirely: Terkowitz said AI adoption metrics still 'correlate well with the outputs that matter,' and encouraged managers to compare the 10th percentile and 90th percentile of AI users on their teams—to understand why some use it least and whether the heaviest users' output is increasing accordingly.