AT&T's AI system prevents 3.1M field visits, cuts downtime 12M+ hours

AT&T's outage-management challenge is acute: the company serves 145 million wireless and 16 million broadband customers, and even brief disruptions trigger financial credits and customer frustration. The February 2024 outage, which blocked over 92 million voice calls and 25,000 attempted 911 calls, underscored the scale and urgency. The company's response—beginning in Q1 2017 with the EIEM project—reflects a deliberate effort to shift from reactive to proactive outage management. The system consolidates and reorganizes 10 petabytes of historical network data, allowing it to identify patterns and predict problems before they affect service.

The architecture reveals AT&T's commitment to scale: MongoDB's sharding allows data to be distributed across multiple machines without re-architecting the platform as new sources are added, while Snowflake, Databricks, and Azure handle analytics and cloud operations. The addition of generative AI in Q1 2022 marked a turning point—the system can now infer root causes from historical patterns—and the Q1 2025 introduction of AI agents enables direct customer interaction and field-technician guidance. Markus noted that AT&T has built over 30 AI models to predict configuration issues, weather impacts, and system failures, making the system genuinely anticipatory rather than merely responsive.