AT&T's AI system prevents 3.1M field visits, cuts downtime 12M+ hours

AT&T began developing its end-to-end incident management system (EIEM) in 2017, assembling a cross-functional team of IT, data, and AI employees alongside field technicians and network operations staff to ensure the system would address real workflow constraints. The company reorganized 10 petabytes of data—equivalent to 5,000 billion pages of printed text—consolidating network logs, alarms, incident dispatch tickets, and outage details to enable both issue identification and predictive capability. MongoDB served a central architectural role, using a technique called "sharding" to distribute data across multiple machines, allowing AT&T to scale without completely rebuilding the platform as new data sources were added.

The system was deployed in phases: the EIEM launched for broadband fiber in Q1 2018, expanded to digital subscriber lines (DSL) in Q1 2019, and added a proactive customer notification system by Q1 2021. In June 2018, AT&T introduced Atlas, a separate field technician application combining machine learning and AI to identify root causes and recommend repair plans. Traditional AI underpinned these early efforts, but after Andy Markus joined AT&T as chief data and AI officer in July 2020, the company accelerated AI adoption. Generative AI capabilities were integrated into the EIEM in Q1 2022, allowing the system to infer root causes from historical data and similar past situations. Most recently, in Q1 2025, AI agents were added to interact directly with customers, gather outage information, take action to resolve issues, and brief field technicians on case details.

The system now processes over 27 billion tokens per day and is accessible to 100,000 AT&T employees. Markus emphasized that AT&T has built more than 30 AI models to predict configuration issues, weather-related failures, and system failures, enabling the platform to be proactive rather than merely reactive. The outcomes have been substantial: over the last year, the AI-enabled EIEM prevented 3.1 million unnecessary field dispatches and reduced customer downtime by more than 12 million hours. For AT&T's 145 million wireless and 16 million broadband customers, this translates to fewer service interruptions and faster resolution when outages do occur—and reduced frustration when customers are proactively informed of disruptions.

AT&T's outage-management challenge is acute: the company serves 145 million wireless and 16 million broadband customers, and even brief disruptions trigger financial credits and customer frustration. The February 2024 outage, which blocked over 92 million voice calls and 25,000 attempted 911 calls, underscored the scale and urgency. The company's response—beginning in Q1 2017 with the EIEM project—reflects a deliberate effort to shift from reactive to proactive outage management. The system consolidates and reorganizes 10 petabytes of historical network data, allowing it to identify patterns and predict problems before they affect service.

The architecture reveals AT&T's commitment to scale: MongoDB's sharding allows data to be distributed across multiple machines without re-architecting the platform as new sources are added, while Snowflake, Databricks, and Azure handle analytics and cloud operations. The addition of generative AI in Q1 2022 marked a turning point—the system can now infer root causes from historical patterns—and the Q1 2025 introduction of AI agents enables direct customer interaction and field-technician guidance. Markus noted that AT&T has built over 30 AI models to predict configuration issues, weather impacts, and system failures, making the system genuinely anticipatory rather than merely responsive.

No comments yet. Be the first to share your thoughts!