AI reasoning for root cause analysis mostly works—if you feed it clean data

The article opens with a recurring pitch: pipe your telemetry into a large language model and let it diagnose what broke. The author, drawing on years in the field, argues this framing conflates two separate tasks: reasoning (the model's ability to connect the dots once it has the data) and the harness (everything around the model—data preparation, tool-calling, context assembly). A model might fail at RCA not because it cannot reason, but because it never received the right data or received it in an unusable form.

To test reasoning in isolation, the author and Coroot built a deterministic preprocessing pipeline that handles all the heavy lifting: correlating signals, identifying anomalies, and preparing a compact, focused context. The model receives findings, not raw telemetry. No tools, no agent loop. Everything needed to answer the question is already in the prompt. This setup lets you measure whether the model's failure is a reasoning problem (it had the data and still missed the answer) or something else entirely.

The test scenario: a network delay experiment (Chaos Mesh NetworkChaos) injected latency between the catalog service and its Postgres database (db-main). The result: slow queries, timeouts, and 502 errors on the front-end. But the database itself is fine. The culprit is the chaos experiment running in the cluster, visible in Kubernetes events. The fix: delete the experiment to restore connectivity, then delete the schedule that would spin it back up—the second step is easy to miss. The trap: Postgres times queries from first byte received to last byte sent back to the client, so the network delay gets counted as query time. A naive read of slow query metrics blames the database, exactly the wrong answer. The real signal (network round-trip time, TCP latency, the Kubernetes event) is right there in the context alongside these misleading breadcrumbs.

The author handed this same 9,800-token context to eleven models and asked each three things: what's the root cause, what's the cause-and-effect chain, and what's the immediate fix. The bar was simple: did it pin the chaos experiment, explain the ripple to the front-end, and specify deleting both the experiment and its schedule? Claude Opus 4.8 nailed it, tracing the chain from front-end errors back through catalog's slow queries to the NetworkChaos experiment, citing correlated RTT and TCP spikes, noting that db-main's CPU was clean, and prescribing the two-step fix.

Results across all eleven models: Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro (the frontier closed-weight models) all passed cleanly. Among large open-weight models, DeepSeek-V4-Pro, GLM-5.2, and MiniMax-M3 got the cause right; Nemotron 3 Ultra (550B parameters) and DeepSeek-V4-Flash got the cause right but had formatting slips (raw widget IDs, incomplete fixes). Gemma 4 31B, the smallest at 31 billion parameters, was the only self-hostable model to pass entirely. Qwen3.6 35B and Qwen3 Coder Next (80B)—both larger—failed: one blamed the database and suggested scaling Postgres; the other found the experiment but named the wrong resource to delete, yielding a command that wouldn't work.

The cost spread was wide: $0.0743 for Claude Opus, down to $0.00965 for DeepSeek-V4-Flash and $0.00324 for MiniMax-M3. Even the most expensive frontier model cost only a few cents per incident, making the choice of a top-tier model economically defensible for reliability. The failures subdivided neatly: real reasoning failures (Qwen models) versus formatting and instruction-following slips (other 🟡 cases). The 🟡 cases got the root cause right but stumbled on packaging—deleting the chaos object without the schedule, leaking raw IDs, ignoring formatting instructions. These are harness problems, fixable on the preprocessing side. The author concludes that the reasoning part of AI RCA is essentially solved at the frontier and even at modest scale; the actual unsolved problem is the data pipeline—telemetry keeps growing, and feeding raw data to an LLM gets slow and expensive fast. The real work is preparing compact, well-structured context before the model is called.

The article tackles a common misconception: that AI can solve root cause analysis (RCA) by simply feeding telemetry into an LLM. The author argues that RCA with an LLM actually comprises two separate problems with different failure modes. The first is reasoning—the model's ability to connect disparate signals into a coherent explanation. The second is the harness—the data pipeline, tool-calling logic, and context preparation that happens before the model sees anything. Most failures attributed to "LLMs can't do RCA" are actually harness failures, not reasoning failures; the model never received the right data in the right form.

To isolate reasoning from harness problems, the author and their team (at Coroot) built a deterministic pipeline that does the heavy lifting of correlating signals and summarizing findings before passing context to the model. This removes tool-calling and agent loops entirely, leaving only the reasoning task: given complete, focused context, can the model identify the true root cause and recommend the correct fix? The test scenario—a network delay injected by a chaos experiment—was deliberately chosen because all the evidence needed to reach the correct answer already exists in the context. The trap is that downstream effects (slow database queries, CPU pressure) can mislead a weaker model into blaming the wrong component.

The results show a clear pattern. Frontier models (Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro) and large open-weight models (DeepSeek-V4-Pro, GLM-5.2) correctly identified the chaos experiment and prescribed both fix steps. Failures fell into two categories: genuine reasoning failures (two Qwen models blamed the database or deleted the wrong resource) and format/instruction-following slips (incomplete fixes, raw IDs leaking into output). Notably, Gemma 4 31B, the smallest self-hostable model tested, outperformed larger open-weight competitors, suggesting that scale alone does not determine RCA capability for self-hosted deployments. The cost per incident ranged from fractions of a cent to a few cents, making frontier models economically viable for single incidents given their reliability.