Sitrep: AI Copilot for Incident Response, Wired Into Grafana

Sitrep is a macOS menu-bar application that helps engineers manage the chaos of incident response by capturing screen context automatically and making it queryable. When you press Start Incident, the app screenshots your screen every ~15 seconds (configurable between 5 seconds and 5 minutes), downscales each frame to 1568 pixels for Claude's vision API, and runs a deduplication step that compares tiles across frames — dropping duplicates and noise while keeping frames where at least 52 tiles moved or any content-bearing tile changed. Each frame that survives deduplication goes through one vision call to Claude, which extracts structured JSON: summary, app context, trace IDs, error messages, services, metrics, URLs, dashboards, and log lines. The prompt forbids completing partially-visible values (a half-read trace ID is worse than none), and all observations are appended to an append-only JSONL timeline file, with screenshots stored locally under ~/Library/Application Support/sitrep/sessions/.

The core interaction is a chat copilot backed by Claude Opus (with adaptive thinking enabled) and a manual tool-use loop that runs up to 8 rounds with a maximum of 20 tool calls per session. You ask a question — "what was that trace ID, and where did the time go?" — and the agent can call five tools: search_timeline (free, local token-overlap search across your frame observations), inspect_frame (re-read a stored screenshot with a focused question, one vision call per use), tempo_search (TraceQL against Grafana Tempo, scoped to the incident window), tempo_get_trace (full span list for a trace ID), and prom_query (instant PromQL, optionally evaluated at a past moment). The agent's entire reply — including thinking blocks if adaptive thinking is enabled — goes back into the conversation. Tool failures come back as text the model can reason over; model call failures are fatal. Citations are recorded as a side effect of opening a frame, not claimed by the model, so the prose is the model's but the receipts are the harness's.

Every LLM call the app makes is itself instrumented with OpenTelemetry GenAI semantic conventions: gen_ai.* spans, token-usage histograms, and a cost counter, all shipped to the same local Grafana instance that powers your incident response. The app deviates from the spec in three intentional ways: gen_ai.operation.name always reports the enum value chat (since all four LLM call types are Messages API completions), and app-level distinctions (ingest_frame, inspect_frame, locate, chat) ride in sitrep.operation instead; cost is sitrep.gen_ai.cost, not gen_ai.client.cost.usd, because there is no cost metric in the conventions; and gen_ai.system is emitted alongside the newer gen_ai.provider.name for backward compatibility with existing dashboards. The interface surfaces cost as a live control — switch the vision model from Opus 4.8 to Haiku 4.5 (roughly 5× cheaper) and the scan cadence, and a projected $/hr recomputes as you click. Blackout mode pauses capture entirely (the OS capture API is never invoked) when you need to open sensitive screens; chat and timeline keep working. To run it, you need Node ≥20 and macOS, optionally deploy the Grafana OTEL-LGTM container (docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 grafana/otel-lgtm:latest), then npm install, copy .env.example to .env (adding ANTHROPIC_API_KEY, or leaving it blank for mock mode), and npm run dev. A demo is runnable with no API key at http://localhost:5187 via npm run demo, using staged data.

Sitrep addresses a real pain point in incident response: the cognitive load of tracking dozens of fast-moving facts across multiple windows and tools. By automating screen capture and vision extraction, it creates a searchable, timestamped record of what happened on the engineer's screen — indexed locally with no API key required for the search itself. The architecture is deliberately lightweight: the model decides strategy (which tool to call next) but the harness enforces guardrails (validating arguments, scoping Tempo queries to the incident window to avoid silent misses, capping rounds and cost). This separation lets the agent adapt when results are boring without risking hallucinated traces or runaway loops.

The self-instrumentation design is noteworthy: every LLM call the app makes (frame extraction, timeline search, trace fetching, reasoning) is wrapped in OpenTelemetry GenAI spans and sent to the same Grafana instance the app helps you query. This means the copilot's own latency, token usage, and errors are debuggable in the dashboard — turning the tool into an observable system itself rather than a black box. The three deliberate deltas from the GenAI spec (routing app-level operations into sitrep.operation rather than corrupting gen_ai.operation.name, coining sitrep.gen_ai.cost for dollars, and emitting the deprecated gen_ai.system for backward compatibility) reflect pragmatic choices to stay compatible with existing Grafana dashboards while extending the spec.

No comments yet. Be the first to share your thoughts!