A guide shows how to monitor LLM inference systems using Prometheus and Grafana, addressing the need for visibility into token throughput, queue behavior, and cache pressure—metrics that traditional API monitoring does not capture.

What happened: A technical guide outlines how to set up monitoring for large language model inference using Prometheus (a metrics collection tool) and Grafana (a visualization platform). It covers what metrics to track—such as requests per second, tokens per second, p95/p99 latency, queue duration, and GPU cache usage—and provides configuration examples for vLLM, Hugging Face TGI, and llama.cpp servers, along with sample PromQL queries and dashboard layouts.

Why it matters: LLM inference monitoring differs fundamentally from traditional API monitoring because latency has two meanings (end-to-end and per-token), throughput is measured in tokens rather than requests, queue depth directly affects user experience, and cache exhaustion often precedes outages. Teams running continuous batching or multi-node setups need visibility into these LLM-specific signals to prevent latency spikes and GPU bottlenecks.

What to watch: The guide emphasizes tracking KV cache usage as a percentage (vllm:kv_cache_usage_perc), inter-token latency via mean time per token histograms, and queue size and duration separately, since cache pressure and queue dynamics are outage precursors that show up only under real load and require different monitoring approaches than standard REST services.