nexa-gauge, a Python package for evaluating LLM and RAG system outputs, introduces cache-aware evaluation graphs that estimate costs before execution and reuse cached artifacts.

nexa-gauge is a Python package and command-line toolkit that replaces manual checks with a typed evaluation graph for LLM, RAG, and agentic systems. It estimates cost, executes only required nodes, reuses cached artifacts, and emits structured per-case reports.

The tool combines deterministic metrics (BLEU, METEOR, ROUGE) with LLM-as-a-judge evaluation across five metric nodes: relevance (whether claims answer the question), grounding (whether claims are supported by context), redteam (bias and toxicity), geval (criteria-based judging), and reference (overlap against known answers).

Cache is deterministic and route-aware—inputs, evaluation criteria, model routing, prompt versions, and upstream artifacts are included in cache keys to prevent stale outputs across incompatible runs. Users can preview uncached work before execution with `nexagauge estimate`, run evaluations with caching enabled, or force fresh outputs with the `--force` flag.