ILTER: Self-hosted AI gateway with cost control, PII masking, smart routing

ILTER is an independent gateway that proxies all AI traffic between applications and cloud AI providers (OpenAI, Anthropic, Google Gemini, DeepSeek, Ollama, etc.), packaged as a single, zero-dependency static binary. Users simply run `./ilter serve` and the entire infrastructure—proxy, dashboard, and metrics—starts in seconds on localhost ports 8181, 9191, and 9192 respectively. No Node.js, Python, external servers, or configuration files are required; the system works out of the box with compiled defaults and optional environment variable overrides.

The core problem ILTER solves is three-fold: (1) unpredictable costs from per-token pricing, (2) data privacy violations when sending customer data to cloud vendors, and (3) disruption from API outages or rate limiting. To address cost control, ILTER tracks spending in real-time per API key and enforces hard daily and monthly limits (in USD); the moment a limit is breached, it returns a 429 budget_exceeded response, cutting off traffic immediately. To protect privacy, a PII Guard detects emails, SSNs, national IDs, credit cards (Luhn-validated), phone numbers, IP addresses, and names (in English and Turkish), then masks, reversibly encodes, or blocks the data before it leaves the network, all in <0.04ms with <5MB memory overhead. For outage resilience, ILTER sits behind a circuit breaker that opens after 5 consecutive failures and automatically fails over to a backup provider or API key without any code changes.

A smart router (real-time complexity scoring at <0.022ms) analyzes incoming requests—word count, reasoning-required phrases, code blocks, and tool calls—and assigns a score from 0–100, then dynamically routes to economy, standard, or premium model tiers. Custom routing rules (e.g., "complexity > 50 → gpt-4o", "prompt contains 'analyze' → claude-sonnet") and load-balancing strategies (weighted round-robin, cost-optimized, latency-optimized, priority-based) are supported. An agent loop detector catches runaway agentic workflows by blocking if more than 30 requests/second are sent, the exact same prompt is repeated >5 times in a 20-request window, >$5 is spent within 5 minutes on a single session, or a session exceeds 100 requests.

Additional features include a semantic cache (Redis Stack-backed vector search with SHA-256 exact-match fallback, async write path to avoid blocking), prompt guardrails (injection detection, toxicity filtering, keyword and regex topic blocking), a cron engine for scheduled AI workflows using standard cron expressions (e.g., `0 9 * * 1`), and full observability (Prometheus metrics at `/metrics` on port 9192, OpenTelemetry tracing for Grafana Cloud/Datadog/Honeycomb, and audit logs). An embedded MCP (Model Context Protocol) gateway allows connecting internal APIs and CRM systems as tools without client-side changes; ILTER automatically injects tools from registered MCP servers, intercepts tool_call responses, and returns results to the model. An OpenAPI bridge converts any REST API to MCP tools instantly, and OAuth PKCE support enables remote MCP clients to authorize via standard endpoints.

The dashboard (built with Astro + React, compiled directly into the Go binary) provides a live KPI overview, monthly cost trends, provider/model breakdowns, a built-in chat playground for testing any registered model, a logs view showing full request history with cost and latency, a jobs scheduler for cron workflows, and management tools for API keys, thresholds, MCP servers, and feature toggles. Performance benchmarks on Apple M4 with Go 1.26.3 show core proxy overhead at <0.8ms, PII Guard scan at 0.031ms, smart router scoring at 0.022ms, and loop fingerprinting at 0.008ms. The Docker image is <20MB (3-stage build to scratch base), and the plain binary from releases is ~35–40MB. Configuration is environment-variable driven (e.g., `ILTER_ADMIN_API_KEY`, `ILTER_PROVIDER_OPENAI_API_KEY`); the service refuses to start if both an admin key and at least one provider key are not set. The project is written in Go 1.26.3 (CGO-free), uses the chi v5 router, SQLite for state (modernc.org/sqlite in WAL mode), and optional Redis Stack 7+ for caching, with graceful degradation if Redis is unavailable. Source code, documentation, and Docker Compose examples (full local stack: ILTER + Redis Stack + Ollama) are available in the project repository.

ILTER targets a specific operational gap: companies integrating AI into production workflows face three uncontrolled risks—runaway costs from per-token pricing, data leakage to cloud vendors (breaking GDPR/HIPAA), and vendor lock-in to a single provider or outage. The tool solves these at the gateway layer by interposing a lightweight proxy that enforces spending caps, strips sensitive data, masks prompts, and auto-failovers without any application-side code changes. This is significant because it means teams can retrofit AI cost control and compliance into existing deployments immediately.

The architecture reveals why this works as a single binary: ILTER is written in Go (CGO-free, goroutine-native), uses SQLite for state, and optionally integrates with Redis Stack for semantic caching. The gateway exposes three ports—8181 for the OpenAI-compatible proxy, 9191 for the embedded dashboard (Astro + React, compiled into the binary), and 9192 for Prometheus metrics. Every feature (smart routing, PII masking, loop detection, cron jobs) runs in the same process, eliminating the operational complexity of external queues or microservices. The quoted performance numbers (core proxy overhead <0.8ms, PII Guard <0.031ms, router scoring <0.022ms) suggest the overhead is negligible for most use cases.

One notable implication: organizations can now audit and control AI traffic at a single chokepoint—the gateway sees every request, every cost, every API key, and every PII event. This transforms the economics of AI adoption: instead of billing surprises or sprawling compliance audits, the dashboard provides real-time KPIs, cost breakdowns, audit logs, and feature toggles. For teams already running Prometheus/Grafana or OpenTelemetry (Datadog, Honeycomb), the native metrics and tracing integration means ILTER fits directly into existing observability stacks.

No comments yet. Be the first to share your thoughts!