llmproxy: Open-source Flask proxy routes local AI clients to NVIDIA cloud API

llmproxy is a lightweight proxy server built in Python using Flask that sits between a client application and NVIDIA's cloud LLM API. The core idea is simple: it accepts HTTP requests in the format that Ollama, OpenAI, or llama.cpp expect, then translates and forwards them to NVIDIA's OpenAI-compatible endpoint at integrate.api.nvidia.com/v1, returning the response back to the client in its native format.

The proxy supports the full suite of LLM operations: chat completions, text completions, and embeddings. It handles streaming responses by aggregating Server-Sent Events (SSE) from NVIDIA and streaming them back to the client in the expected format. Clients can discover available models through the proxy's multi-model discovery endpoint, and inbound requests can be authenticated via optional credentials. If NVIDIA's API returns a transient error, the proxy automatically retries the request before giving up.

To reduce latency and API costs, llmproxy includes an optional response cache for non-streaming replies. The cache TTL (time-to-live) and maximum size are configurable, allowing teams to tune the tradeoff between freshness and efficiency for their workload. A live /stats endpoint exposes metrics and process information, useful for monitoring proxy health in production.

Deployment is straightforward. Users configure their NVIDIA API key in a .env file, then run either `docker compose up -d` for a full stack or `docker run -d -p 11434:11434 --env-file .env lordraw/llmproxy:latest` to start the prebuilt image from Docker Hub. The repository includes a test suite (scripts/tests.sh) written in bash with an optional terminal UI, plus deployment instructions for building and publishing the Docker image via a Makefile. The entire project is released under the MIT License, permitting free use, modification, and distribution.

llmproxy addresses a practical friction point: many AI tools and SDKs are written to speak Ollama, OpenAI, or llama.cpp protocols, but teams may want or need to use a different inference backend—in this case, NVIDIA's cloud API. Rather than forking or rewriting client code, the proxy sits in the middle as a protocol translator, intercepting requests in the client's native format and converting them to OpenAI format for NVIDIA. This is especially valuable for teams already invested in local-first development workflows or tools like Open WebUI that natively support those APIs.

The tool also bundles practical features: automatic retries shield against transient upstream failures, response caching reduces redundant API calls for identical queries, cost tracking through metrics visibility helps teams understand their NVIDIA spending, and inbound authentication gates who can use the proxy. The fact that it ships as a prebuilt Docker image (lordraw/llmproxy on Docker Hub) and includes Docker Compose configuration lowers the barrier to production deployment, making it accessible even to teams without deep DevOps experience.

No comments yet. Be the first to share your thoughts!