Open-source engine runs 5 AI models on one GPU efficiently

Production AI systems are increasingly built from multiple specialized smaller models rather than one large model doing everything. A concrete example is a flood insurance claim processing pipeline: one model parses the claim form and policy document into clean markdown, another extracts labeled fields like name and policy number from text, a third detects flood-damaged categories in claim photos, a fourth reranks policy chunks against a query, and a fifth generates the final review. While using small models reduces the computational cost per task, the savings evaporate when serving infrastructure is not designed for this pattern.

Today's standard approach forces teams into two impractical choices. The first is to give each model its own GPU. This avoids resource contention, but because a single claim moves sequentially through stages—document parsing, then extraction, then reranking, then generation—each GPU spends much of its time idle waiting for its turn. GPU infrastructure is paid for by time held, not seconds of actual compute, so idle hardware multiplies the bill. Moreover, many specialized models (extractors, rerankers, small vision models) need only a fraction of an L4 GPU's 24 GB of memory, yet each demands its own machine. As new pipeline stages are added, GPU count grows with every capability, defeating the original goal of reducing hardware per task.

The second approach—packing multiple models onto one GPU—looks cheaper but breaks down in practice. Serving processes like vLLM and TEI are built around a single model and don't communicate with each other. vLLM has a --gpu-memory-utilization setting that defaults to 0.92, telling it how much GPU memory it can claim, but it has no way to know what other processes are using or what memory is truly available. If you manually allocate memory to each process before you know actual traffic, you either starve a model (a long document or sudden batch pushes it past its limit, and it crashes, taking every other model on the card down with it) or waste memory (a process holds unused capacity idle while another model suffocates). The core issue is that each tool sees only its own requests; the document server doesn't know what the reranker is waiting for.

Superlinked Inference Engine (SIE) solves this by treating the multi-model pipeline as a single coordinated system. SIE is an open-source inference engine that runs as a production cluster for multi-model pipelines on shared infrastructure, supporting 100+ models through one unified API. For the flood insurance example, a single SIE cluster handles all five stages without separate serving stacks: three different primitives (extract for parsing/extraction/vision, score for reranking, generate for final output) run through the same serving layer.

The engine accomplishes this through five core mechanisms. First, models load only when requests need them; when GPU memory becomes constrained, SIE evicts the least recently used model and makes room for another, so the GPU becomes a shared pool that follows traffic instead of permanently holding idle models. Second, SIE puts all work behind a shared queue instead of isolated per-process queues, giving the serving layer a view of the entire workload across all models and enabling global scheduling decisions. Third, it batches requests by estimated compute cost rather than fixed count, so short inputs don't waste GPU time being padded to match long ones. Fourth, a gateway and worker layer around the runtime lets the system scale beyond a single GPU: workers pull from the shared queue as capacity opens up, and the cluster adds or removes workers as demand changes. Fifth, SIE's model catalog packages 112 supported models with their pre-configured serving settings—memory requirements, batching behavior, precision—so adding a new model is a reference, not a months-long engineering effort.

In production, SIE runs as a Kubernetes-oriented cluster with monitoring and cloud deployment support for AWS and GCP. For teams building multi-model AI pipelines, this means GPU utilization and costs can finally match the promise of smaller specialized models: infrastructure scales with demand, memory is reclaimed from idle models, and every new capability doesn't demand its own machine.