Small Models Alone Don't Cut Costs—Shared GPU Serving Does

The article opens by observing that production AI systems are typically stacks of individual tasks, each handled by a model best suited to it. A single user request might involve parsing a document into text, embedding that text into vectors for retrieval, reranking retrieved results by relevance, extracting specific fields or entities, and checking the output against a safety policy. Until recently, teams defaulted to using the largest general-purpose model available and having it handle as much as possible, paying frontier rates for every call. That reflex is now reversing toward using small models fine-tuned for each task, since most tasks do not require frontier reasoning.

However, the catch teams miss is that a cheaper model does not imply a cheaper system. The cost of a model is not only what it charges per call—it is also the hardware kept running to answer that call. A pipeline that replaced one large model with multiple small ones now has multiple models to serve, each occupying a GPU of its own. GPU providers bill for a whole GPU for a span of wall-clock time, with the meter running whether the card is busy or idle. So every GPU held adds another term to the full bill: C × T, where C is the price per second of each GPU and T is how long it is held. Four models on four cards means four terms in that sum. If one card can carry all four models' traffic, three of those terms disappear—the bill drops by 75% with no change in the actual work performed.

The memory argument also holds up. A small embedding model, a cross-encoder reranker, and an entity extractor together take a few gigabytes. An L4 GPU has 24GB, so they can fit with room to spare. Yet almost nobody does this because every layer of the stack—from the serving framework down to the driver—was built around one model owning one GPU. vLLM is built around one model as well, and TEI, Hugging Face's embedding and reranking server, takes a single --model-id at launch. Putting four models on one card requires running four separate server processes on it, with nothing in either tool coordinating those processes. Each process claims its memory upfront and cannot see its neighbours. vLLM's --gpu-memory-utilization defaults to 0.9, pre-allocating that fraction of the card at startup rather than adjusting it as per demand. The setting is a per-instance limit that ignores any other vLLM instance on the same GPU, so running two means setting the value to 0.5 for each by hand.

The memory split becomes a constant computed before any traffic arrives, and it must cover each model's peak, because activation memory grows with batch size and sequence length. Four worst-case slices pinned onto one card is the same over-provisioning teams were trying to escape, now moved inside a single GPU. If two models spike together later, they will take the whole card down. For instance, if the extraction model hits an unusually long document and asks for more memory than its slice allows, it does not just fail on its own—it takes OCR, embedding, and reranking with it.

The article explains that the ideal solution is one server that holds all four models instead of four servers holding one each. It owns the card's memory so nothing has to be split by hand; it sees every request and can decide what runs first; and it loads a model when a request needs it, then drops it when memory runs short, so a model nobody is calling is not sitting on memory it does not need. Standard serving tools achieve none of this.

The Superlinked Inference Engine (SIE) is presented as an open-source solution. It runs as one cluster inside your own cloud, built for pipelines where several small models of different kinds run back to back on shared GPUs. It runs every kind of model through one API with four calls: encode turns text or images into vectors; score reranks a query against a set of documents (returns a number); extract pulls structured fields, entities, or markdown out of a raw document (returns span of text); and generate runs an open LLM on a prompt (returns text and token usage). TEI can hand back vectors and relevance scores, but one TEI process serves exactly one model, so embedding plus reranking already means two separate deployments. TEI does not cover extraction, OCR, vision, or generation, so each still needs a server of its own. A single SIE cluster can serve all four shapes in one server.

Even inside a full batch there is a second kind of waste. Requests come in different lengths, and the simple way to run them together is to pad the short ones out to match the longest, then process the padding as if it were real. SIE batches by compute cost instead of item count, grouping requests of similar length and padding only to the longest item in that batch, never to a global maximum, so almost none of the card's time goes to padding. Standard stacks route each request to a worker first, and every worker batches only the slice of traffic it happens to receive, so queues run unevenly and mixed sizes pack poorly. SIE inverts this order: the gateway publishes all work into one shared queue, and each worker pulls from it.

The article frames a practical problem that arises in production AI systems: teams building multi-step agent pipelines naturally move toward smaller, task-specific models to cut per-call inference costs, but this move only works if the underlying infrastructure can actually consolidate those models onto shared hardware. The infrastructure mismatch is not accidental. vLLM and Hugging Face's TEI were each designed to serve one class of models efficiently—LLMs and embeddings respectively—and neither was built to coordinate with others or dynamically share GPU memory. As a result, when teams deploy a typical agent pipeline that needs document parsing, embedding, reranking, entity extraction, and safety checks, they end up running four or more separate server processes on separate GPUs, each pre-allocating 90% of its card's memory upfront and unable to yield to others. This recreates the exact waste—idle GPU time billed at the hourly rate—that the switch to smaller models was supposed to escape.

The article argues that solving this problem requires a fundamentally different architectural approach: one server that owns the entire GPU's memory, sees all incoming requests, and can decide scheduling and model loading dynamically. The Superlinked Inference Engine is presented as an open-source implementation of this design, capable of running diverse model architectures (embeddings, rerankers, OCR, vision models, extraction models, and generation LLMs) through a single API and a shared serving layer.