Six common strategies for LLM inference in production

The newsletter sets out the constraint that drives all of this: running a model across four GPUs does not merge their memory into one shared pool, so data stored on another GPU must cross the interconnect before a layer can use it. That is why parallelism is presented as a trade-off — local memory or compute limits go away, but communication overhead arrives in their place.

The piece then matches each strategy to the limit it removes. Tensor parallelism splits matrix operations inside each transformer layer, cutting weight memory per GPU while forcing the GPUs to combine partial results throughout the layer stack. Pipeline parallelism assigns layer groups to different GPUs, moving activations between stages and leaving some stages idle when micro-batching is insufficient. Context parallelism splits sequence positions two ways: prefill context parallelism divides a long input across GPUs to target lower TTFT, while decode context parallelism divides the stored KV history to increase cache capacity, with every decoder layer merging partial attention results. Expert parallelism distributes MoE experts across GPUs, adding network cost and idle time from token routing and uneven expert load. Hybrid parallelism combines several of these in one deployment, for instance tensor parallelism within a node, pipeline parallelism across nodes, and data parallelism across model replicas, matching each communication pattern to the available links at the cost of more coordination paths.

The practical advice is to treat each method as a fix for a measured bottleneck: tensor or pipeline parallelism when weights do not fit, the corresponding context parallelism when long-prompt prefill or KV-cache capacity is the limit, expert parallelism for MoE models that need expert weights placed, and data-parallel replicas when one replica meets its latency target but requests still queue. Tensor parallelism belongs on the fastest links, and after every change the relevant metrics should be measured again before another method is added.