DEV Community

Manohar Papasani
Manohar Papasani

Posted on

When a single GPU isn't sufficient to host or compute the model, how would you split a large LLM across multiple GPUs or…

#aiinterview #llminference #genai #aiarchitect

Asked in an LLM inference systems engineer interview.

What they are really testing:

Whether you know where the communication cost lands in each parallelism scheme and how that maps onto hardware topology, not just the names of the schemes.

A strong answer:

  • Start with memory. Weights, KV cache and activations set the minimum GPU count. Size the KV cache per token and per concurrent request before choosing a layout.
  • Tensor parallelism shards each layer's matrices across GPUs. It needs collective all-reduces in every transformer block, so it belongs inside a node on NVLink. It lowers per-token latency but is sensitive to interconnect bandwidth.
  • Pipeline parallelism assigns contiguous layer ranges to stages and passes activations between them. Communication is point-to-point and much smaller, so it suits cross-node links. However, each token traverses every stage, which adds hop latency and pipeline bubbles that micro-batching or multiple in-flight requests must hide.
  • A common production layout is tensor parallelism within each node and pipeline parallelism across nodes, with data-parallel replicas behind a router for extra throughput.
  • Balance stages by compute and memory, not just layer count. Prefill is heavier than decode, and embedding and output layers add uneven load. The slowest stage sets the throughput.
  • For reliability, use a separate process group per replica, run health checks with fast failover, and never mix replicas with different weight versions or quantization settings behind one endpoint.

Common mistakes:

  • Using pipeline parallelism to cut latency. It usually adds per-token latency and mainly helps throughput.
  • Running tensor parallelism across slow inter-node Ethernet, where all-reduce time dominates each step.
  • Splitting layers evenly by count and ignoring that uneven stage costs create the bottleneck.

How would you split a 70B-class model across eight GPUs on two nodes? Share your layout and the trade-offs you'd weigh.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)