Serving an LLM needs memory for three things: the weights, the KV cache for every token in flight, and runtime overhead. Enter your model and workload to see the total, and whether one 8-GPU node holds it: NVIDIA B200 (1,440 GB) or NVIDIA B300 (2,304 GB).
For mixture-of-experts models, enter the total parameter count: every expert has to be in memory even if only a few run per token. Models with multi-head latent attention (DeepSeek-style) keep a much smaller KV cache than this formula assumes, so treat the KV figure as an upper bound for them.
Weights = parameters × bytes per parameter: 2 for BF16/FP16, 1 for FP8, 0.5 for 4-bit formats. A 70B model is about 140 GB in BF16 and 70 GB in FP8.
KV cache = 2 (keys and values) × layers × KV heads × head dimension × bytes per value × tokens in flight, where tokens in flight is context length × concurrent requests. Grouped-query attention (few KV heads) is why modern models fit long contexts at all.
Overhead covers activations, CUDA graphs and fragmentation. 10% is a reasonable starting point for inference servers like SGLang and vLLM; training needs far more (optimizer state and gradients), so size training jobs separately.
A model also has to split cleanly across GPUs: with tensor parallelism over 8 GPUs, each GPU holds roughly an eighth of the weights and KV cache, so the per-GPU figure must stay under one GPU's memory.