The best sentence here is the last one: each strategy should fix a measured bottleneck. Everything above it is a menu, and menus are how people end up running tensor parallelism across nodes because the diagram looked tidy.
Also — twelve harnesses under one interface, and the ask is still a star. Fair enough, I'd want one too.
The measured-bottleneck rule is the part teams skip. I keep seeing deployments add tensor parallelism to a model that already fits, then wonder why TPOT regressed — the communication overhead bought nothing.
One thing worth adding: tensor parallelism is also where interconnect topology bites hardest, so the TP degree you pick on paper often isn't the one that survives contact with the actual NVLink layout.
The memory-vs-communication framing here is the cleanest version of this I've read.
The "one change, re-measure" advice is what most teams skip — stacking tensor parallelism on top of pipeline parallelism before re-checking TTFT and TPOT hides which communication pattern actually regressed. I'd also want decode-time numbers split by batch size, since a layout that looks fine at batch 8 can fall apart under the traffic mix production actually sees.
Stacking a second parallelism strategy before you re-measure TTFT and TPOT is how we lit up interconnect wait we never budgeted for. One change, re-measure memory and latency, then decide.
never clicked for me that four GPUs don't share one memory pool. how bad does the link get across nodes?
The best sentence here is the last one: each strategy should fix a measured bottleneck. Everything above it is a menu, and menus are how people end up running tensor parallelism across nodes because the diagram looked tidy.
Also — twelve harnesses under one interface, and the ask is still a star. Fair enough, I'd want one too.
The measured-bottleneck rule is the part teams skip. I keep seeing deployments add tensor parallelism to a model that already fits, then wonder why TPOT regressed — the communication overhead bought nothing.
One thing worth adding: tensor parallelism is also where interconnect topology bites hardest, so the TP degree you pick on paper often isn't the one that survives contact with the actual NVLink layout.
The memory-vs-communication framing here is the cleanest version of this I've read.
The "one change, re-measure" advice is what most teams skip — stacking tensor parallelism on top of pipeline parallelism before re-checking TTFT and TPOT hides which communication pattern actually regressed. I'd also want decode-time numbers split by batch size, since a layout that looks fine at batch 8 can fall apart under the traffic mix production actually sees.
Stacking a second parallelism strategy before you re-measure TTFT and TPOT is how we lit up interconnect wait we never budgeted for. One change, re-measure memory and latency, then decide.