# Inside TPU and GPU Clusters: The Anatomy of Collective Communication # 深入 TPU 与 GPU 集群内部：集合通信剖析 ## A deep dive into All-Gather, Reduce-Scatter, All-Reduce, and All-to-All across TPU and GPU clusters. ## 深入探讨 TPU 和 GPU 集群中的 All-Gather（全收集）、Reduce-Scatter（规约散播）、All-Reduce（全规约）与 All-to-All（全对全）操作。 July 14, 2026 2026年7月14日 --- In this post, I'll do a deep dive into TPU and GPU cluster topologies, and cover the core collective operations used during transformer training and inference. 在本文中，我将深入探讨 TPU（张量处理单元，谷歌专为机器学习定制的 ASIC）和 GPU（图形处理单元，如英伟达的 AI 加速卡）的集群拓扑结构，并涵盖 Transformer 模型在训练和推理过程中使用的核心集合通信操作。 Why should we care? 我们为什么要关注这些？ In 2026, training and serving transformers is a massively distributed systems problem. 在 2026 年，训练和部署 Transformer 架构的模型（随着模型参数量跃升至万亿甚至更高规模，单张卡的显存和算力早已遇到瓶颈）已经成为了一个极其庞大的分布式系统难题。 To shard models across a cluster, we rely on techniques such as **data parallelism**, **tensor/model parallelism**, **FSDP**, and **expert parallelism**. Under the hood, these techniques are built on a small set of core collective operations: 为了将庞大的模型切片（Shard）并分布到整个集群中，我们依赖于**数据并行（Data Parallelism）**、**张量/模型并行（Tensor/Model Parallelism）**、**FSDP（Fully Sharded Data Parallel，完全分片数据并行，用于将优化器状态、梯度和参数分片到不同设备上）**以及**专家并行（Expert Parallelism）**等技术。在底层，这些技术都是建立在一小撮核心的集合通信（Collective Communication）操作之上的： 1. Data parallel training requires gradient synchronization during backprop, this is typically implemented with **all-reduce**. As we'll see later in this post, all-reduce itself can be decomposed into **reduce-scatter** followed by **all-gather**. 数据并行训练在反向传播（Backprop）期间需要进行梯度同步，这通常是通过 **All-Reduce（全规约）** 来实现的。正如我们将在后文中看到的，All-Reduce 本身可以被分解为先执行 **Reduce-Scatter（规约散播）**，随后再执行 **All-Gather（全收集）**。 2. Tensor parallelism and FSDP rely heavily on **all-gather** and **reduce-scatter** in the forward and backward passes. 张量并行和 FSDP 在前向和反向传播中严重依赖 **All-Gather** 和 **Reduce-Scatter**。 3. Expert parallelism, used in MoE models, rely on the **all-to-all** primitive. 在混合专家（MoE，Mixture-of-Experts）模型中使用的专家并行，则依赖于 **All-to-All（全对全）** 通信原语。 These are just a few important examples, but they're enough to show why understanding collective communication is useful -> if you want to reason about the performance of modern transformer systems, you eventually have to reason about how data moves through the cluster. 这些仅仅是几个重要的例子，但足以说明为什么理解集合通信如此有用——如果你想分析和优化现代 Transformer 系统的性能，你最终必须去弄清楚数据是如何在整个物理集群中流动的。 We'll start from the hardware, with the TPU and GPU cluster topologies. Understanding the physical layout of the cluster makes the collective algorithms much more grounded and easier to reason about. 我们将从硬件拓扑讲起，即 TPU 和 GPU 的集群拓扑结构。了解集群的物理布局，能让这些集合通信算法变得更加接地气，也更容易理解。 From there, we'll dive into the most common implementations of the core collective operations. 在此基础上，我们将深入剖析这些核心集合通信操作最常见的实现方式。 I'll focus primarily on ring-style algorithms, since they are the natural starting point for large-message communication. 我将主要把重点放在环形（Ring-style）算法上，因为它们是理解大消息（大负载）数据通信的最自然起点。 For smaller payloads, latency starts to dominate, and tree-style algorithms can be a better fit (as they require only `log2` steps). 对于较小的数据负载，网络延迟开始占据主导地位，此时树形（Tree-style）算法可能会更合适（因为它们只需要 `log2(N)` 步即可完成通信）。 This post is structured into seven parts: 本文分为以下七个部分： 1. [TPU cluster topology](#cpt1): Superpods, Slices, DCN, PCIe, ICI [TPU 集群拓扑](#cpt1)：超级集群（Superpods）、切片（Slices）、数据中心网络（DCN）、PCIe、芯片间互连（ICI） 2. [Inside All-Gather](#cpt2): 1D/2D Rings, and Chains [深入 All-Gather](#cpt2)：一维/二维环（Rings）与链（Chains） 3. [Reduce-Scatter (and All-Reduce)](#cpt3): The Dual of All-Gather [深入 Reduce-Scatter（与 All-Reduce）](#cpt3)：All-Gather 的对偶操作 4. [All-to-All](#cpt4): A Sharded Transpose [深入 All-to-All](#cpt4)：一种分片矩阵转置 5. [NVIDIA GPU cluster topology](#cpt5): Nodes, Scalable Units, Fat Tree [NVIDIA GPU 集群拓扑](#cpt5)：节点（Nodes）、可扩展单元（Scalable Units）、胖树架构（Fat Tree） 6. [GPU Collectives Within the Node](#cpt6): Rings, Trees, and SHARP [节点内的 GPU 集合通信](#cpt6)：环形、树形算法与 SHARP 技术 7. [GPU Collectives Across Nodes](#cpt7): Hierarchical Algorithms over InfiniBand [跨节点的 GPU 集合通信](#cpt7)：基于 InfiniBand 的分层通信算法 --- ## TPU cluster topology / TPU 集群拓扑 I'll start with TPUs because their topology is more uniform, and therefore arguably easier to reason about, than GPU cluster topology. 我将从 TPU 开始讲起，因为它们的拓扑结构更加均匀（Uniform），因此相对于 GPU 集群拓扑来说更容易进行推理。 The key difference between TPU and GPU clusters is the **nearest-neighbor connectivity**. TPU chips are connected directly to neighboring TPU chips, and each chip has either **4 or 6 nearest neighbors**, depending on the TPU generation: TPU 和 GPU 集群之间的关键区别在于**最近邻连接（Nearest-neighbor connectivity）**。TPU 芯片会直接与其相邻的 TPU 芯片物理互连，根据 TPU 代际的不同，每个芯片有 **4 个或 6 个最近邻节点**： 1. TPU `v2`, `v3`, `v5e`, and `v6e` use a 2D torus topology, with 4 nearest neighbors. TPU `v2`、`v3`、`v5e` 和 `v6e` 使用二维环面拓扑（2D Torus Topology），拥有 4 个最近邻节点。 2. TPU `v4p`, `v5p`, `TPU7x` (Ironwood), and `8t` use a 3D torus topology, with 6 nearest neighbors. TPU `v4p`、`v5p`、`TPU7x` (Ironwood) 和 `8t` 使用三维环面拓扑（3D Torus Topology），拥有 6 个最近邻节点。 ![Figure 1: TPU connectivity classes](https://www.aleksagordic.com/blog/collective-operations/tpu_classes.png)

Figure 1: TPU connectivity classes / 图 1：TPU 连接分类

> 💡 **Boardfly topology / Boardfly（板蝇）拓扑:** > Notably, Google's [new inference TPU chip](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive) [1] `8i` deviates from the 2D/3D torus, and instead uses boardfly, a hierarchical high-radix topology. In this post I'll ignore it. > 值得注意的是，谷歌[用于推理的新型 TPU 芯片](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive) [1] `8i` 偏离了传统的 2D/3D 环面结构，转而采用一种名为“Boardfly”的分层高基数（High-radix）拓扑网络。但在本文中，我们暂不讨论它。 Here is one way to build intuition for the 4-neighbor, 2D torus pattern. A 2D torus can be represented as a grid with wraparound/periodic boundaries: moving off the left edge brings you back on the right edge (and vice versa), and moving off the top edge brings you back on the bottom edge (and vice versa). Bear in mind that the TPU torus is a discrete grid; imagine it overlaid on this donut. The visualization is just for intuition: 这里提供一种方法来帮助大家直观理解 4 邻居、二维环面（2D torus）的模式。二维环面可以被看作是一个具有折返（Wraparound）或周期性边界的网格：从左侧边缘移出后，会直接回到右侧边缘（反之亦然）；从顶部边缘移出后，会直接回到底部边缘（反之亦然）。请记住，TPU 环面是一个离散的网格；你可以想象将它覆盖在这个甜甜圈（Donut）表面上。这种可视化仅为了直观理解： ![Figure 2: 2D torus intuition: a grid with wraparound boundaries](https://www.aleksagordic.com/blog/collective-operations/torus.jpg)

Figure 2: 2D torus intuition: a grid with wraparound boundaries / 图 2：二维环面的直观理解：带有边界折返的网格

Similarly, here is the 6-neighbor, 3D torus connectivity pattern: 同样地，这是 6 邻居、三维环面（3D torus）的连接模式： ![Figure 3: 3D torus connectivity](https://www.aleksagordic.com/blog/collective-operations/topo_3d.png)

Figure 3: 3D torus connectivity / 图 3：三维环面连接方式

This looks a bit messy, but the idea is simple -> each chip has neighbors along ±x, ±y, and ±z, and the edges wrap around in all three dimensions. 这张图看起来有点乱，但其核心思想很简单——每个芯片在 ±x、±y 和 ±z 轴上都有邻居，且所有三个维度上的边缘链路都会环绕折返。 > 💡 **Interactive Visualization Tool / 交互式可视化工具:** > Here is a neat little [visualization tool](https://tpu-visualizer.uc.r.appspot.com/) [2] I used to generate the above image, you can use it to interact with these more complex topologies. > 这是一个非常巧妙的[可视化小工具](https://tpu-visualizer.uc.r.appspot.com/) [2]，我用来生成了上面的图像，你可以使用它来与这些更复杂的拓扑进行交互。 We'll use `v5e` as the running example in this post, since 2D connectivity is easier to visualize. 本文将使用 TPU `v5e` 作为贯穿全文的示例，因为二维连接在视觉上更容易呈现。 TPU chips communicate with their neighbors over **ICI**, short for inter-chip interconnect. TPU 芯片与其邻居之间通过 **ICI** 进行通信，ICI 是芯片间互连（Inter-Chip Interconnect）的缩写。 The largest ICI-connected island of TPU chips is called a **TPU Pod**. You'll also sometimes hear people refer to this as a **"superpod"**; I'll use the terms interchangeably here. 通过 ICI 连接在一起的最大规模的 TPU 芯片孤岛（Island）被称为一个 **TPU Pod**。你有时也会听到人们称之为 **“超级集群”（Superpod）**；在这里，我将交替使用这两个术语。 For example, a `v4` pod contains 16 × 16 × 16 chips, for a total of 4096 TPUs. A `v5p` pod contains 16 × 20 × 28 chips, or 8960 TPUs! 例如，一个 `v4` Pod 包含 16 × 16 × 16 个芯片，总计 4096 个 TPU。而一个 `v5p` Pod 包含 16 × 20 × 28 个芯片，即 8960 个 TPU！ > 💡 **GPUs scale-up domains / GPU 向上的扩展域:** > Contrast this with GPUs, where the scale-up domain has traditionally been much smaller: commonly 8 GPUs in a single NVLink domain, and more recently 72 GPUs in an NVLink domain with NVIDIA GB200 NVL72. We'll analyze GPUs in much more depth a bit later. > 将此与 GPU 进行对比（在引入大规模机架级互连技术前），GPU 传统的垂直扩展（Scale-up）域要小得多：通常在单个 NVLink 域中只有 8 个 GPU，最近在 NVIDIA GB200 NVL72 架构中，一个 NVLink 域内可达到 72 个 GPU。我们稍后会更深入地分析 GPU 架构。 For TPU pods whose chips have 6 neighbors,the smallest full 3D torus is a 4×4×4 cube. If you request a smaller topology, for example 2×2×2, you lose the wraparound links, so the slice stops being a torus and becomes a mesh. 对于拥有 6 个邻居的 TPU Pod 来说，最小的完整 3D 环面结构是一个 4×4×4 的立方体。如果你在云端申请了更小的拓扑结构（例如 2×2×2），由于失去了边界折返链路（Wraparound links），该切片就不再是一个环面，而退化成了一个普通的网格（Mesh）。 > 💡 **Terminology / 术语解释:** > A TPU slice is a collection of chips within a single TPU Pod connected through ICIs (an "ICI-island"). For `v5e` those can be: 2×2, 2×4, 8×8, etc. The largest such slice is the (super)pod itself. > TPU 切片（Slice）是指单个 TPU Pod 内通过 ICI 连接在一起的芯片集合（即“ICI 孤岛”）。对于 `v5e`，可以切分成诸如：2×2、2×4、8×8 等规格。这种切片的最大形态就是（超级）Pod 本身。 This can roughly double the time of ring-style collectives along that axis, as we'll soon see. So this is something to be aware of - if your application is communication-heavy, you may want to request slice shapes that preserve the torus structure, for example 4×4×8. 我们很快就会看到，缺少折返链路可能会导致沿该轴运行环形集合通信算法的时间大约翻倍。因此这是一个需要注意的细节——如果你的应用程序对通信要求很高，你应该去申请保留了环面结构的切片形状，例如 4×4×8。 `v5e`, on the other hand, has a 16×16 2D torus pod. The wraparound exists at size 16, so if you take a smaller slice, such as 8×16, you lose the wraparound along the shorter axis (again, collective ops along that axis pay roughly a 2× penalty). 另一方面，`v5e` 具有 16×16 的二维环面 Pod。只有在尺寸达到 16 时才存在折返链路，因此如果你使用较小的切片（例如 8×16），在较短的轴上就会失去折返连接（同样，沿该轴的集合通信操作将付出约 2 倍的时间惩罚）。 To scale beyond a single pod, TPUs use **DCN**, short for data center networking. DCN has much lower throughput than ICI, so you have to be careful: if too much communication crosses pod boundaries, it can easily become the bottleneck during training. 为了突破单个 Pod 的限制进行更大规模的扩展，TPU 会使用 **DCN**（数据中心网络，Data Center Networking 的缩写）。DCN 的吞吐量远低于 ICI 互连，因此你必须小心：如果有过多的跨 Pod 通信，网络将很容易成为训练过程中的性能瓶颈。 Putting all of that together, let's visualize the topology of a 16×16 `v5e` pod. Spend some time analyzing this (zoom in if needed): 综上所述，让我们可视化一个 16×16 `v5e` Pod 的拓扑结构。花点时间分析一下这张图（需要时可以放大）： ![Figure 4: Topology of a v5e TPU 16x16 superpod](https://www.aleksagordic.com/blog/collective-operations/tpu_topology1.png)

Figure 4: Topology of a v5e TPU 16x16 superpod / 图 4：v5e TPU 16x16 超级集群的拓扑结构

We can connect multiple pods into an even larger compute cluster through a shared DCN fabric (maybe this is what we should call a superpod, heh): 我们可以通过一个共享的 DCN 架构骨干网络（Backbone）将多个 Pod 连接成一个更大的计算集群（也许这才是我们该称之为超级集群的东西，哈）： ![Figure 5: Multiple pods connected through a DCN backbone](https://www.aleksagordic.com/blog/collective-operations/tpu_topology2.png)

Figure 5: Multiple pods connected through a DCN backbone / 图 5：通过 DCN 骨干网连接的多个 Pod

A few more details are worth knowing. 还有一些值得了解的细节。 As can be seen in the figure above, within the pod the 0th row connects back to the 15th row, and the 0th column connects back to the 15th column. This is what makes the topology a torus/donut instead of a regular mesh. The torus geometry constrains path lengths, because data often has to flow through intermediate TPUs to get from the source chip to the target chip. 正如上图所示，在 Pod 内，第 0 行连接回第 15 行，第 0 列连接回第 15 列。正是这一点使得该拓扑结构成为一个环面/甜甜圈，而不是一个普通的网格。由于数据从源芯片到达目标芯片通常必须流经中间的 TPU，环面的几何特性限制了最大路径长度（Hop 数量）。 For example, if TPU `(15, 15)` wants to send data to TPU `(2, 15)`, the shortest path goes through the wraparound link: 例如，如果 TPU `(15, 15)` 想要向 TPU `(2, 15)` 发送数据，最短的路径是经过折返链路： `(15, 15)` → `(0, 15)` →`(1, 15)` → `(2, 15)` instead of going the long way through `(14, 15)`, `(13, 15)`, and so on. 而不是绕远路，依次经过 `(14, 15)`、`(13, 15)` 等。 > 💡 **Extra info / 额外信息:** > TPUs also support “twisted torus” configurations, which alter the wraparound connectivity to reduce hop counts for communication patterns such as All-to-All. This is an implementation detail that improves efficiency, but we don't need it for the purposes of this post. > TPU 还支持“扭曲环面（Twisted Torus）”的配置模式，这种模式改变了边界折返的连接方式，从而能够有效减少 All-to-All 等通信模式下的跳步数（Hop counts）。这是一个用于提升效率的底层实现细节，对理解本文主旨来说不必深入探究。 Each TPU chip is also connected to a "dedicated" CPU host through **PCIe**. In the case of `v5e`, one host is connected to a 2×4 block of TPU chips, giving us 8 PCIe connections between the host and those 8 chips. 每块 TPU 芯片还通过 **PCIe**（高速外设组件互连通道）与“专属”的 CPU 主机（Host）相连。在 `v5e` 的架构中，一台主机连接着一个 2×4 的 TPU 芯片块，这意味着该主机与这 8 个芯片之间存在 8 条独立的 PCIe 连接。 Importantly, notice that to reach the DCN backbone, data has to flow through PCIe first, which means DCN communication is even slower than PCIe. 重点注意，要到达 DCN 骨干网络，数据必须首先经过 PCIe，这意味着 DCN 跨集群通信的速度比 PCIe 还要慢。 Concretely, cross-pod data flows from the source TPU chip's HBM, over PCIe to the source host, then egresses over the DCN fabric, ingresses into the target host, and finally goes back over PCIe into the target TPU chip's HBM. 具体而言，跨 Pod 的数据流动路径是：从源 TPU 芯片的 HBM（高带宽内存）出发，经由 PCIe 传输到源主机，接着从网卡流出进入 DCN 交换网络，再流入目标主机的网卡，最后再次经过 PCIe 写入到目标 TPU 芯片的 HBM 中。 This gives us a natural **bandwidth hierarchy**. The closer we are to the compute die on the TPU chip, the faster the data movement is; the farther we move out into the cluster, the slower it gets. 这就天然形成了一个**带宽层级（Bandwidth Hierarchy）**。数据所在位置离 TPU 芯片上的计算核心越近，数据的移动速度就越快；当数据移动到集群外围时，速度就会变得越来越慢。 Let's put the relevant `v5e` cluster bandwidths on one picture: 让我们在一张图里汇总所有相关的 `v5e` 集群带宽数据： ![Figure 6: Bandwidth hierarchy in a v5e TPU cluster](https://www.aleksagordic.com/blog/collective-operations/pyramid.png)

Figure 6: Bandwidth hierarchy in a v5e TPU cluster / 图 6：v5e TPU 集群中的带宽层级

Now that we have the topology and bandwidth hierarchy in mind, let's look at a few concrete examples of how data actually flows through a TPU slice. 现在我们对拓扑和带宽层级有了概念，接下来通过几个具体的例子看看数据是如何真正在 TPU 切片中流动的。 > 💡 **Book recommendation / 书籍推荐:** > A few of the examples in this blog, as well as the broader motivation for this post, were inspired by the excellent Scaling Book [3], which I highly recommend. > 这篇博客中的一些例子，以及我撰写本文的广泛动机，都受到了优秀的《Scaling Book》[3] 的启发，我强烈推荐这本书。 Suppose we request a 4×4 `v5e` slice from GCP. Since both axes are smaller than 16, we don't get any wraparound links. So this slice is not a torus, it is a regular 2D grid, and some paths between chips are longer than they would be with wraparound links. 假设我们从谷歌云（GCP）上申请了一个 4×4 的 `v5e` 切片。由于长宽均小于 16，该切片没有任何折返链路。因此它不再是一个环面，而是一个常规的二维网格结构，芯片之间的部分传输路径会比拥有折返链路时更长。 Let's pose the following question: how long does it take to move a `(2048, 2048)` `bf16` matrix from TPU chip `(3, 3)` to TPU chip `(0, 0)`? 我们提出如下问题：将一个尺寸为 `(2048, 2048)`、数据类型为 `bf16` 的矩阵从 TPU 芯片 `(3, 3)` 移动到 TPU 芯片 `(0, 0)` 需要多长时间？ ![Figure 7: Moving an 8 MiB bf16 matrix across a 4×4 v5e mesh using two ICI paths](https://www.aleksagordic.com/blog/collective-operations/tpu_example1.png)

Figure 7: Moving an 8 MiB bf16 matrix across a 4×4 v5e mesh using two ICI paths / 图 7：利用两条 ICI 路径在一个 4x4 的 v5e 网格上移动一个 8 MiB 的 bf16 矩阵

Note that we ignored link latency in the calculation above. In practice, ICI links have roughly 1 μs of latency per hop. 请注意，在上述计算中我们忽略了链路延迟。在实际应用中，ICI 链路每跳（Hop）大约存在 1 微秒（μs）的延迟。 In our example, each path is 6 hops long. Since the two paths run in parallel, this adds roughly 6 μs of latency to the transfer, which is negligible here, but may not be negligible for smaller messages. 在我们的例子中，每条路径长达 6 跳。由于两条路径是并行运行的，这大约为传输增加了 6 μs 的延迟，在此场景中可以忽略不计，但如果传输的消息（载荷）较小，延迟就不能再忽略了。 This is why it's important to understand whether we are in a **latency-bound** or **throughput-bound** regime. 这就是为什么理解我们处于**延迟限制（Latency-bound）**还是**吞吐量限制（Throughput-bound）**的区间如此重要。 A simple way to estimate this is to ask: assuming we saturate the unidirectional ICI bandwidth of 45 GB/s, how much data can flow through one link in 1 μs? 一个简单的估算方法是问自己：假设我们能够打满 45 GB/s 的单向 ICI 带宽，在 1 μs 内可以通过一条链路流动多少数据量？ The answer is: 答案是： 45 GB/s × 1 μs = 45 KB So if our message chunks are around this size, latency matters a lot. Ignoring a 1 μs per-hop latency could make the estimate off by a large factor, so the simple bandwidth-only approximation is no longer valid. 因此，如果我们的消息块大约在这个大小（45 KB），延迟就显得极其关键。此时忽略 1 μs 的单跳延迟会导致估算结果偏差巨大，单纯基于带宽的近似计算方法将不再适用。 Let's do one more example, this time using PCIe, ICI, and HBM → VMEM links as well. 让我们再看一个例子，这次同时涉及 PCIe、ICI 以及从 HBM 到 VMEM（片上 SRAM）的链路。 > 💡 **what is VMEM? / VMEM 是什么？** > VMEM is fast on-chip SRAM, roughly equivalent to programmer-managed shared memory on GPUs. It feeds directly into the matmul units; the systolic array, in the case of a TPU chip. > VMEM 是快速的片内 SRAM，大致相当于 GPU 上需要程序员手动管理的共享内存（Shared Memory）。它直接向矩阵乘法单元供数；在 TPU 芯片中，就是直接喂给脉动阵列（Systolic Array）。 > > The details of the TPU chip are not important for understanding the main topic of this post, so we won't spend more time on VMEM here. > TPU 芯片的底层计算细节对于理解本文的主旨并不重要，因此我们在这里就不花更多时间讨论 VMEM 了。 Assume we have a (`128 * 1024`, `128 * 1024`) bf16 matrix sharded over a 4 × 4 TPU slice. Each chip therefore owns a (`32 * 1024`, `32 * 1024`) submatrix (128/4=32). 假设我们有一个尺寸为 (`128 * 1024`, `128 * 1024`) 的 bf16 矩阵，它被分片存储在 4 × 4 的 TPU 切片上。因此，每个芯片上都保存着一个 (`32 * 1024`, `32 * 1024`) 的子矩阵（由于 128 / 4 = 32）。 Also assume these submatrices have been offloaded to host DRAM. 同时假设这些子矩阵被卸载（Offload）到了主机端的 DRAM（即 CPU 内存）中。 How long does it take to move all of this data to TPU `(0, 0)` and do a matmul with a (`128 * 1024`, `128`) bf16 matrix? 要想把所有这些数据转移到 TPU `(0, 0)`，并与一个尺寸为 (`128 * 1024`, `128`) 的 bf16 矩阵完成一次矩阵乘法（Matmul），需要多长时间？ ![Figure 8: Gathering a sharded matrix onto TPU(0,0) for matmul](https://www.aleksagordic.com/blog/collective-operations/tpu_example2.png)

Figure 8: Gathering a sharded matrix onto TPU(0,0) for matmul / 图 8：将分片矩阵全收集到 TPU(0,0) 以执行矩阵乘法

With the TPU topology and bandwidth hierarchy in mind, we're ready to jump into collective operations. 了解了 TPU 的拓扑结构和带宽层级之后，我们就做好了准备，可以深入探讨集合通信操作了。 Let's start with All-Gather. 我们先从全收集（All-Gather）开始。 --- ## Inside All-Gather: 1D/2D Rings, and Chains / 深入 All-Gather：一维/二维环与链 Let's start with a motivating example. 我们从一个动机性例子（Motivating example）开始。 ![Figure 9: All-Gather motivation: gathering shards of A so each chip can run the matmul locally](https://www.aleksagordic.com/blog/collective-operations/gather_motivation.png)

Figure 9: All-Gather motivation: gathering shards of A so each chip can run the matmul locally / 图 9：All-Gather 的作用：收集 A 的全部分片，以便每个芯片都能在本地运行矩阵乘法

So how do we efficiently implement All-Gather? 那么我们应该如何高效地实现 All-Gather 呢？ A common approach is to use a ring algorithm. Before looking at the algorithm itself, let's first understand where this "ring" comes from: 一种常见的方法是使用环形（Ring）算法。在深入研究算法之前，首先让我们弄清楚这个“环”是从哪里来的： ![Figure 10: 1D bidirectional rings naturally appear along both axes of a 16×16 v5e torus](https://www.aleksagordic.com/blog/collective-operations/ring.png)

Figure 10: 1D bidirectional rings naturally appear along both axes of a 16×16 v5e torus / 图 10：在一维双向环中，一个 16×16 的 v5e 环面在两个轴上自然形成了环路

For easier visualization, let's use a smaller ring of 8 TPU chips. In real `v5e` pods, the wraparound happens at size 16, but for the next few diagrams we'll pretend that an 8-chip row also wraps around: 为了便于可视化呈现，我们使用由 8 个 TPU 芯片组成的较小环路来做演示。在真实的 `v5e` Pod 中，边界折返只有在尺寸达到 16 时才会出现，但在接下来的几个图表中，我们将假装这一行 8 个芯片也实现了折返连接： ![Figure 11: Simplified representation of an 8-chip ring](https://www.aleksagordic.com/blog/collective-operations/representation.png)

Figure 11: Simplified representation of an 8-chip ring / 图 11：由 8 个芯片组成的环的简化表示

Here is All-Gather over a bidirectional 1D ring. It's worth spending a bit of time on this diagram, zoom in and follow one shard as it moves around the ring: 这是在一维双向环上执行 All-Gather 的过程。这张图值得多花一点时间仔细研究，请放大图片，并追踪某一个数据切片是如何在这个环中流动的： ![Figure 12: All-Gather over a bidirectional 1D ring](https://www.aleksagordic.com/blog/collective-operations/all_gather1.png)

Figure 12: All-Gather over a bidirectional 1D ring / 图 12：在 1D 双向环上的 All-Gather 过程

As an exercise, what would the algorithm look like if the ICI links were not full-duplex? 作为练习，大家可以思考一下：如果 ICI 链路不是全双工（Full-duplex，既能同时发也能同时收）的，这个算法会变成什么样？ In that case, we could run All-Gather over a unidirectional ring. To keep the diagram small, let's use a hypothetical ring of size `N = 4`, so the algorithm completes in only `N - 1 = 3` steps: 在那种情况下，我们只能在一维单向环上运行 All-Gather。为了使图表保持紧凑，我们假设一个规模为 `N = 4` 的单向环，该算法仅需 `N - 1 = 3` 步就能完成： ![Figure 13: All-Gather over a unidirectional 1D ring](https://www.aleksagordic.com/blog/collective-operations/all_gather2.png)

Figure 13: All-Gather over a unidirectional 1D ring / 图 13：在 1D 单向环上的 All-Gather 过程

More realistically, if we take a TPU slice without wraparound links - for example, a 4 × 4 `v5e` slice - then we no longer have a ring along that axis. Instead, the topology becomes a 1D path, so we fall back to All-Gather over a path (or chain): 更贴近现实的情况是，如果我们获得了一个没有折返链路的 TPU 切片——例如一个 4 × 4 的 `v5e` 切片——那么沿着相应坐标轴的环形结构就不复存在了。此时，拓扑退化为一条一维路径，因此我们不得不回退到沿路径（或链条，Chain）执行的 All-Gather 算法： ![Figure 14: All Gather - chain/path](https://www.aleksagordic.com/blog/collective-operations/all_gather3.png)

Figure 14: All Gather - chain/path / 图 14：基于链/路径的 All-Gather

Finally, sometimes we have an array that is sharded across both axes of the TPU slice, and we want to perform a 2D All-Gather. 最后，有时候我们会遇到一个跨 TPU 切片两个坐标轴均分片存储的数组，我们需要对其执行二维 All-Gather 操作。 In that case, we can use all 4 neighboring ICI links, which gives us a 2× speedup compared to using only a single axis! 在这种场景下，我们可以并行利用所有 4 个邻居方向上的 ICI 链路，相比于仅使用单一坐标轴，这能够带来 2 倍的性能提升！ Let's build intuition for why. Assume a hypothetical 4 × 4 slice with wraparound links along both axes, so both rows and columns form rings: 让我们建立直观理解，看看这是为什么。假设存在一个 4 × 4 的切片，并在两个坐标轴上都有折返链路，因此行和列都构成了环路： ![Figure 15: All Gather over a bidirectional 2D ring](https://www.aleksagordic.com/blog/collective-operations/all_gather4.png)

Figure 15: All Gather over a bidirectional 2D ring / 图 15：在二维双向环上的 All-Gather

That's a wrap for All-Gather. All-Gather 部分就到此结束了。 Next, let's look at its dual, Reduce-Scatter, and then use it to build All-Reduce. 接下来，我们将探讨它的对偶操作 Reduce-Scatter，并用它来构建出 All-Reduce。 --- ## Reduce-Scatter (and All-Reduce): The Dual of All-Gather / Reduce-Scatter 与 All-Reduce：All-Gather 的对偶操作 As before, let's start with a motivating example: 和以前一样，让我们从一个动机性例子开始： ![Figure 16: Reduce-Scatter motivation](https://www.aleksagordic.com/blog/collective-operations/reduce_scatter_motivation.png)

Figure 16: Reduce-Scatter motivation / 图 16：Reduce-Scatter 的作用

So how do we efficiently implement Reduce-Scatter? 那么我们该如何高效地实现 Reduce-Scatter（规约散播）呢？ It turns out to be very similar to All-Gather. In fact, you can think of it as All-Gather's dual: the communication schedule is very similar, but instead of copying shards as they move, we reduce them as they move. 事实证明，它与 All-Gather 非常相似。实际上，你可以把它看作是 All-Gather 的对偶（Dual）操作：二者的通信调度几乎完全相同，只是在数据切片移动的过程中，我们不再是简单地复制数据块，而是对它们进行规约聚合（Reduction，例如求和、取最大值等）。 Because the communication pattern is almost identical, the throughput-bound time complexity is the same: 正因为两者的通信模式几乎一致，它们在吞吐量限制下的时间复杂度也是一样的： ![Figure 17: Reduce-Scatter over a bidirectional 1D ring](https://www.aleksagordic.com/blog/collective-operations/rs1.png)

Figure 17: Reduce-Scatter over a bidirectional 1D ring / 图 17：在一维双向环上的 Reduce-Scatter 过程

> 💡 **Side note: All-Gather/Reduce-Scatter duality / 旁注：All-Gather 与 Reduce-Scatter 的对偶性:** > There is a useful duality between All-Gather and Reduce-Scatter. For many sharding patterns, if the forward pass uses All-Gather, the backward pass uses Reduce-Scatter. Conversely, if the forward pass uses Reduce-Scatter, the backward pass uses All-Gather. This is worth being aware of, but since the focus of this post is the mechanics of collective operations, I won't expand on it further here. > 在 All-Gather 和 Reduce-Scatter 之间存在一种非常有用的对偶性。在许多张量分片模式（如张量并行、FSDP）中，如果前向传播使用的是 All-Gather，那么反向传播就会使用 Reduce-Scatter；反之，如果前向使用了 Reduce-Scatter，反向就会使用 All-Gather。这一点非常值得了解，但由于本文的重点在于集合通信操作的具体底层机制，在这里就不作进一步展开了。 Continuing the motivating example above, we can now see how to build All-Reduce from Reduce-Scatter followed by All-Gather: 继续利用上面的例子，我们现在可以清楚地看到如何通过先执行 Reduce-Scatter，再执行 All-Gather 来构建出 All-Reduce 操作： ![Figure 18: All-Reduce over a bidirectional 1D ring](https://www.aleksagordic.com/blog/collective-operations/all_reduce.png)

Figure 18: All-Reduce over a bidirectional 1D ring / 图 18：在一维双向环上的 All-Reduce 过程

For consistency's sake let's also cover Reduce-Scatter over a hypothetical (because we'd need 16 TPUs not 4 to get a ring on `v5e`) unidirectional 1D ring: 为了保持文章内容的完整性，我们也讨论一下在假设的单向一维环上执行 Reduce-Scatter 的过程（说是“假设”，因为实际上在 `v5e` 架构中我们需要 16 个而不是 4 个 TPU 才能形成真正的环）： ![Figure 19: Reduce-Scatter over a unidirectional 1D ring](https://www.aleksagordic.com/blog/collective-operations/rs2.png)

Figure 19: Reduce-Scatter over a unidirectional 1D ring / 图 19：在一维单向环上的 Reduce-Scatter

Finally, if we take a TPU slice without wraparound links, for example, a 4 × 4 `v5e` slice, then there is no ring structure along that axis. In that case, we fall back to Reduce-Scatter over a 1D path / chain: 最后，如果我们的 TPU 切片没有折返链路（例如一个 4 × 4 的 `v5e` 切片），那么在坐标轴方向上就不会存在环形结构。在这种情况下，我们退回到在一维路径/链条上执行 Reduce-Scatter： ![Figure 20: Reduce-Scatter - chain/path](https://www.aleksagordic.com/blog/collective-operations/rs3.png)

Figure 20: Reduce-Scatter - chain/path / 图 20：基于链/路径的 Reduce-Scatter

As with All-Gather, if the data is sharded across multiple topology axes, we can use more ICI links in parallel. In the throughput-bound regime, the total time scales roughly inversely with the number of topology axes we use. 和 All-Gather 一样，如果数据在多个拓扑轴上都进行了分片，我们可以并行利用更多的 ICI 链路。在吞吐量限制区间内，总耗时大致上与我们使用的拓扑轴数量成反比。 That wraps up Reduce-Scatter and All-Reduce! Reduce-Scatter 和 All-Reduce 的讨论到此结束！ --- ## All-to-All: A Sharded Transpose / All-to-All：一种分片矩阵转置 Let's now look at the final primitive: **All-to-All**. 现在让我们来看看最后一个通信原语：**All-to-All（全对全通信）**。 A canonical place where All-to-All shows up is in MoE (Mixture-of-Experts) models. All-to-All 出现的一个典型场景就是在 MoE（混合专家）模型中。 > 💡 **MoE simplified / 简化的 MoE:** > For simplicity, assume top-1 routing and one expert per chip. > 为了便于说明，我们假设使用的是 Top-1 路由机制，并且每个芯片上只部署了一个专家（Expert）。 In an MoE layer, each token is assigned to one expert by the router. You can think of the router as attaching a destination expert ID to each token. 在 MoE 层中，每个 token 都会被路由器（Router）分配给一个对应的专家。你可以把路由器想象成是在给每个 token 贴上一个目标专家的 ID 标签。 Assume that the Expert 0 lives on TPU 0, Expert 1 lives on TPU 1, and so on. Then the expert ID tells us which TPU chip the token needs to be sent to. 假设专家 0 位于 TPU 0 上，专家 1 位于 TPU 1 上，以此类推。那么专家的 ID 就等于告诉了我们应该把这个 token 发送到哪个 TPU 芯片上。 So each chip starts with a local batch of tokens, but those tokens may be destined for many different experts on many different chips. 因此，在初始阶段，每个芯片都拥有属于自己本地批次的 tokens，但这些 tokens 的目标归宿可能指向部署在许多其他不同芯片上的各个专家。 All-to-All is the collective that performs this exchange: every chip sends its tokens for Expert 0 to TPU 0, the tokens for Expert 1 to TPU 1, and so on. All-to-All 正是执行这一庞大重组交换的集合通信操作：每块芯片都会将发给专家 0 的 token 打包发送给 TPU 0，将发给专家 1 的 token 发送给 TPU 1，以此类推。 In other words, All-to-All is a kind of distributed transpose: we start grouped by source chip, and after the collective we are grouped by destination expert. 换句话说，All-to-All 就是一种分布式的矩阵转置：我们一开始是按照**源芯片**来对数据进行分组的，而在执行完该集合通信后，数据就变成了按**目标专家**来分组排列。 For the diagrams below, we'll assume a perfectly balanced routing pattern: each chip has the same amount of data destined for every other chip. Real MoE routing can be imbalanced, but the balanced case gives us the cleanest mental model for understanding the All-to-All primitive. 在下方的图表中，我们假设了一种完美平衡的路由模式：每块芯片发往所有其他芯片的数据量都是完全相等的。真实的 MoE 模型路由往往是不平衡的，但平衡情况能为我们理解 All-to-All 这一通信原语提供一个最纯粹的心智模型。 Here is how we do All-to-All over a bidirectional 1D ring: 下面是在一维双向环上执行 All-to-All 的图解： ![Figure 21: All-to-all over a bidirectional 1D ring](https://www.aleksagordic.com/blog/collective-operations/ata1.png)

Figure 21: All-to-all over a bidirectional 1D ring / 图 21：在一维双向环上的 All-to-All 过程

Here is All-to-All over a unidirectional 1D ring: 这是在一维单向环上的 All-to-All 过程： ![Figure 22: All-to-All over a unidirectional 1D ring](https://www.aleksagordic.com/blog/collective-operations/ata2.png)

Figure 22: All-to-All over a unidirectional 1D ring / 图 22：在一维单向环上的 All-to-All

Same story as before: without wraparound links, the ring becomes a path. So on a 4 × 4 `v5e` slice, All-to-All falls back to a 1D path / chain algorithm: 情况与之前类似：如果没有折返链路，环形结构就会变成一条普通的传输路径。因此，在一个 4 × 4 的 `v5e` 切片上，All-to-All 也只能回退到基于一维路径/链条的通信算法： ![Figure 23: All-to-all - chain/path](https://www.aleksagordic.com/blog/collective-operations/ata3.png)

Figure 23: All-to-all - chain/path / 图 23：基于链/路径的 All-to-All

To wrap up the TPU part of the blog, let's summarize the throughput-bound communication-time results: 作为博客中关于 TPU 部分的收尾，我们来总结一下在吞吐量限制下各个集合操作的通信耗时情况： ![Figure 24: Summary of TPU collective communication costs](https://www.aleksagordic.com/blog/collective-operations/totaltime.png)

Figure 24: Summary of TPU collective communication costs / 图 24：TPU 集合通信成本总结

Next up, NVIDIA GPUs! 接下来，我们要讨论的是英伟达（NVIDIA）的 GPU！ --- ## NVIDIA GPU cluster topology: Nodes, Scalable Units, Fat Tree / NVIDIA GPU 集群拓扑：节点、可扩展单元、胖树架构 In contrast to TPU clusters, which use nearest-neighbor torus connectivity, GPU clusters are usually organized as **hierarchical switching networks**. 相较于使用最近邻环面连接模式的 TPU 集群，GPU 集群通常被组织为**分层交换网络（Hierarchical Switching Networks）**。 In this post, I'll focus on the NVIDIA DGX H100 SuperPod **reference architecture** -> a 1024-GPU cluster organized as a "fat tree". I'll explain what "fat tree" means in a moment. 在本文中，我将重点关注 NVIDIA DGX H100 SuperPod 的**参考架构**——这是一个由 1024 块 GPU 组成的集群，以“胖树（Fat tree）”结构组织。稍后我将解释“胖树”的具体含义。 I'll focus solely on the compute fabric: the network used for GPU-to-GPU communication during distributed training. I'll ignore the storage fabric used for checkpoints, weights, logs, and datasets; UFM for InfiniBand monitoring/management; and the in-band / out-of-band management fabrics used for cluster operations. i.e. the other subsystems that make up a functional NVIDIA cluster. 我将仅仅聚焦于“计算网络（Compute Fabric）”本身：也就是在分布式训练期间专门用于 GPU 到 GPU 之间通信的高速网络。我将忽略用于存储检查点（Checkpoints）、权重、日志和数据集的存储网络；忽略用于 InfiniBand 网络监控和管理的 UFM；以及用于集群运维的带内/带外管理网络。也就是忽略构成一个完整 NVIDIA 集群正常运作的其他子系统。 Here are the basics. 以下是基本概念。 The first organizing unit is the **node**. In this section, I'll use "node" to mean the local NVLink / NVSwitch **scale-up domain**. For example, a DGX H100 node has 8 H100 GPUs connected through NVLink fabric (and e.g. GB200 NVL72 has 72 GPUs). 最底层的组织单元是**节点（Node）**。在这一节中，“节点”专门代指局部的 NVLink / NVSwitch **向上扩展域（Scale-up domain）**。例如，一台 DGX H100 节点拥有 8 块通过 NVLink 网络互连的 H100 GPU（而在最新一代如基于 Blackwell 架构的 GB200 NVL72 方案中，一个节点更是集成了高达 72 块 GPU）。 > 💡 **Terminology note / 术语解释:** > NVSwitch / NVLink fabric is just NVIDIA's terminology for high bandwidth switches / interconnects that connect the GPUs. > NVSwitch / NVLink Fabric 仅仅是 NVIDIA 对内部用于连接各 GPU 的高带宽交换机及互连通道的专有叫法。 Inside the node, GPUs are effectively connected all-to-all through the NVSwitch fabric: every GPU can reach every other GPU in one NVSwitch hop. 在节点内部，GPU 实质上是通过 NVSwitch 交换网络实现了“全对全”物理连接的：任何一块 GPU 都可以在经历单一 NVSwitch 跳步（Hop）后直达该节点内的任何其他 GPU。 Nodes are then connected together through InfiniBand (IB), which forms the **scale-out** network. In the DGX H100 SuperPod reference architecture, 32 nodes form a **Scalable Unit** (SU), connected through InfiniBand **leaf switches**. Multiple SUs are then connected through higher-level **spine switches**. 接下来，多个节点之间通过 InfiniBand（IB）网络相互连接，由此构成了**横向扩展（Scale-out）**网络。在 DGX H100 SuperPod 参考架构中，每 32 个节点组成一个**可扩展单元（Scalable Unit，简称 SU）**，通过 InfiniBand **叶交换机（Leaf switches）**进行连接。多个可扩展单元随后再通过更高层级的**脊交换机（Spine switches）**进行互连。 This hierarchy forms a fat tree. 这样的一种分层设计便构成了所谓的“胖树（Fat Tree）”。 This architecture is called a "fat tree" because the tree gets "fatter" toward the root: upper levels have more aggregate bandwidth, so traffic from many nodes does not collapse into a narrow bottleneck. 之所以称之为“胖树”，是因为从网络末端向核心根部走时，这棵树会变得越来越“胖”：上层节点拥有更高的聚合带宽，从而能够确保汇聚自下层众多节点的庞大流量不会由于端口变窄而造成严重瓶颈。 A **full fat tree** is the non-oversubscribed / special version of this idea. At each level, the uplink bandwidth matches the downstream injection bandwidth (this will become much clearer in the figure below). **完全胖树（Full fat tree）**则是这一理念下无收敛（Non-oversubscribed）的特殊理想版本。在网络分层的任何一级中，其上行聚合带宽与下行注入带宽都是完全匹配、1:1 等价的（在稍后的图表中这一点会看得非常清楚）。 > 💡 **Terminology note / 术语解释:** > Oversubscription means the downstream devices can inject more traffic than the upstream links can carry. For example, if a group of nodes can inject 12.8 TB/s into the leaf layer, but the leaf switches only have 6.4 TB/s of uplink bandwidth to the spine layer, then the fabric is 2:1 oversubscribed. A full fat tree is non-oversubscribed: at each level, upstream bandwidth matches downstream injection bandwidth. > 超额订阅（Oversubscription，或称收敛比）意味着下游设备能够注入网络的流量超出了上游链路所能承载的极限。例如，如果一组节点能够向叶交换机层注入 12.8 TB/s 的流量，但叶交换机连接到脊交换机层只有 6.4 TB/s 的上行带宽，那么该网络的超额订阅比就是 2:1（收敛比为 2:1）。而一个完全胖树架构是无超额订阅（无收敛）的：在每一个层级，上行带宽完全等于下游注入的带宽。 The key consequence of having a full fat tree is **full bisection bandwidth**. 采用完全胖树架构所带来的核心结果就是实现了**完全对分带宽（Full Bisection Bandwidth，也称截面带宽）**。 **Bisection bandwidth** is the bandwidth available across an equal split of the cluster. So if we split a 128-node cluster into two groups of 64 nodes, each side can communicate with the other at its _full_ aggregate injection bandwidth! **对分带宽**是指如果你将整个网络集群平均切割成等同的两半，它们两半之间能够提供多大的通信带宽。所以，假设我们将一个由 128 个节点组成的集群平分成两组，每组 64 个节点，其中的任何一侧都能以其*所有的*聚合注入带宽上限与另一侧通信！ More generally, for any partition, the cross-partition bandwidth is limited by the number of nodes on the smaller side of the split. 在更普遍的意义上，无论你怎么分割集群，跨分区的带宽上限总是受限于切分后节点数量较少的那一侧。 For example, each DGX H100 node has 8 × 50 GB/s links into the IB compute fabric, giving it 400 GB/s of unidirectional injection bandwidth. In a full fat tree, any group of N nodes can communicate across a partition at N × 400 GB/s in one direction, assuming N is the smaller side of the split. 例如，一台 DGX H100 节点拥有 8 条单向速度为 50 GB/s 的网卡链路接入 IB 计算网络中，这为其赋予了 400 GB/s 的单向数据注入带宽。在一个完全胖树架构内，任何由 N 个节点组成的分组都能以 N × 400 GB/s 的单向带宽横跨分区进行通信，前提是 N 是分割后较小的一方。 Inside a node, the same idea applies to the local NVLink/NVSwitch fabric. If we split the 8 GPUs into two groups of 4, one side can send to the other at: 在一个节点内部，同样的理念也适用于本地局域的 NVLink/NVSwitch 架构。如果我们将节点内的 8 块 GPU 平分为两组，每组 4 块，一侧能够向另一侧发送数据的峰值速度为： 4 × 450 GB/s = 1.8 TB/s (which is the max bandwidth for those 4 GPUs!) 4 × 450 GB/s = 1.8 TB/s（这也是这 4 块 GPU 理论上能达到的最大物理带宽边界！） Since the fabric is full-duplex, the bidirectional bisection bandwidth is: 因为该网络链路是全双工运作的，双向对分带宽则是： 2 × 1.8 TB/s = 3.6 TB/s 2 × 1.8 TB/s = 3.6 TB/s At the Scalable Unit level, we have 32 nodes. For a true bisection, we split the SU into two groups of 16 nodes. One side can send to the other at: 在单个可扩展单元（SU）层级上，由于包含 32 个节点。为了计算真正的对分带宽，我们将 SU 分为两个均包含 16 个节点的组群。一侧发送给另一侧的速度为： 16 × 400 GB/s = 6.4 TB/s (bidirectional bisection bw -> 12.8 TB/s) 16 × 400 GB/s = 6.4 TB/s（双向对分带宽 -> 12.8 TB/s） Finally, if we split the cluster into 64 nodes and 64 nodes, one side can send to the other at: 最后，如果我们将整个包含 128 个节点的集群分成两半（各 64 个节点），单向跨区通信带宽上限是： 64 × 400 GB/s = 25.6 TB/s (bidirectional bisection bw -> 51.2 TB/s) 64 × 400 GB/s = 25.6 TB/s（双向对分带宽 -> 51.2 TB/s） For an uneven partition, say 88 nodes on one side and 40 nodes on the other, the cross-partition bandwidth is limited by the smaller side: 如果是遇到不均匀的分区切分情况（例如一边是 88 个节点，另一边是 40 个），跨分区的带宽上限自然就由较小的那一方决定： 40 × 400 GB/s = 16 TB/s With that mental model, let's analyze the DGX H100 reference architecture in more detail. Feel free to zoom in: 带着这样的心智模型，让我们来更为细致地分析一下 DGX H100 的参考架构。你可以随时放大图片进行查看： ![Figure 25: NVIDIA DGX (H100) SuperPod (1024 GPUs) reference architecture](https://www.aleksagordic.com/blog/collective-operations/gpu_topology.png)

Figure 25: NVIDIA DGX (H100) SuperPod (1024 GPUs) reference architecture / 图 25：NVIDIA DGX (H100) SuperPod（1024 GPUs）参考架构

With the GPU cluster topology in mind, we're ready to shift our focus to collective operations. 在了解了 GPU 的集群拓扑之后，我们已经准备好将焦点转移到集合通信操作上了。 --- ## GPU Collectives Within the Node: Rings, Trees, and SHARP / 节点内的 GPU 集合通信：环形、树形算法与 SHARP 技术 In the intra-node setting (i.e. within a single GPU node), the collective algorithms look similar to the TPU case because they both operate on an abstract ring structure, but the underlying topology is different. 在节点内部通信场景下（即在单一的一台 GPU 物理节点内），由于集合通信算法同样也是在抽象的环形结构上运行的，因此它们看起来与在 TPU 上执行的情况很类似。不过，二者底层的硬件连接拓扑是不一样的。 On TPUs, the ring is a physical path through nearest-neighbor ICI links. On GPUs, the ring is a logical ordering chosen over the NVSwitch fabric. 在 TPU 上，所谓的“环”就是依靠相邻之间互相拉线的 ICI 链路走出来的**物理路径**。而在 GPU 节点内，由于 NVSwitch 使得卡与卡之间完全直达，因此“环”只是基于全连接之上刻意挑选出的一种**逻辑排序**。 The NVSwitch fabric gives us effective all-to-all connectivity between GPUs, and the collective algorithm chooses a ring on top of that connectivity pattern. NVSwitch 网络在各 GPU 之间提供了高效且完全的“全对全（All-to-All）”互连连通性，而集合算法则只是顺势在这层极度丰饶的连接模式之上选择性地勾勒出一个环来运作。 Let's break this down: 让我们拆解开来看： ![Figure 26: Constructing a logical ring inside a GPU node](https://www.aleksagordic.com/blog/collective-operations/gpu_ring.png)

Figure 26: Constructing a logical ring inside a GPU node / 图 26：在 GPU 节点内构建逻辑环路

As we already know, All-Reduce can be implemented as Reduce-Scatter followed by All-Gather, so without special hardware support its communication cost is roughly 2× higher than either primitive alone. 由于我们早已知晓，All-Reduce 操作在底层实际上是可以拆分为先执行 Reduce-Scatter 再执行 All-Gather 的组合，因此在没有任何特定底层硬件的黑科技支持下，它不可避免地需要花费单一基础原语（Primitive）近 2 倍的通信成本。 NVIDIA GPU switches (both NVSwitch and IB switches) have an important optimization called SHARP, an in-network reduction compute unit. Normally, a switch just routes data between GPUs. With SHARP, the switch can also perform the reduction operation itself. For example, instead of GPUs exchanging partial sums and reducing them locally, the GPUs send their partial values into the switch, the switch sums them, and the reduced result is sent back. NVIDIA 的 GPU 交换机（无论是在机箱内部署的 NVSwitch 还是在外部负责跨节点的 IB 交换机）都拥有一个被称作 **SHARP（Scalable Hierarchical Aggregation and Reduction Protocol，可扩展分层聚合与规约协议）** 的极为重要的优化机制，它本质上是在交换网络中间直接嵌入了“在线规约计算单元”。一般情况下，交换机的工作仅仅负责在 GPU 与 GPU 之间进行数据的路由和搬运。但在开启了 SHARP 之后，交换机自己就能顺手把规约运算给做了！例如，GPU 不需要再在彼此之间互相频繁交换自身分块计算好的部分求和结果然后再拉回本地进行规约计算了；相反，GPU 会直接将它的部分值抛给上层的交换机，交换机会即时在“半空中”将它们累加，然后将最终汇总好的唯一结果直接发回给下方所有的 GPU。 So instead of spending SM cycles and HBM bandwidth on a largely memory-bound All-Reduce, the network performs the reduction in flight, leaving the GPUs free for useful computation. 如此一来，我们将不再需要消耗 GPU 内宝贵的 SM（流多处理器）算力以及 HBM（高带宽内存）带宽去死磕这种属于典型访存密集型（Memory-bound）的 All-Reduce 操作，转而由网卡/交换机这种“网络层”直接在数据流动的飞行途中执行规约计算。这让 GPU 能够腾出双手去处理真正有价值的核心计算任务。 > 💡 **SHARP's FLOPs/s / SHARP 的计算性能:** > As a reference point, the NVLink 4 NVSwitch (found in H100 nodes) SHARP has 400 GFLOP/s of FP32 reduction throughput [4]. > 作为参考数据，NVLink 4 架构下的 NVSwitch（应用于 H100 节点）其内嵌的 SHARP 计算引擎具备高达 400 GFLOP/s 的 FP32（单精度浮点）规约计算吞吐量 [4]。 SHARP can theoretically make All-Reduce close to 2× faster! In the limit where `N`, the number of GPUs in the node, grows large, the improvement approaches 2×; for 8-GPU nodes, the ideal improvement is closer to 1.75×. 在理论层面上，SHARP 可以让 All-Reduce 操作快上近 2 倍！在节点内 GPU 数量 `N` 趋于极大的极限情况下，性能提升会逐渐逼近 2 倍；而对于一个标准的 8 卡 GPU 节点来说，理论上的理想提升幅度大约在 1.75 倍左右。 So how does SHARP work? 那么，SHARP 究竟是如何运作的呢？ ![Figure 27: SHARP - in-network reduction compute unit](https://www.aleksagordic.com/blog/collective-operations/sharp.png)

Figure 27: SHARP - in-network reduction compute unit / 图 27：SHARP - 网络内规约计算单元

> 💡 **Theory vs practice / 理论与实践的差异:** > Empirically, All-Reduce on GPUs can require very large messages to approach peak BW. In [older NCCL measurements](https://jax-ml.github.io/scaling-book/gpus/#intra-node-collectives) [5] on an 8×H100 node, performance was still ramping even at multi-GB message sizes, and bandwidth dropped noticeably once the message size fell below roughly 100 MB. This is one practical difference from TPUs, which tend to reach near-peak collective BW at much smaller message sizes (~10 MBs). > 根据经验，在 GPU 上执行 All-Reduce 操作通常需要非常大尺寸的报文才能真正逼近其标注的峰值带宽极限。在一份针对 8×H100 节点的 [早期 NCCL 基准测试](https://jax-ml.github.io/scaling-book/gpus/#intra-node-collectives) [5] 中，即便消息大小已经堆到了数个 GB，性能曲线却依然呈现爬坡态势；而且只要消息的尺寸一掉到差不多 100 MB 以下，可利用带宽就会出现明显的断崖式暴跌。这也是它与 TPU 在实际应用中的一个显著区别：TPU 往往在极小的消息尺寸（几十兆 MB 量级）下就能摸到其集合通信的峰值带宽天花板。 > > Even with SHARP enabled for All-Reduce, we should still account for overhead: the reduce + multicast pipeline will not overlap perfectly in practice. In practice speedups from SHARP are **only about 30%** [5][6]! Always run microbenchmarks on your concrete cluster setup. > 即使在启用了 SHARP 进行 All-Reduce 的场景下，我们仍然需要客观评估随之带来的损耗开销（Overhead）：因为“规约求和 + 全局组播下发（Multicast）”组成的流水线在真实世界中根本无法做到完美无缝的重叠交织。在现实实践中，借助 SHARP 带来的性能飞跃**实际上只有差不多 30% 左右** [5][6]！这再次告诉我们：在任何具体的生产集群架构上跑一跑微基准测试（Microbenchmarks）永远是必修课。 > 💡 **Additional SHARP context / 关于 SHARP 的更多背景补充:** > NVLink SHARP is not limited to reductions, it also accelerates the All-Gather phase through hardware multicast. A memory region can be registered as a multicast target for a group of GPUs. After that, a normal CUDA store to the multicast address is replicated by the NVSwitch fabric to every GPU in the group - the kernel itself does not need a special multicast instruction. > 基于 NVLink 架构的 SHARP 并不仅仅只能用于做规约计算（Reductions），它同样能够依靠底层的硬件级组播（Hardware Multicast）机制来大幅加速 All-Gather 阶段。你可以在底层的逻辑配置里，将某一段内存地址范围专门注册登记为某一特定 GPU 群组的“组播目标区”。在这之后，只要向这个映射的组播地址发出任何一个普普通通的 CUDA 存数指令，底层的 NVSwitch 网络结构就会立刻在硬件层面自动把这份数据等比例复制、分发给该群组里的每一块 GPU。对于跑在上层的计算内核（Kernel）代码而言，它甚至不需要知道去调用任何特殊的“组播专属指令”。 > > One inefficiency is that the multicast includes the source GPU as well, even though it already has the data! (wasting 1/8th of BW in an H100 node). > 其中一个稍显尴尬的低效点在于，这套硬件级组播会憨憨地向“数据发射源 GPU 本身”也同样下发一份拷贝数据，尽管它本地早已存有这份数据！（在一个 8 卡 H100 的节点内部，这就硬生生白白浪费掉了 1/8 的带宽。） Inside an NVSwitch node, balanced All-to-All can require roughly 2× less communication time than on a bidirectional 1D torus ring (assuming equal bandwidths), because every GPU can send directly to every destination GPU: 由于每一张 GPU 都能“直连直达”地将数据精准投递至任何一张目标 GPU，在一个基于 NVSwitch 网络互联的节点内部跑一组负载均衡对等的 All-to-All 通信，相较于在传统的“双向一维环面拓扑”上执行（假定总带宽对等的前提下），通信耗时大约可以生生削减 2 倍之多： ![Figure 28: Dense All-to-All over a logical unidirectional ring inside an 8-GPU node](https://www.aleksagordic.com/blog/collective-operations/gpu_ata.png)

Figure 28: Dense All-to-All over a logical unidirectional ring inside an 8-GPU node / 图 28：在 8 卡 GPU 节点内部基于逻辑单向环执行的高密度 All-to-All

> 💡 **All-to-All on NVL72 / NVL72 机柜上的 All-to-All:** > The dense, balanced All-to-All model is a reasonable simplification for an 8-GPU H100 node, but it becomes less representative on systems such as GB200 NVL72. In an MoE inference workload with 72 GPUs and only eight experts selected per token, each token needs to reach only about 8/72≈11% of the GPUs. Communication therefore becomes sparse and ragged rather than a uniform All-to-All: the router first determines each token's destination experts, and the system sends that token only to the GPUs hosting those experts. > 采用这种高密度、负载均衡对等的 All-to-All 抽象模型，对于 8 卡的 H100 节点而言是一种合理的简化建模；但对于像最新的 GB200 NVL72 这样拥有多达 72 张显卡的庞大系统而言，这种模型就显得不那么具有代表性了。在 72 块 GPU 执行 MoE 推理的场景下，假设每个 token 依然只挑选八个匹配专家去路由，那就意味着单个 token 的数据流量只需要覆盖到总 GPU 池子的区区 8/72 ≈ 11% 左右。原本均匀平铺式的通信模式会瞬间破防瓦解为极为稀疏（Sparse）且参差不齐的碎片化请求，而不再是规整的 All-to-All：路由器会首先判定好每一个 token 落在哪几个特定的目标专家头上，最后底层系统只会单纯地顺着网线将对应的数据孤岛定点发送至承载了那些专家的对应 GPU 上。 Before we move to inter-node collective ops, I want to briefly cover the main idea behind tree-based collectives. 在正式转入跨节点集合通信操作之前，我想先简单聊聊“基于树形结构的集合算法”背后的核心思路。 The main idea is to pair GPUs in rounds, doubling the amount of data each GPU has after every round. This gives us `log₂(N)` communication steps instead of `N - 1`. 其核心要义在于：把所有参与的 GPU 进行配对并按“轮（Round）”为单位推进执行，在每一轮通信结束后，保证让每个 GPU 本地所拥有的数据块规模翻一倍。这种做法将原有的 `N - 1` 次串行漫长通信步骤大幅浓缩至短短的 `log₂(N)` 步。 Here is tree-based All-Gather, implemented as recursive doubling: 下面展示的是树形 All-Gather 的运转过程，它底层采用了“递归倍增（Recursive Doubling）”的逻辑来实现： ![Figure 29: Tree All-Gather](https://www.aleksagordic.com/blog/collective-operations/ag_tree.png)

Figure 29: Tree All-Gather / 图 29：基于树的 All-Gather

A few notes on tree vs. ring algorithms. 关于树形算法和环形算法之间的权衡，需要稍微说明几点。 Both ring and tree algorithms have step dependencies, but rings are usually easier to pipeline. In a ring, many chunks can be streamed continuously around the ring, keeping links busy. Recursive doubling can have lower latency complexity, but for large messages it may achieve lower effective bandwidth than ring because it is less pipeline-friendly. 环形与树形算法本质上都绕不开所谓的步骤依赖限制（Step Dependencies），但是，环形结构往往更加容易实现“流水线化（Pipeline）”。在一个稳定的环形回路里，大量的数据碎块能源源不断地首尾相连进行接力传输，这样能够让底层链路时时刻刻处于火力全开的繁忙装填状态。尽管基于递归倍增逻辑的树形算法理论上拥有极低的延迟时间复杂度，可一旦真的遇上超级“大块头”的重负载大消息传输，因为其对流水线并行的适配极其差劲，最终导致实际能跑出来的真实有效带宽反而被环形算法吊打。 In the ideal bandwidth model, the byte cost is the same (as you can see in the figure above), but tree-style algorithms reduce the number of communication rounds from `N - 1` to `log₂(N)`. In practice, rings often achieve higher effective bandwidth on large tensors, so libraries like NCCL choose between ring, tree, and hybrid algorithms based on message size and topology. 放在完美的理想带宽理论模型下来算，两者所需要搬运和付出的总字节代价是完全对等的（这一点你可以从上面的配图中清晰看出来），但采用树形算法的好处就在于它确确实实将总体经历的通信交互轮数从 `N - 1` 猛砍到了 `log₂(N)`。但在真实的工业界实践里，当面临超大体量的张量时，环状网络经常性能够吃满更高的有效带宽；正是基于此，像英伟达底层的 NCCL 这一类核心通信库，都会极其“审时度势”地依据下发消息载荷的大小、以及底层物理拓扑结构，在这三者（环状、树形以及组合型的混合算法）之间进行动态和最优化的抉择调用。 For completeness, here is Reduce-Scatter via recursive halving: 为了确保逻辑知识图谱的闭环，这是通过“递归折半（Recursive Halving）”原理实现的树形 Reduce-Scatter： ![Figure 30: Tree Reduce-Scatter](https://www.aleksagordic.com/blog/collective-operations/rs_tree.png)

Figure 30: Tree Reduce-Scatter / 图 30：基于树的 Reduce-Scatter

Next, let's see what changes when we move outside a single GPU node and into the inter-node setting. 接下来，当我们冲破单一物理机壳的禁锢跨入浩瀚的**多节点联合通信层**时，让我们看看一切又将发生怎样的变局。 --- ## GPU Collectives Across Nodes: Hierarchical Algorithms over InfiniBand / 跨节点的 GPU 集合通信：基于 InfiniBand 的分层通信算法 The low-level details of algorithms become more involved due to pipelining over multiple hierarchy levels (node-level, SU-level, spine-level). 因为牵扯到需要在极为错综复杂的多重物理层级（节点内级、SU 汇聚层级、Spine 骨干层级）之间去做重度流水线交织协同（Pipelining），其最底层算法的运转脉络与微观细节就不可避免地开始变得极为烧脑和繁琐起来。 A good mental model is to imagine running one inter-node ring over every node in the cluster (this we can do thanks to the non-oversubscribed/full fat tree). 这里提供一个绝佳且便于直觉理解的宏观心智模型：你可以在脑海里想象有一张横跨覆盖整个庞大集群、跨越节点边界的宏伟“大环线网络”（而这恰好正是得益于它底层是一张不打折扣的无收敛完全胖树架构底座才得以实现）。 For large messages, a good first-order approximation for All-Gather or Reduce-Scatter is therefore (where `D` is the tensor size in bytes): 针对于超大规模的沉重消息负载包而言，关于 All-Gather 抑或是 Reduce-Scatter，一个非常靠谱的一阶数学近似估算公式应当如下（这里的 `D` 代表需要传输的张量总体积，单位为字节 Bytes）： `T_total` ≈ `D` / `BW_node` = `D` / `400e9` A slightly more accurate model also accounts for the intra-node stage. In hierarchical collectives, we usually have both scale-out traffic over IB and local traffic over NVLink/NVSwitch, and these stages can often be pipelined. So the total time is closer to the slower of the two terms: 为了再稍微精准严谨一些，我们的测算模型还可以把“本地节点内部搬运”的这一环给加上。在复杂庞大的分层集合通信（Hierarchical Collectives）场景之下，我们一般不仅有沿着 IB 专属通道跑来跑去的横向扩散数据流，同样还有挤在节点本地 NVLink/NVSwitch 专属通道里的局部内循环数据流；而且往往这两块流程大多时候是能够进行并行互锁流水线作业的。因此，最为贴近真实总耗时的数值，应当是取决于这二者之中“拖后腿”比较慢的那一端： `T_total` ≈ max(`D`/`BW_gpu`, `D`/`BW_node`) Let's start with the first cross-node collective - All-Gather: 让我们先从跨节点打通的第一个老伙计——All-Gather 开始说起： ![Figure 31: Hierarchical All-Gather over the scalable-unit](https://www.aleksagordic.com/blog/collective-operations/ag_hierarchy.png)

Figure 31: Hierarchical All-Gather over the scalable-unit / 图 31：在可扩展单元级别上的分层 All-Gather

> 💡 **Why no spine switch? / 为什么不画出 Spine（脊）交换机？** > For these examples, I'm only showing communication within a single SU. Adding the spine fabric doesn't qualitatively change the algorithm, we'd still run the same kind of hierarchical collective (just across one more level of the topology). It does however complicate visualizations, so I'll skip doing spine-level collectives. > 在本节的举例中，我刻意地只局限在一套单独的 SU（可扩展单元簇）的内循环体系中做演示。即使生硬地把最核心总控的 Spine（脊交换机）网络部分也画进去，也并不会让这套核心算法在本质原理上发生多大的质变颠覆。我们依旧跑着如出一辙的“分层套娃式”集合通信协议（唯一的变化仅仅只是多包上了一层物理拓扑网而已）。但它却会让整张解说图变得犹如乱麻一般令人窒息难以阅览，所以在此我便彻底跳过 Spine 骨干层级别的演示了。 > 💡 **Note on rail optimization / 关于网络轨道（Rail）优化的说明:** > A node's aggregate inter-node bandwidth is not always fully fungible across all GPUs (like in the simple model I shared above). GPU clusters are often organized into parallel network rails, with particular GPUs having preferred paths into the network. Reaching the full node-level bandwidth therefore depends on rail-aware rank placement and balanced traffic across those paths. > 现实中，一台物理节点向外输出的总体聚合互联带宽能力，其实并不完全等同于其内部所有 GPU 兄弟能够毫无壁垒、“如胶似漆”般地完美无脑平分共享（不像我在上文分享的那个极致简化的理想国模型）。最顶尖的超算 GPU 集群一般被高度精心划分并被编织成了若干并列的“网络平移轨道（Parallel Network Rails）”，某些个特定的“命定” GPU 在向大网输送或者捞取流量时，会被强行圈定一条优先走特定网卡的专属通道。所以，如果想要将一台几百万造价物理服务器的峰值满血带宽压榨到极致，极端依赖于排兵布阵般具有轨道感知（Rail-aware）能力的秩节点布局，以及在这些海量高压通道之间实施极其苛刻的负载均衡流量调度策略。 Next up, let's look at hierarchical All-Reduce (I'll skip hierarchical Reduce-Scatter because its structure closely mirrors hierarchical All-Gather): 接下来登场的是分层版的 All-Reduce（因为原理极为雷同，我就不在分层版的 Reduce-Scatter 身上浪费笔墨了，毕竟它的组织架构基本可以被看作是 All-Gather 的“克隆兄弟镜像”）： ![Figure 32: Hierarchical All-Reduce over the scalable-unit](https://www.aleksagordic.com/blog/collective-operations/ar_hierarchy.png)

Figure 32: Hierarchical All-Reduce over the scalable-unit / 图 32：在可扩展单元级别上的分层 All-Reduce

One subtlety: the tensor being All-Reduced may itself be sharded over another parallelism axis. 然而有一个极其微妙的细节暗坑必须要提防：当前正面临着被强行实施跨网络 All-Reduce 规约的倒霉张量，其自身极有可能在出厂时刻早已被另一种毫不相关的并行策略轴线（Parallelism Axis）狠狠地切成了粉末残躯。 For example, in Megatron-style training (tensor/model parallelism), a weight matrix may be sharded over the tensor-parallel axis `Y`, while its gradients are reduced over the data-parallel axis `X`. 譬如，在臭名昭著的 Megatron 流派暴力训练法（基于张量/模型并行切割）中，一个本应完整无缺的权重矩阵早就已经被在张量切片坐标轴 `Y` 上给剁碎了，而随后针对其产生的那一堆衍生品梯度，又要被横穿在广袤无垠的数据并行轴 `X` 的各个远端副本去跨时空进行艰难的重逢规约。 The All-Reduce is not performed across the tensor-parallel ranks. Those ranks own different shards, so there is nothing elementwise to reduce between them. 请千万不要将 All-Reduce 张冠李戴地强行跨域在那些“只用来玩张量拼图切割把戏的 Rank 小圈子”内部强搞。这些 Rank 各自把持着张量截然不同的碎片残肢，它们彼此之间根本不存在什么能在逐个元素级别去相加求和规约的对象与意义。 Instead, for each fixed tensor-parallel shard, we AllReduce across the corresponding data-parallel replicas. 恰恰相反，对于任何一片被固定死、用于张量并行的切割残片，我们只会在那些分布在全网、扮演着它“平行世界克隆体”的对应数据并行副本组之间去开展 All-Reduce 通信交互。 Let's work through a concrete example: 光说不练假把式，我们直接上一张具象化详图来跑一遍整个心智流程吧： ![Figure 33: Hierarchical sharded All-Reduce over the scalable-unit](https://www.aleksagordic.com/blog/collective-operations/ar_shard_hierarchy.png)

Figure 33: Hierarchical sharded All-Reduce over the scalable-unit / 图 33：在可扩展单元级别上的分层式分片 All-Reduce

Finally, let's look at hierarchical All-to-All over a SU. 压台大戏：让我们俯瞰宏伟可怖的 SU 跨节点分层级别之下的终极绝招 All-to-All 究竟是一副什么模样。 Unlike All-Reduce, All-to-All cannot be compressed by doing a local reduction first (each chunk has a specific destination): 与 All-Reduce 那套讨巧的偷懒把戏截然不同的是，面对 All-to-All，你彻底丧失了能够依赖提前在本地先将大家就地揉搓压瘪（局部规约）来强行缩水总体量的物理合法性（因为人家发出去的每一块小破包裹，上面都已经死死定下了唯一的特快收货地址专人专送，容不得掺假）： ![Figure 34: Hierarchical All-to-All over the scalable-unit](https://www.aleksagordic.com/blog/collective-operations/ata_hierarchy.png)

Figure 34: Hierarchical All-to-All over the scalable-unit / 图 34：在可扩展单元级别上的分层 All-to-All

As we did in the TPU section, let's collect the GPU cost models in one place: 与我们在上一章节对 TPU 收尾时的做法如出一辙，在此，我们将针对 GPU 全链路集合操作成本消耗的一阶心智模型汇总在了一张清单中： ![Figure 35: Summary of GPU collective communication costs for intra-node and inter-node collectives](https://www.aleksagordic.com/blog/collective-operations/totaltime_gpu.png)

Figure 35: Summary of GPU collective communication costs for intra-node and inter-node collectives / 图 35：面向节点内及跨节点全场景的 GPU 通信核心成本清单全景总结

--- ## Epilogue / 后记 Phew, that was a long one! 呼，这可真是长篇大论的一期！ What originally started as "let me maybe just make four figures covering All-Gather, Reduce-Scatter, All-Reduce, and All-to-All so I can understand them better, it shouldn't take more than a day, right, right" somehow turned into this. 一开始我天真地以为：“也许我只需要画 4 张图把 All-Gather、Reduce-Scatter、All-Reduce、All-to-All 覆盖一下，从而帮自己更好地理解它们就好，这绝对花不了超过一天的时间的，对吧，对吧”，结果怎么就变成了现在这副模样。 Along the way, I realized that the collective algorithms only really make sense once you understand the underlying hardware topology. TPUs were a bit easier to reason about, but I couldn't skip GPUs, I love them too much. Rings are cool, but I also wanted to understand tree algorithms. But also SHARP, and fat trees, and hierarchical collectives. :') 在这个过程中，我逐渐意识到，只有当你彻底弄懂了底层冷冰冰的硬件拓扑连接之后，那些飘在半空中的所谓集合通信算法才真正具有现实落地的逻辑实感。TPU 推理起来还相对容易些，但我绝对不能略过 GPU 不谈，毕竟我对它们爱得深沉。环形算法确实酷炫，但我同时也按捺不住想去弄懂树形算法的运转逻辑。可是还有 SHARP、胖树架构，还有复杂的层级集合通信…… :') So the scope slowly expanded, and little by little, this blog post came to fruition. Just a side-quest. 于是，探讨的范围就像滚雪球一样越滚越大，一点一滴累积起来，这篇如同史诗长卷般的长博客终于瓜熟蒂落。就当是一次副本支线任务吧。 Hope you liked it! :) 希望你能喜欢！:) > 💡 **Get in touch / 取得联系:** > If you spot any errors in the post, please DM me - feel free to drop me a message on [X](https://x.com/gordic_aleksa) or [LinkedIn](https://www.linkedin.com/in/aleksagordic/) or via [anon feedback](https://docs.google.com/forms/d/1z1fEirrN2xtGxAsJvptpM7yV4ByT5SF25S-XiMPrXNA/edit). > 如果你发现了文章中的任何错误，请私信告诉我——欢迎在 [X](https://x.com/gordic_aleksa) 或 [LinkedIn](https://www.linkedin.com/in/aleksagordic/) 上给我留言，或者通过 [匿名反馈表单](https://docs.google.com/forms/d/1z1fEirrN2xtGxAsJvptpM7yV4ByT5SF25S-XiMPrXNA/edit) 提出你的建议。 --- ## Acknowledgements / 致谢 Thanks to my friends [Aroun Demeure](https://github.com/ademeure) (ex GPU & AI at Magic, and ex-GPU architect at Apple and Imagination), [Axel Feldmann](https://x.com/axel_s_feldmann) (ML perf engineer at Jane Street), and [Pranjal Shankhdhar](https://x.com/pranjalssh) (prev. GPU kernel engineer at xAI) for reading pre-release version of this blog post and providing feedback! 非常感谢我的朋友 [Aroun Demeure](https://github.com/ademeure)（曾在 Magic 负责 GPU 与 AI 工作，前苹果和 Imagination 的 GPU 架构师）、[Axel Feldmann](https://x.com/axel_s_feldmann)（简街的机器学习性能工程师）以及 [Pranjal Shankhdhar](https://x.com/pranjalssh)（前 xAI 的 GPU 内核工程师），感谢他们审阅了本文的抢先发行版并提出了宝贵的反馈意见！ Arun broadened the GPU discussion beyond balanced All-to-All, highlighting NVL72’s any-to-any topology, sparse and imbalanced MoE routing, and NVLink SHARP’s memory-style multicast/reduction model and practical trade-offs. Arun 将有关 GPU 讨论的视野拓展到了完美平衡的 All-to-All 之外，并重点强调了最新的 NVL72 架构任意点对点的拓扑特征、极其稀疏且极度不平衡的 MoE 路由通信困局，以及剖析了 NVLink SHARP 类内存存储式的组播规约模型与它们在真实落地实践中必须做出的严苛性能权衡。 Axel refined the GPU section by highlighting rail optimization and SHARP’s SM/HBM offload benefits, then grounding the performance discussion with H100 measurements showing roughly 1.3× practical SHARP speedups and near-saturated collective bandwidth at around 1 GB. Axel 为庞杂的 GPU 章节带来了极大的打磨升华，深入剖析了极度烧脑的通道轨道调度优化，以及彻底盘点了 SHARP 将庞大的规约计算从前线 SM 运算单元与 HBM 存储显存的无底洞卸载出去的革命性助推优势。随后更是用极为扎实的 H100 第一线实战跑分性能参数进行了逻辑定调，即在严酷生产实践中 SHARP 真实平均测速提效基本恒定在 1.3 倍左右游走；同时给出了诸如只有当消息体量被堆叠并塞满到高达逼近 1 个 GB 的门槛时，才有可能令硬件底层的集合交互聚合带宽呈现全线爆表逼近打满的极限常态。 Pranjal independently reinforced the points on rail optimization and the gap between SHARP’s theoretical 2× speedup and the roughly 30% improvement typically observed in practice. Pranjal 则基于自己独立的认知和实践，坚定不移地印证并强化了有关于网络通道平移排兵布阵的心血经验之谈，并再次严正戳破了 SHARP 理论上看似完美的 2 倍狂暴加速幻梦与其在充斥着冷血现实与无数不可控损耗开销制约下基本在 30% 上下尴尬徘徊的性能天花板落差。 --- ## References / 参考文献 1. "TPU 8t and TPU 8i technical deep dive", [https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive) 2. "TPU topology visualizer", [https://tpu-visualizer.uc.r.appspot.com/](https://tpu-visualizer.uc.r.appspot.com/) 3. "How to Scale Your Model", [https://jax-ml.github.io/scaling-book/](https://jax-ml.github.io/scaling-book/) 4. "NVSwitch Hot Chips 2022", [https://hc34.hotchips.org/assets/program/conference/day2/Network%20and%20Switches/NVSwitch%20HotChips%202022%20r5.pdf](https://hc34.hotchips.org/assets/program/conference/day2/Network%20and%20Switches/NVSwitch%20HotChips%202022%20r5.pdf) 5. "How to Scale Your Model: GPUs / intra-node collectives", [https://jax-ml.github.io/scaling-book/gpus/#intra-node-collectives](https://jax-ml.github.io/scaling-book/gpus/#intra-node-collectives) 6. Axel independently re-ran the benchmark using NCCL 2.29.7 on a 32-GPU H100 InfiniBand setup, corroborating the reported result: he likewise observed only a ~1.3× speedup on a 4 GB All-Reduce. Pranjal also reported the same result. Arun said this is more of a NCCL problem than a SHARP problem. Axel 独立在一个由 32 块 H100 组成、搭载 InfiniBand 网络的集群之上，使用 NCCL 2.29.7 库重新执行了跑分测试。其得出的实测结论再次高度确凿地佐证了早前公开抛出的那份成绩：他在一个体积高达 4 GB 的巨型张量 All-Reduce 规约测试用例下，仍然同样只勉强观察到大约 1.3 倍左右的综合加速效能。Pranjal 也独立反馈了如出一辙甚至更为严苛的跑分实情。Arun 则敏锐点出：与其把这一口大黑锅结结实实地全部扣在硬件底层的 SHARP 身上，这更像是一个因为由于当下 NCCL 底层系统软件工程调度栈实在跟不上而导致严重拖垮拖慢了全局硬件大盘的尴尬困局。