<p>Prioritizing networking to boost computing power, and using storage in place of computation: Huawei's computing modules unlock a new paradigm for releasing computing power.</p>

As large models and Agents move toward scaled deployment, the focus of AI infrastructure is gradually shifting.

Over the past few years, industry attention has been almost entirely on compute power. However, as compute clusters scale from thousands to tens of thousands, and even hundreds of thousands of cards, hidden bottlenecks begin to emerge.

In training scenarios, model parameters are advancing from hundreds of billions to tens of trillions. Communication latency and network congestion severely drag down effective compute utilization. In inference scenarios, context sequences expand to millions, with rapid growth in KV Cache size. Memory and storage bandwidth, capacity, and hit rates directly impact inference throughput and Time-To-First-Token (TTFT).

The industry needs more than just stronger CPUs or GPUs/npus; networking, storage, and DPUs must upgrade synchronously. Meeting performance demands across diverse scenarios while adapting to customers' existing tech stacks has become a critical new challenge.

Huawei's computing module solution lies in providing highly reliable hardware products and full-link technical support, collaborating with partners and developers to build a robust industry ecosystem.

On one hand, openness and compatibility activate ecosystem vitality. Huawei's computing modules fully support ARM/x86 architectures and are compatible with mainstream operating systems like openEuler and Ubuntu, continuously breaking through technical adaptation boundaries to promote co-construction and shared prosperity within the ecosystem.

On the other hand, we adhere to independent innovation. Key performance indicators for products such as network interface cards (NICs), SSDs, DPUs, and RAID cards lead the industry, forming a complete matrix of storage and networking modules that cover both general-purpose and intelligent computing scenarios.

Notably, the AI NIC and KVU (KV Cache Unit) heavily launched at Huawei Connect 2026 specifically target network communication and KV Cache bottlenecks.

Strengthening Compute via Networking: Building a "Highway" for Compute Clusters

In many people's perception, networking is seen merely as supporting infrastructure for compute clusters, where matching bandwidth numbers suffices. By 2026, this traditional logic has been completely overturned.

As frontier large models scale to tens of trillions of parameters and training sequences enter the millions, nodes require frequent synchronization of massive gradients and intermediate parameters. Relevant research indicates that in large-scale distributed training, network communication time accounts for 30-50% of the total duration.

Simultaneously, the proliferation of Agents, long-context processing, and multimodal applications has qualitatively changed network traffic structures: collective communication in inference tasks now accounts for over 30%. Due to long-tail effects, even minor fluctuations in network communication can be perceived by users as "severe lag."

Faced with increasingly prominent network communication bottlenecks, Huawei's domestically first-ever 800G AI NIC, the SP560 series, builds a "highway" for data flow across tens of thousands of compute nodes, using hardcore technology to directly answer three essential questions crucial to the industry.

Question 1: Is the bandwidth large enough?

The currently recognized compute-to-connectivity ratio remains between 2 and 2.5. NIC bandwidth requirements have risen to 800 Gbps, expected to reach 1.6 Tbps by 2028.

Designed for super-node Scale-out architectures, the SP560 series supports large-scale RoCE networking up to 100,000 nodes. It innovatively adopts a dual-bus architecture combining Lingqu and PCIe, with maximum bus bandwidth reaching 1.6 Tbps, enabling customers to fully unleash network performance.

Question 2: Is the protocol flexible enough?

AI networks exhibit distinct load characteristics: numerous short flows coexist with few long flows, potentially causing severe link congestion and overall network utilization below 40%.

To meet top-tier customers' needs for custom protocols, the SP560 series NICs, based on a programmable NP architecture, allow adaptive extension of RDMA, TCP, and various advanced congestion control algorithms. Hardware-level implementation enables multi-path distribution, out-of-order reassembly, and selective retransmission, boosting cluster-wide network link utilization to over 85%.

Question 3: Is communication efficient enough?

In traditional server architectures, data transfer between NICs and compute relies primarily on CPU-issued control commands, resulting in significant bus bandwidth loss and interaction latency.

The SP560 series NICs optimize in two directions:

First, GPU/NPU direct drive and communication task orchestration offloading. This establishes a direct path for compute cards to control and manage NICs, bypassing secondary CPU relay. It also supports parsing communication matrices and issuing packets directly at the granularity of collective communication groups, significantly reducing communication task scheduling overhead.

Second, a control-data separation architecture isolates control-plane traffic from data-plane traffic. Control-plane traffic is sent to the CPU, while massive data-plane business traffic passes directly through the Lingqu high-speed bus to GPU/NPU memory, achieving CPU-free data copying and reducing NPU communication latency.

If compute is the "heart" of a cluster, networking is the "aorta." Compute determines the ceiling, while networking defines the boundary. Only by truly "stringing together" every card can an AI cluster evolve from being "stacked" to "running," thereby supporting larger, smarter, and more numerous intelligent computing clusters.

Replacing Compute with Storage: Finding a New Home for "Bursting" KV Cache

Having resolved "inter-card transport capacity," network obstacles for large model training and inference are cleared. To further improve compute utilization, the "storage wall" and "memory wall" must be broken.

Large model inference is autoregressive; each new Token generated requires moving historical KV Cache in and out of VRAM. In million-token context scenarios, KV Cache VRAM overhead often exceeds 90% of the total.

Keeping all KV Cache in VRAM causes hardware procurement costs and energy consumption to skyrocket, turning commercialization into a mirage. Conversely, pushing KV Cache down to standard SSDs leads to queue congestion under massive concurrent reads, causing inference TTFT to surge.

This shift in compute bottlenecks marks a reconstruction of storage's role: previously just a "data warehouse" for storing training data and model weights, storage now actively participates in the inference process. Through KV Cache tiering, caching, and reuse, it helps release valuable compute resources.

Addressing the dilemma where traditional storage media cannot balance capacity, latency, and bandwidth, Huawei Computing Module introduced the KVU (KV Cache Unit), designed specifically for large model inference.

Unlike passive hard drives, the KVU deeply integrates compute cores within the module, enabling local data preprocessing. Leveraging Lingqu interconnects, a single KVU card supports up to 64TB of storage, achieves sequential read bandwidths of 40 GB/s, sequential write bandwidths of 20 GB/s, and random read IOPS exceeding 5 million.

Capitalizing on the KVU's outstanding performance, Huawei Computing Module addressed the pain points of "wanting million-token long contexts but hitting limits due to insufficient VRAM, high costs, low hit rates, and unstable latency" by creating a four-tier KV Cache storage system balancing performance and cost.

L1 Layer (HBM): Offers TB/s-level ultra-high bandwidth, used for resident and processing the most active, high-frequency hot KV data currently participating in computation;

L2 Layer (DDR Memory Pool): Provides nanosecond-level low latency, handling transitional hot KV data under small-to-medium scale concurrency, acting as an elastic buffer layer between HBM and lower-tier media;

L2.5 Layer (KVU Computing Module): Serves as the core anchor connecting VRAM and traditional flash memory, efficiently handling large-capacity semi-hot and warm KV Cache with ultra-high bandwidth and microsecond-level read/write latency.

L3 Layer (General-Purpose Large-Capacity Local SSD): Responsible for the scalable, long-term retention of cold KV data and historical state logs accessed infrequently.

Relevant test results show: leveraging the KVU module's built-in compute power, extensive prefix lookups, cache validation, and format conversion operations are completed in parallel internally. In actual large model inference workloads, the Prefix KV-Cache tiered caching management mechanism improved NPU utilization by 30% and reduced TTFT by 50%.

To date, the KVU solution has been tested and validated across multiple long-text Agent, multimodal inference, and enterprise-level intelligent computing center projects, significantly reducing VRAM occupancy and truly realizing "deploying ultra-high-spec long-context AI workloads without stacking exorbitantly priced HBM."

Conclusion

Entering the Agentic AI era, a profound transformation is underway.

With ever-larger model parameter scales and longer Agent task chains, AI infrastructure must advance deeper. It must not only solve the "availability" of core compute but also address the "strength" of key modules.

The era of simply stacking compute power is over. Technological innovation and ecosystem openness are the decisive factors in the intelligent computing age. Breaking industry bottlenecks through independent innovation, empowering industry partners via open compatibility, and meeting differentiated needs with diverse products are both the engineering 答卷 delivered by Huawei's computing modules and the inevitable path for AI to transition from parameter competitions to scaled deployment.