Stripped Down CXL Scales Memory Wall

//php echo do_shortcode('[responsivevoice_button voice="US English Male" buttontext="Listen to Post"]') ?> The first member of Credo Technology Group’s OmniConnect family aims to boost memory bandwidth and memory density to optimize computing efficiency of accelerators and XPUs for AI inference workloads.Credo’s Weaver is a memory fanout gearbox that overcomes the memory bottlenecks in AI inference workloads as scalability, bandwidth and efficiency demands in data centers continue to rise, Don Barnetson, senior product VP at Credo, told EE Times in a briefing.“AI inference is a new set of challenges for the industry.” He said memory quantity and throughput has increasingly become the limiting factor for AI inference workloads—not compute power. “The challenge that inference workloads have is that they’re different from training workloads in that they are very memory intensive. You would ideally like to have the entirety of your model, however big it is, surrounding one compute engine.” LPDRR5 and GDDR memories are constrained by their bandwidth, density and power consumption, Barnetson said, and high-bandwidth memory (HBM) is often too pricey for many applications and not readily available, which was what drove the development of Weaver.The memory fanout gearbox combines advanced 112G very short reach (VSR) SerDes with Credo’s proprietary design to boost I/O density by up to 10×. Barnetson said the company’s unconventional architecture enables up to 6.4TB of memory and 16TB/s bandwidth using LPDDR5X.Barnetson said Weaver address the pressure placed on memory by inference as entire codebases are uploaded to an engine, citing Sora 2 as an example of the current limitations. “Sora 2 can only do 15- or 20-second video clips because it runs out of memory after that.”He says the way this problem has been solved in the past is by placing LPDDR on a giant substrate surrounding an inference IC. “You’re limited by how much memory you can fit on this substrate.”With a maximum of 16 memory stacks, capacity tops out at 256GB of memory and 1.3 terabytes of bandwidth. “You end up surrounding the entire beachfront of your chip with memory interfaces,” Barnetson said.Weaver is Credo’s lower cost solution that moves memory off the substrate by using a SerDes rather than a typical wide parallel interface to move data, he added.Credo has developed a SerDes with a reach of 250 mm with a high beachfront density of about 2 terabits per millimeter, Barnetson said. It is also very low power at about 1 picojoule per bit.The next step was to add a data link layer, which leverages the CXL protocol, he said. “We’ve gutted CXL, and we’ve put it on a much lower power SerDes. We’ve taken CXL to its bare bones.”Weaver keeps the CXL FLIT structure, which has a low overhead, but drops others like speed hopping. Barnetson said Credo wanted to create a highly efficient AXI over SerDes mapping while preserving flexibility, anything can be done locally over OmniConnect.“We designed this so it would be a really short latency, about 40 nanoseconds round trip.”He said anything that can be done with a DDR controller and ASIC can be done through the Weaver chip, and the fan-out gearbox allows for attachment of the SerDes lanes to the chip, which has the memory interfaces.“It gets you physical fan out, so that you can mount memory at server scale not just inside of a package.”Barnetson said this approach allows memory placement as far away as 10 inches from the chip. “You can increase the memory density by a factor of about 30.”He added that one of the goals of Weaver and the OmniConnect family overall was to make it future proof since it is difficult to predict the requirements from AI several years out. “You just put a whole bunch of OmniConnect interface on the outside, and then you can decide how much memory and how much network you want to put down in three years when you’re building your boards.”Credo recently acquired Hyperlume, a developer of microLED-based optical interconnect technology for chip-to-chip communication. The company’s technology uses ultra-fast microLEDs and ultra-low power circuitry to overcome the energy and bandwidth bottlenecks inherent in traditional electronic interconnects.Barnetson said Credo will be integrating Hyperlume’s microLED technology into its portfolio to further expand options for scaling up AI networks. “We’ll be supporting a micro-LED near-port optic chip, so you could choose some of those lanes for micro-LED communication or standard scale-up or scale-out communication as well.”