[News] NVIDIA-Linked Groq Reportedly Eyes Shift From SRAM to 3D-Stacked DRAM for Next-Gen AI Chips

As AI workloads demand greater memory capacity, chipmakers are exploring alternatives to traditional memory architectures. According to ZDNet, industry sources say Groq, the AI chip company that licensed its inference technology to NVIDIA, is considering 3D-stacked DRAM for its next-generation chips. The potential shift away from its SRAM-centric design, which offers high speeds but limited capacity, could significantly increase on-chip memory capacity.

The report indicates that Groq is exploring 3DS (3D Stacked Memory), which uses through-silicon vias (TSVs) to vertically connect DRAM dies. While both 3DS and high-bandwidth memory (HBM) rely on TSVs, their priorities differ: HBM focuses on high data transfer speeds, while 3DS aims to increase memory capacity while retaining conventional DRAM interfaces.

NVIDIA’s Groq 3 LPX is a rack-scale AI inference accelerator designed for low-latency, large-context workloads on its Vera Rubin platform. According to NVIDIA, each Groq 3 LPU integrates 500MB of SRAM with 150TB/s of on-chip memory bandwidth, while a full LPX rack combines 256 LPUs to deliver 128GB of SRAM and 40PB/s of aggregate SRAM bandwidth.

However, SRAM capacity is struggling to keep pace with the growth of AI models. According to 36Kr, citing d-Matrix CEO Sid Sheth, SRAM capacity increases by only around 10–20% every two years, while AI model sizes could double every six to nine months. This widening gap makes it increasingly difficult for SRAM alone to meet growing demands for memory capacity and bandwidth.

Notably, NVIDIA’s existing Groq 3 LPX platform already combines SRAM with conventional DRAM. According to the company, each rack provides 12TB of DDR5 memory alongside 128GB of SRAM.

Samsung Produces Groq 3 as Next-Gen Foundry Competition Takes Shape

On the manufacturing side, Samsung Foundry produces NVIDIA’s Groq 3 LPU using its 4nm process at its Pyeongtaek facility, according to Seoul Economic Daily. Samsung executives said in June that the company was discussing foundry cooperation with NVIDIA on the next-generation Groq LPU. Meanwhile, TSMC Chairman C.C. Wei said during the company’s April earnings call that TSMC was working with customers on next-generation LPU development, according to Economic Daily News.

Cerebras Also Pursues 3D-Stacked DRAM for AI Inference

Groq is not alone in exploring higher-capacity memory architectures for AI inference. Cerebras, another AI chipmaker that relies heavily on SRAM, is also pursuing 3D-stacked DRAM. According to the company, its future CS-6 system will integrate stacked DRAM with wafer-scale SRAM and compute to significantly expand memory capacity while preserving fast data access.

Read more

• [News] NVIDIA Reportedly Pushes TSMC 3nm NVHBM Base Dies for Samsung, SK hynix, Micron, Uniting Foundry Rivals
• [News] NVIDIA’s $20B Groq Deal Spotlights SRAM Shift—MediaTek NPU Already On Board

(Photo credit: NVIDIA)