Why AI Inference Is Quietly Rewriting the Memory Market?

“The memory system of AIs is going to cause the storage system to be completely revolutionized.”
– Nvidia founder and CEO Jensen Huang

At GTC Taipei in June 2026, Nvidia founder and CEO Jensen Huang pointed to the memory system as one of the hardest parts in AI infrastructure. This challenge encompasses managing KV caching for the agent’s working memory, as well as retrieving structured and unstructured data and establishing data ontology.

Figure 1. Nvidia GTC Taipei 2026 showing the architecture of an AI agent, defined as LLM plus Harness.

Source: Nvidia

To address the surging KV cache storage demands of the AI inference era, Nvidia introduced the CMX Context Memory Storage Platform in January 2026, managed by the BlueField-4 DPU, which adds a pod‑level context tier between local SSD and shared storage.

Meanwhile, the rise of Agentic AI is reshaping CPU architecture. Jensen noted that agents live in a world of nanoseconds, where every moment of waiting prevents them from advancing to the next step, making ultra-low latency the primary requirement. With Nvidia and Arm both launching CPU rack solutions purpose-built for agents, the industry is shifting from throughput-oriented to latency-oriented architectures, opening up an incremental market for CPU RAM.

Test-time Scaling “Thinking”: >5X Tokens Per Year

According to Nvidia’s public data, the average output token count per question has surged at a rate exceeding 5x per year since the second half of 2024, reaching approximately 30,000 to 40,000 tokens, indicating that the industry has entered the Test-time Scaling “Thinking” stage of Nvidia’s Three Scaling Laws. This explosion in per-question token output translates directly into greater demands on memory and compute resources.

Figure 2. Nvidia scatter plot showing average output tokens per question from 2023 to 2025. Reasoning models surge to 10,000 to 30,000 tokens driven by Test-Time Scaling “Thinking,” reflecting growth exceeding 5x per year.

Source: Nvidia

In the AI inference era, hardware requirements for AI chips and overall systems differ fundamentally from those of AI training. Inference places three key demands on hardware:

1. higher queries per second (QPS)
2. longer context windows
3. more inference steps and agentic AI loops

Each of these drives structural changes in memory demand. We will examine this across three dimensions: model weights, KV cache, and agentic AI.

AI Servers Absorbing LPDRAM Capacity, Signaling Tight Supply as the New Norm

NVIDIA is trimming memory capacity on next-gen platforms to keep shipments on track amid an LPDRAM crunch. AI servers are closing in on smartphones as the top consumer, and supply is only getting tighter.

Stay Updated

Model weights

Model weights are the numerical parameters stored within an AI model. When a model is loaded, these weights occupy a fixed portion of memory, which is a static allocation. The more parameters a model has, the more memory its weights consume. The formula for calculating model weight memory is:

KV Cache

KV cache stores the key-value vectors generated during the inference prefill stage, avoiding redundant computation during the decode stage, which is a dynamic allocation. As conversation length and batch size grow in inference workloads, KV cache memory consumption grows accordingly.

Figure 3. KV cache application in the LLM inference workflow, showing the prefill and decode stages.

The formula for calculating the total size of KV cache is:

SSD POD Demand Driven by KV Cache Offloading

As KV cache memory footprint expands dramatically with conversation length and batch size, its management and placement have become critical in AI inference applications. In long-context, high-batch workloads, when the GPU’s HBM capacity is insufficient, the system must discard KV cache and rerun prefill computation, increasing latency and raising total cost of ownership (TCO).

To address this, Nvidia released Dynamo, a KV cache offloading software, in March 2025. Dynamo offloads infrequently accessed KV cache to lower-bandwidth, higher-capacity, and more cost-effective tiers such as CPU RAM and SSD, ensuring this data remains reusable during the inference decode stage without requiring prefill recomputation.

Dynamo can be paired with the CMX Context Memory Storage Platform, which Nvidia introduced in January 2026, designed to store and manage the massive KV cache generated by long-context workloads. Built around the BlueField-4 STX Rack, CMX uses 64 BlueField-4 DPUs (4 BlueField-4 DPUs x 16 compute trays) to manage approximately 9,600 TB of capacity per rack, adding a pod‑level context tier between local SSD and shared storage.

Figure 4. Nvidia BlueField-4 DPU structural model showcased at COMPUTEX 2026, featuring SK hynix's PEB210 E1.S and PE9010 M.2 SSD samples.

Source: TrendForce

Figure 5. Nvidia Dynamo KV Cache Offloading Sequence (G1 to G4)

Agentic AI: More CPUs, More RAM

The proliferation of AI inference is also accelerating the deployment of agentic AI applications. In AI agent workflows, models must actively plan, call tools, make decisions, and execute actions on behalf of users, all of which require CPUs to handle orchestration, tool calls, data routing, and sub-agent evaluation. As a result, the CPU-to-GPU workload ratio in agentic AI deployments is expected to shift from a traditional ratio of 1:4 or 1:8 to approximately 1:1, creating significant growth potential for the CPU market and driving new demand for CPU RAM.

2026 Agentic AI Wave: CPU Shortage and GPU Ratio Structural Changes

NVIDIA debuted its standalone Vera CPU Rack at GTC, and Arm followed with its own AGI CPU racks. CPUs are turning critical as supply tightens and Intel, AMD raise prices. Who wins the CPU race?

Stay Updated

Figure 6. Comparing the CPU to GPU workload ratio between traditional LLM and agentic AI applications, reflecting a shift toward a 1:1 CPU to GPU ratio.

In 2026, Nvidia introduced the Vera CPU, purpose-built for agentic AI workloads. Based on the original specification, Vera supports up to 1.5 TB of LPDDR5X memory capacity, three times the capacity of the previous-generation Grace CPU.

Figure 7. NVIDIA Vera CPU Architecture

Source: Nvidia

Figure 8. SK hynix showcased its SOCAMM2 and HBM4 samples designed for Nvidia Vera Rubin Superchip at COMPUTEX 2026.

Source: TrendForce

Beyond Nvidia, the broader CPU landscape is also accelerating. Traditional x86 vendors Intel and AMD launched new products in 2026, with Intel’s Xeon 6+ (Clearwater Forest) and AMD’s EPYC Venice. On the Arm side, Arm launched the Arm AGI CPU, while Ampere’s AmpereOne MX is expected to enter mass production later this year. 2026 is shaping up to be the year of a full-spectrum CPU refresh for agentic AI.

Figure 9. CPU Specifications by Vendor

For full specifications and related market data, please visit the Report Page, leave a Message, or Email (SR_MI@trendforce.com) the Sales Department.

In sum, we are seeing two new drivers of memory demand generated by AI inference. On one front, inference workloads are driving KV cache consumption to expand rapidly, and KV cache offloading technology enables large volumes of KV cache to be offloaded to CPU RAM and SSD POD. As Nvidia, Google, and others introduce new SSD POD platforms, demand in this segment is expected to continue rising.

On the other front, agentic AI is shifting the CPU-to-GPU workload ratio toward 1:1, creating significant growth potential for CPU demand and driving a corresponding increase in CPU RAM demand.

2026 Trends: Memory for New AI Inference Demand

NVIDIA's CMX platform, run by BlueField-4 DPU, extends memory between local SSD and shared storage for AI inference's KV cache demand. NVIDIA and Arm's CPU racks for agentic AI are opening a new CPU memory market.

Master the Market Trends