SemIsrael Blogs

Introduction

For years, the AI conversation had one default answer: GPU. Training, inference, everything — add more accelerators. That framing is now broadening. GPUs remain the engine for dense, high-concurrency math, and nothing about the CPU’s return changes that. But as inference shifts toward agents, tool calls, and multi-step reasoning, a growing share of the work lands back on general-purpose cores. Intel has pointed to the CPU-to-GPU ratio moving from roughly 1:8 in training toward 1:1 — and as high as 4:1 in some agentic deployments. The two aren’t competitors so much as a division of labor: GPUs brute-force the matrix math, CPUs run the orchestration, control flow, and I/O that modern AI actually depends on.

There’s a catch that doesn’t make the headlines, though: a stronger role for the CPU isn’t a core-count story. It’s a bandwidth story.

The wall nobody upgraded

Server core counts have climbed steeply — some designs have nearly tripled their cores over the past several years — while per-core memory bandwidth has moved far more slowly. The result is a processor that can be data-starved long before it’s compute-bound. Add cores past that point and you don’t add useful work; you add cores waiting on memory. This is the memory wall, and in 2026 it’s no longer a niche concern. It’s the constraint processor vendors and memory suppliers alike are now naming out loud.

It shows up hardest exactly where the CPU is taking on more: AI inference and KV-cache reads, in-memory databases, large-scale analytics, and highly parallel server workloads. In these environments, delivered throughput is often set by the memory subsystem, not the theoretical compute of the socket.

Why standard DDR5 RDIMM runs out of runway

The obvious fix — keep pushing DDR5 signaling rates — runs into real physics. Higher rates strain signal integrity, power, and channel loading, and each step gets harder. There’s also a structural limit worth understanding, because it’s the heart of the problem.

A conventional DDR5 RDIMM splits the module into two independent 40-bit sub-channels, but within each sub-channel it can access only one rank at a time. You can add ranks to grow capacity, but rank access stays serialized. In practical terms, an RDIMM can deliver up to two cache lines concurrently — one per sub-channel — and no more, regardless of how many ranks you install. As request concurrency rises, rank conflicts, refresh activity, and scheduling constraints cause throughput to saturate earlier, and latency climbs sharply as queues build. More capacity, in other words, doesn’t buy more parallelism — and parallelism is exactly what a busy, many-core socket needs.

To continue reading the full blog – click here