We’ve spent billions optimizing how AI chips compute. We’ve largely ignored how they remember.

Modern AI hardware has an efficiency problem: moving data consumes far more energy than the computation itself — and the architecture of AI models is making this worse.

The basic problem: Traditional computing separates computation from memory. Every operation requires data to be fetched, processed, and written back. Research shows data movement between logic and off-chip memory consumes 10–100x more energy than the arithmetic itself (Horowitz, 2014). In memory-bound workloads — which describes most modern AI inference — memory access routinely dominates the power budget, often accounting for 30–50% of total system energy.

The industry has been attacking this for decades through caches, high-bandwidth memory (HBM), and near-memory computing. These definitely help. But AI workloads are introducing memory demands that make the underlying problem much harder.

Why LLMs specifically are brutal on memory: LLMs access memory in patterns that are particularly expensive under conventional architecture.

The most acute example is the KV cache. During inference, transformers must retain the attention keys and values for every token in the context window to generate each new token. This cache grows linearly with sequence length and must be read in full at every generation step. For large models serving long contexts, the KV cache alone can occupy tens of gigabytes of HBM and must be streamed through the memory bus continuously, at every single token generation step.

This creates a brutal bandwidth problem. The GPU becomes memory-bandwidth-bound, sitting idle waiting for data to arrive. This is why techniques like flash attention produced such dramatic real-world speedups despite being algorithmically equivalent to standard attention. The bottleneck was never the math.

Other LLM-specific memory pressures compound this:

• Model weights for frontier models run to hundreds of gigabytes, often requiring multi-GPU setups purely for memory capacity, not compute
• Speculative decoding and batching strategies exist largely to amortize memory access costs across requests
• Quantization (reducing weight precision from FP16 to INT8 or INT4) is widely deployed not primarily to reduce compute, but to shrink memory footprint and increase effective bandwidth

The industry is responding incrementally: HBM generations (HBM2, HBM3, HBM3E) have delivered meaningful bandwidth increases, and NVIDIA’s GB200 NVLink architecture moves toward tighter memory pooling across chips. But these are bandwidth increases within the same fundamental architecture — the processor and memory remain separate, and data still has to travel between them. Each HBM generation is also more expensive and more power-hungry to manufacture and operate.

The question is whether incremental bandwidth scaling can keep pace with context windows that are growing from thousands to millions of tokens, and model sizes that continue to expand.

Where more fundamental approaches come in: This is the legitimate engineering case for architectural alternatives. Processing-in-memory (PIM) chips from Samsung and SK Hynix move compute closer to where data lives. Analog in-memory computing (IMC) goes further, performing computation directly within the memory array and eliminating the fetch-process-writeback loop at its root, which is particularly good for matrix-vector operations that dominate transformer inference.

These approaches have real limitations: precision constraints, programmability challenges, and manufacturing maturity gaps. But the KV cache problem illustrates exactly why they’re attracting serious research investment: it’s a workload that is structurally, not incidentally, memory-bound.

The bottom line The von Neumann bottleneck isn’t a new problem, but LLMs have made it newly urgent. The KV cache, weight loading, and attention patterns of modern transformers are architecturally mismatched with processors that treat memory access as an afterthought. Algorithmic workarounds like flash attention and quantization help — but they’re navigating around an architectural constraint, not resolving it.

Breaking the bottleneck To truly scale next-generation AI, we must eliminate the physical separation between processing and storage.

TetraMem solves this fundamental crisis at the architectural level. By using TetraMem’s analog in-memory computing solution, you can perform computations directly inside the memory array itself. This radically eliminates the Von Neumann bottleneck, slashes data movement energy costs, and delivers the ultra-high bandwidth required for intensive LLM inference.

Don’t let memory constraints throttle your AI performance. Learn about TetraMem’s analog in-memory compute solution today to unlock unprecedented energy efficiency and eliminate the memory bottleneck for good.