YWR: The Future is Memory

Disclaimer: this is not financial advice!

This summer Korean retail trades were margined to the max buying SK Hynix shares.

4 months later these same traders are selling their shares down 56%.

It is sad. And I feel bad for them.

But maybe it is an opportunity for everyone else..

The opportunity in memory is unique and still the same trade. (Most) investors don’t see the broader trend. That the center of gravity in AI computing is shifting to memory.

Memory is the future.

Let’s start with forward P/E ratios of the most important AI companies.

Does something look weird here?

I get there are concerns about AI and whether it’s a bubble and whether there will be funding for datacenters in the future and on and on. But isn’t that a problem for everyone on this list?

An AI bubble crash would kill everyone from Nvidia to ASML, to TSMC, Micron and SK Hynix.

And yet the valuations are hugely skewed against memory.

Maybe memory earnings estimate are crashing?

Nope. Earnings estimates for SK Hynix and Micron look fine so far.

The problem is investors still see memory stocks as a commodity product. A commodity product always on the verge of crashing, which is why the sector is so reactive to any news from Chinese memory company CXMT.

We are fighting that old adage.

“When China walks into a room. Profits walk out.”

Except, I would argue CXMT is not in the most important room, and may never be.

Investors are also missing where this sector is going and how the center of gravity in AI is shifting from GPU’s to memory.

Let’s go back.

What was the story of Nvidia?

I’m simplifying, but what Jensen saw first is that a GPU could be more than just a graphics chip. A GPU could also be used for processing, and was especially good for AI applications. This was the game changer. Investors in Nvida caught the shift in relative importance from the CPU to the GPU, and got rich.

I see the same thing happening with memory.

We’ve talked before about the Memory Wall; that memory chips are unable to keep up with the speed of the GPU. Memory chips cannot feed data to the GPU fast enough. The GPU does its computations then sits around twiddling its thumbs waiting for more data. The memory chip, but also getting the data through the network to the memory chip is the constraint. Memory is the big bottleneck and where all the great technical minds are focused.

The memory bottleneck is expected to get worse along two dimensions.

First, conversation length and context windows are growing. These longer context windows are shipped to the LLM in every message and increase the memory requirement.

The second dimension is agent runtime. AI Agents are working on tasks for longer, using more tools, and holding memory state while they work.

The solution so far has been two fold. SK Hynix invented HBM (high bandwidth memory); the 12 story tall DRAM memory chips (HBM) which have been incredibly profitable. The other solution has been on the software side in finding ways to compact context.

HBM is working for now, but the memory problem will get worse from here as AI agents increase in use and complexity. We see this with Meta Muse and Grok bots which are taking off in popularity. These are not chat bots answering questions. They are always on agents with their own server. Meta Muse Agents and Grok Bots are a growing workforce slaving away all day returning things to Amazon, finding the highest interest bank account or trading shares on IBKR (Asset Management as a Service).

How do we handle millions of agents running all day?

The solution from here might not be an even taller HBM chip, because the problem is increasingly not just the speed of the HBM chip. It’s the data network connections all the way to the GPU.

What if catching up to the GPU’s means rethinking everything?

What if the solution from here is something totally different?

And what if SK Hynix is changing the game again?

Same way they did with HBM.

Memory Centric AI Rack System (MARS)

SK Hynix has a culture of crazy engineering.

They were inventing HBM chips when there was no practical use for a memory chip that fast. But they thought it was cool and some use case would eventually come along. That use case was AI. It’s why they have 55% market share in HBM and are the first in each new generation.

I say this to point out their culture is unique and why I’ve been keeping tabs on their press releases around the future of memory.

SK Hynix has been hinting that solving the memory bottleneck from here requires thinking everything from first principles.

When they looked at the problem from every angle it was hard to avoid an obvious conclusion. Too much energy and time was being spent moving data to the GPU. Ideally, energy should be spent on purely on computing, not data movement.

What if instead of trying to move the data faster to the GPU the solution was bringing the GPU closer to the data? And also possibly do some early computation within the memory itself?

Maybe instead of all the memory chips clustered around the GPU there could be a memory server rack where all the GPU’s go to get a pooled memory resource.

As I said SK Hynix has been hinting at this concept before, but last week presented it for the first time at the OCP Global Summit in San Jose. They call it MARS (a memory centric AI rack system).

At this demo, the company presented the results of validating—at the PoC level—whether a KV cache*-sharing solution in an agentic AI environment and an embedding-sharing solution in ‘OmniServe’, an omni-modal service, actually work. Having analyzed real customer systems and workloads to confirm the applicability of its CXL-based solutions, SK hynix has laid the groundwork for a full-scale expansion of these solutions.

Why do I find this so interesting even if I don’t really understand KV-cache sharing?

Because I like where this is going. Especially, the ‘memory centric’ part.

A memory server where GPU’s go to get their data sounds like SK Hynix shifting from a component supplier to an AI systems architect. Sounds Nvidia’ish. It sounds like SK Hynix changing the relative importance of memory and potentially where the value accrues. It also sounds like SK Hynix is extending the number of years they can make these super normal profits.

Memory centric AI rack systems sounds like a 15x P/E business, not 3x.

It also makes me not care if a recently IPO’d Chinese memory company is working on a new DRAM chip. The future of memory is moving to a complex interplay of high bandwidth NAND (HBF), High bandwidth DRAM (HBM), and pooled memory. It’s whole system not a chip.

What you also notice is the technical partnerships. The memory rack was developed together with AMD, Penguin, Xcena, and Arcee. These AI memory technologies require tight working relationships. The whole enterprise AI server is designed together.

It’s also a long development pipeline. Next generation AI servers are co-designed years in advance by AMD and Nvidia with SK Hynix and will for national security reasons not be designed with Chinese chips. These close-knit technical partnerships and all the data sensitivities around AI make it really hard for a Chinese player to muscle in and smash the AI enterprise party. It’s an AI version of the Japanese keiretsu supplier relationships, which American companies used to complain about.

In summary here is what I see.

• GPU’s are fast enough for now.

• The problem moves to speeding up the memory.

• Memory products are getting increasingly complex.

• HBF and HBM are still difficult to package and difficult to commoditise.

• From here the next product is potentially a memory-centric server. Memory as a system.

Given all this does it make sense that SK Hynix is trading on 3x forward estimated earnings? Does this sound like SK Hynix asleep at the wheel about to get run over by CXMT or Samsung?

It’s sad that Korean retail traders have lost 56%, but if they could see the bigger picture and where memory is going maybe they wouldn’t be selling.

But remember, this is not financial advice! Always ask your financial advisor what they think is right for you.