Memory Hierarchy Explained: How Computers Remember Everything

7 min readDec 23, 2025

This article will deep dive into the concepts of memory organisation and memory hierarchy.
So, sit back, relax your mind, and let’s gain some knowledge!

credit : https://media.geeksforgeeks.org/wp-content/uploads/20250407161208309014/frame_1.webp

The above diagram might be new to some people, or it might look confusing at first. But don’t worry — I am here to simplify everything for you.

What Is Memory?

First, tell me — what is memory?
Basically, anything that stores something can be called memory.

For example, your brain. It stores tons of information, hence it is memory.

Similarly, a computer also has memory to store the content of every process it performs. The CPU is the brain of the computer that performs computation, but memory is the storage container for both the inputs to computation and the outputs of computation.

RAM and ROM — You’ve Heard These Before

I hope you’ve already heard the terms RAM and ROM.

RAM — Random Access Memory

You can perform read and write operations million of times.
The data stored in it is temporary. As you’ll turn off your computer, its data will be gone.

This property is volatility.

ROM — Read Only Memory

As the name suggests, in this type, we can only read the content of memory. We can not change the content of it.
It does not require power supply. Even when power if off, your computer is off, it still holds the same data. When you’ll start your computer again, it will hold the same data.

Do you know what this property is called?
👉 Non-volatility

Which One Is More Important — RAM or ROM?

Now that you know RAM and ROM, tell me — which one is a must for your computer?

I know, I know 😄
Both have their own benefits, and that’s why both are essential.

One stores important data permanently.
The other allows writing and executing new data.

That’s why both RAM and ROM are present in your computer.

Together, they are called Primary Memory —
“Primary” because they are the main and essential memories.

Secondary Memory — Extra but Crucial

The other type of memory is called Secondary Memory, which is additional memory added based on our requirements.

You all are computer geeks, and whenever you buy a laptop, computer, or smartphone, the first thing you check is memory capacity. That’s how important memory is!

To check the memory capacity of your computer:

Settings → About Your PC

There, you’ll find information about your RAM capacity.

Now, here’s something interesting.
You’ll notice something like:

RAM: 16 GB
Storage: 477 GB

Why is there extra storage?

That extra storage is your SSD, which is secondary memory. It stores:

Images
Videos
Windows files
Programs
And much more

What About ROM Capacity?

ROM capacity is generally around 4–8 MB.

Do you see the huge difference?

RAM → GB
ROM → MB

Why such a big difference?
We’ll discuss that soon!

Why Secondary Memory Became Necessary

Till the early 1970s, computers were used only for complex calculations and logic.

In the 1980s, brilliant minds like Bill Gates and Steve Jobs envisioned computers as daily-use devices — capable of storing and sharing images, videos, and media.

Even today, we use computers mostly for multimedia.
Honestly, when was the last time you actually programmed something on your computer? 😭

But there was a big problem.

Images, songs, and videos take huge storage space.
That’s why secondary memory was introduced. They have huge storage capacity.

What all comes in secondary memory? Hard Disk, Floppy Disk, CD, DVD , Pen Drive etc.

So, what you earlier saw in that pic, 477 GB, that is actually my Hard disk storage.

If you’ll further go on to System->Storage — you will find they stores images, videos, your programs and temporary files —

Why CDs, DVDs, and Pen Drives?

Even though hard disks exist, these devices provide portable memory — meaning you can carry your data anywhere.

Also, do you notice how size has increased —

ROM ( 4–8 MBs ) , RAM (16 GB ) , HardDisk ( 512 GB )

Latest computer’s hard disk are having capacity of 1 TB as well.

Hard disks have properties of both RAM and ROM:

You can write new data
Data is non-volatile

They also offer huge storage at a lower cost compared to RAM and ROM.

Then Why Not Replace RAM and ROM?

If secondary memory is cheap and large, why not replace RAM and ROM entirely?

👉 Because secondary memory is very slow.

Get Ayushi Pal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Why slow?

Secondary memory devices are disk-based.
They need to rotate to access data.
This takes significantly more time.

How Primary and Secondary Memory Work Together

Primary and secondary memory work together to make the computer fast:

Primary memory → Execution (fast)
Secondary memory → Storage (cheap & large)

For example:

When you download a movie, it’s stored in secondary memory.
When you play it, it’s loaded into RAM and then played.

That’s why there’s always a small delay before a video starts playing.

You don’t load the entire 2-hour movie into RAM at once.
Instead, the program is divided into smaller independent parts, which are loaded and executed one by one.

If you’ve studied Operating Systems, you might recognize this concept.

👉 Paging 😱🔥

We fetch data/processes from secondary memory into main memory for execution — that’s exactly paging!

(I really love how different subjects connect together 🙂)

Let’s not go deeper here, or we’ll drift fully into OS topics.

What Does ROM Actually Do?

We know RAM stores running programs.
But what about ROM? Why we need that?

Before answering that, tell me — where is your OS stored?

Windows, macOS, Linux — all are stored on the hard disk.

When you start your computer, does it start immediately?
No — it takes 3–5 seconds.

Why?

Because the OS needs to be loaded into RAM.

But what triggers this process?

👉 BIOS

And where is BIOS stored?

👉 ROM

That’s also the reason why ROM size is so small (I told you earlier, i will explain later 😉) — it mainly stores BIOS instructions.

RAM Speed = Computer Speed

If RAM executes every instruction, then yes —
the speed of RAM determines the speed of the computer.

Now, RAM are of two types — SRAM and DRAM.

SRAM — Static RAM

DRAM — Dynamic RAM

SRAM is much faster than DRAM but also far more expensive.
That’s why most computers use DRAM.

So how do we make computers faster yet affordable? Here comes Cache!

Cache — The Game Changer

Cache is:

Very small
Extremely fast
Built using flip-flops

Parts of RAM are loaded into cache, and the CPU executes programs directly from cache.

And no — cache isn’t a recent invention.

Cache was introduced in 1986, when Intel’s 8086 processor included cache memory.

Now, when you see an ad of computer product, you will easily understand what all that specification means. Isn’t that is what makes you a good computer engineer? Computer engineering isn’t just understanding how to reverse a linked list :)

Cache Inside the Processor

There was a problem they faced. The data between processor and cache is shared through data bus. Though, both processor and cache are fast, but data bus is very slow.

So, there was a bottleneck of max speed acheived.

Then came Intel’s Pentium processor , which shook the world with the high speed, but how? —

They said let’s move cache inside processor! No data bus involvment!

Now we had:

L1 Cache (inside the processor)
L2 Cache (outside initially)

Now, the internal bus inside cache and processor is very high speed and beyond that , it could be expanded.

Do you remember, in one of my article, i told you one of the restriction in increasing the processor’s speed was databus expansion ( Link )

You could not believe , pentium was having 256 bits internal data bus !! and no, they didn’t even stop there. They stored two cache (L1 is now a split cache ) — I-cache ( for storing instructions ) and D-cache (for storing data ) .

Also, relating my previous atricle ( Link ) on pipeline hazards, this split cache aligns perfectly with the Harvard architecture model !!