Building PebbleFS — An experimental distributed storage engine written in Go.
Instead of relying on a single storage device, PebbleFS aggregates heterogeneous storage devices — USB drives, HDDs, SSDs, and flash drives — into a single logical storage pool.
Rather than letting perfectly usable storage sit idle because each device is too small to be useful on its own, PebbleFS combines their capacity into one unified system. Files are split into fixed-size, content-addressed (SHA-256) chunks, distributed across registered disks using a pluggable placement strategy, and transparently reconstructed during reads, making a collection of disparate storage devices behave like a single storage engine.
X Thread(demo) : https://x.com/berzzdotdev/status/2084165943492858074
GitHub Repo : https://github.com/berzz26/pebbleFS
V0.1 Release : https://github.com/berzz26/pebbleFS/releases/tag/v0.1.0
Why PebbleFS Exists
I have a decent number of storage devices lying around. USB drives, SSDs, HDDs, old flash drives, new flash drives, pretty much every random storage device you accumulate over the years.
Most of them aren’t broken.
They’re just too small to be useful on their own.
A 16 GB USB drive isn’t particularly exciting. Neither is a spare 64 gig one. Even an old 500 GB hard drive isn’t something I’d build a server around anymore. But together, they represent storage that’s simply sitting unused.
That got me thinking about a much more interesting problem.
How do I use every storage device I own as one storage system?
Not just the drives attached to one computer, but storage connected to multiple Linux machines as well (later version of pebbleFs).
That question eventually became PebbleFS.
More than just combining disks
At first glance, this doesn’t sound particularly difficult.
Modern operating systems already have technologies like RAID and LVM that can combine multiple disks into what appears to be a single volume.
So why build something else?
The answer is that RAID and LVM solve a different problem.
They’re designed for disks inside a single machine. They assume relatively stable hardware, predictable storage devices, and direct access to every drive participating in the array.
My requirements were very different.
I wanted to use whatever storage happened to be available.
A USB drive plugged into my desktop.
A SSD inside my laptop.
Spare HDDs .. etc
Those devices have different capacities, different speeds, different filesystems, and sometimes different levels of reliability.
I wanted a storage engine that treated every mounted storage device as another place where data could live.
Storage shouldn’t care where it comes from
One of the principles that guided PebbleFS from the beginning was that storage should be abstracted away from the hardware providing it.
An application shouldn’t have to know whether its data is sitting on an NVMe SSD, a USB flash drive, or a hard disk attached to another computer.
It should only care that the data exists and can be retrieved.
If Linux can mount a storage device, PebbleFS should be able to use it.
That sounds simple, but it changes the way you think about storage.
Instead of managing individual disks, you start managing a pool of capacity.
Instead of asking which drive should receive a file, you ask how the storage engine should distribute data across the available devices.
Physical hardware becomes an implementation detail.
The interesting problems come later
Combining storage is actually the easy part.
The moment you move beyond a proof of concept, the real questions start appearing.
Suppose someone uploads a large file.
Where should it go?
- What happens when a drive runs out of space?
- What if one device is significantly faster than another?
- What if a USB drive is unplugged halfway through a write?
- How do you find data again after distributing it across multiple disks?
- How do you avoid storing identical data twice?
- How do you recover when a device fails?
- How do you add another storage device without reorganizing the entire system?
Every answer introduces another question.
That’s exactly what makes distributed storage interesting.
Most of the complexity doesn’t come from writing bytes to disk.
It comes from making those bytes reliable, discoverable, and scalable as the system grows.
Why build another storage engine?
Now why would someone spend time building another storage engine when systems like Ceph, GlusterFS, and MinIO already exist.
PebbleFS isn’t trying to compete with them.
Those projects are mature, production-ready systems built by teams of experienced engineers over many years. They’re designed for environments that are significantly more demanding than my homelab.
PebbleFS has a different purpose.More specific to catering my personal needs for my homelab
It’s an opportunity to understand how modern storage systems work by building one from first principles.
Part 1: Everything Is Bytes
When people think about storing files, they usually think in terms of documents, photos, videos, or music. Operating systems reinforce that idea by presenting everything as folders full of named files with icons and extensions. To us, a .jpg is a photo, an .mp4 is a video, and a .pdf is a document.
A storage engine doesn’t have that luxury.
Once a file reaches the storage layer, every bit of meaning attached to it disappears. The filename is just metadata. The extension is just metadata. Even the directory it came from is just metadata. Strip all of that away and what remains is nothing more than a sequence of bytes.
photo.jpg
↓
FF D8 FF E0 00 10 4A 46 49 46 ...Those bytes happen to represent a JPEG image because some application understands the JPEG format. The storage engine doesn’t.
To PebbleFS, the bytes above are no different from the bytes of a Linux ISO, a SQLite database, or a Go executable.
That realization shaped one of the earliest design decisions in PebbleFS.
The storage layer should never understand file formats.
If I have to write special handling for images, videos, databases, archives, or any other file type, then something has gone wrong.
A storage engine shouldn’t care what the bytes represent. It should only care about storing them reliably and returning them unchanged when asked.
This might sound obvious, but it’s a surprisingly important distinction because it defines the responsibility of the entire project.
Storage engine vs filesystem
There’s an important difference between a storage engine and a filesystem, even though they’re often discussed together.
A filesystem is responsible for everything users interact with.
It manages directories.
It keeps track of filenames.
It stores permissions.
It handles timestamps.
It resolves paths.
It implements symbolic links.
It exposes a familiar interface so applications can open files with a path like:
/home/user/Documents/report.pdfA storage engine sits underneath that.
It doesn’t care whether the file came from /home, /var, or /mnt/backup.
It doesn’t even know those paths exist.
Its job begins only after someone has already decided what bytes need to be stored.
That means PebbleFS is not trying to replace ext4, XFS, or NTFS. Those filesystems already solve problems related to organizing files on a single storage device.
PebbleFS solves a different problem.
Given an arbitrary stream of bytes, how do we store it across multiple heterogeneous storage devices and retrieve it later?
Everything else belongs somewhere above it.
Thinking in layers
Once I separated the idea of a storage engine from a filesystem, the architecture became much easier to reason about.
Instead of trying to build one giant application that did everything, I could split responsibilities into layers.
Applications
(CLI • SDK • REST API)
│
▼
Future FUSE Filesystem Layer
(Mount as a local filesystem)
│
▼
┌────────────────────────────────────────────────────────────────────────────┐
│ PebbleFS Core │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Metadata │──▶│ Chunking │──▶│ Placement │──▶│ Storage │ │
│ │ Engine │ │ Engine │ │ Strategy │ │ Manager │ │
│ └─────────────┘ └─────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ SHA-256 hashing Pluggable Read / Write │
│ │ Deduplication algorithms Device I/O │
│ │ │ │
└────────┼────────────────────────────────────────────────────────┼──────────┘
│ │
▼ ▼
Logical Distributed Storage Pool
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────┐
│ USB Drive│ │ SSD │ │ HDD │ │Remote Node │
└──────────┘ └──────────┘ └──────────┘ └────────────┘
▲ ▲ ▲ ▲
└──────────────┴──────────────┴───────────────┘
Heterogeneous Storage DevicesApplications interact with files.
The filesystem translates file operations into storage operations.
PebbleFS manages chunks, metadata, placement, and storage.
Finally, the physical devices store the bytes.
Every layer has exactly one responsibility.
That separation is intentional because it allows each layer to evolve independently.
If I eventually decide to expose PebbleFS through FUSE, the storage engine doesn’t need to change.
If I later build an S3-compatible gateway, the storage engine doesn’t need to change.
If I expose it over HTTP, nothing changes internally.
Every interface simply becomes another way of talking to the same storage engine.
The operating system has already solved enough problems
One of the easiest mistakes to make when building systems software is reinventing things that already exist.
Linux already knows how to talk to disks.
It already understands USB drives.
It already mounts ext4, XFS, NTFS, exFAT, and dozens of other filesystems.
Trying to replace all of that would be an enormous project with very little benefit.
PebbleFS doesn’t need to know how to read sectors from a disk.
Linux already does that.
PebbleFS doesn’t need to parse an ext4 filesystem.
Linux already exposes a mounted directory.
That means PebbleFS can simply treat every mounted path as usable storage.
/mnt/ssd
/mnt/hdd
/mnt/usbAs far as PebbleFS is concerned, they’re all the same thing.
Directories.
This turned out to be one of my favorite design decisions because it makes the storage engine completely independent of the underlying filesystem.
Whether a drive is formatted as ext4, XFS, NTFS, or exFAT is irrelevant.
If Linux can mount it and PebbleFS has permission to write into it, then it becomes part of the storage pool.
That also makes heterogeneous storage almost effortless.
A slow USB drive, a high-speed NVMe SSD, and a network-mounted directory can all coexist because PebbleFS never communicates with the block device directly.
It communicates with the filesystem that Linux has already mounted.
Files are temporary. Data isn’t.
Applications think in files.
Storage engines think in data.
Imagine uploading a 10 GB virtual machine image.
The application sees one file.
PebbleFS eventually breaks that file into smaller pieces, distributes those pieces across multiple storage devices, and records enough metadata to reconstruct the original file later.
Internally, the file no longer exists as one continuous object.
It becomes a collection of independent pieces that just happen to represent a file when assembled in the correct order.
That distinction becomes more important as the project grows.
Replication won’t happen at the file level.
Deduplication won’t happen at the file level.
Balancing won’t happen at the file level.
Recovery won’t happen at the file level.
Almost every operation inside PebbleFS will eventually operate on chunks rather than complete files.
The file is simply the interface users understand.
Internally, everything revolves around data.
What’s next?
Now that we’ve established that PebbleFS only stores bytes, a much more practical question appears.
If someone uploads a 50 GB file, where exactly do those bytes go?
Storing the entire file as one object seems like the obvious answer.
It also turns out to be one of the worst possible designs.
The next chapter begins with a simple idea that fundamentally changes how the rest of the storage engine works:
Instead of storing files, PebbleFS stores chunks.
Part 2: Breaking Files into Chunks
Where do those bytes actually go?
The most obvious answer is to store every uploaded file as a single object.
If someone uploads a file called movie.mp4, simply write the entire file onto one storage device. right?
movie.mp4
↓
SSDThe implementation is almost trivial.
Open the destination.
Copy bytes.
Close the file.
Done.
For a storage engine running on a single disk, this isn’t even a bad design. In fact, it’s exactly how many applications store data internally.
The problem only appears once storage stops being a single device.
One file, one disk
Imagine PebbleFS is managing three storage devices.
SSD HDD USBNow suppose someone uploads a 30 GB virtual machine image.
Which disk should receive it?
If the SSD only has 20 GB free, the upload fails.
The hard drive might have another 2 TB available.
The USB drive might have another 500 GB.
Together there’s plenty of free space.
Individually, there isn’t enough room on the chosen device.
The storage engine has accidentally limited itself to the capacity of its largest disk instead of the capacity of the entire storage pool.
That defeats the entire purpose of combining storage in the first place.
Large files become immovable
Even if the upload succeeds, another problem appears later.
Suppose PebbleFS decides to store the entire file on the SSD.
A month later, another SSD is added to the cluster.
The original SSD is now nearly full.
The new one is completely empty.
Ideally, the storage engine should spread data across both devices.
But because files are indivisible, the only option is moving the entire 30 GB file.
That’s expensive.
Every rebalance becomes a massive copy operation.
Every migration moves gigabytes of data.
Every interruption means restarting a huge transfer.
The storage engine has very little flexibility.
Failure becomes catastrophic
Now imagine something worse.
The SSD dies.
Every file stored on it disappears.
Recovery is all or nothing.
If a disk contained one thousand files, every one of those files is suddenly unavailable.
There is no opportunity for partial recovery because every file was tied entirely to a single storage device.
That’s not a particularly resilient system.
Thinking smaller
What if a file didn’t have to live on one device?
Instead of treating the file as one giant object, what if we divided it into smaller pieces?
Suppose we choose a chunk size of 4 MB.
A 10 MB file suddenly becomes three independent chunks.
example.bin┌─────────────┐
│ Chunk 1 │ 4 MB
├─────────────┤
│ Chunk 2 │ 4 MB
├─────────────┤
│ Chunk 3 │ 2 MB
└─────────────┘Those chunks don’t have to stay together.
Chunk one could be stored on an SSD.
Chunk two on a hard drive.
Chunk three on a USB drive.
Chunk 1 → SSDChunk 2 → HDDChunk 3 → USBAs long as PebbleFS remembers where every chunk lives, reconstructing the original file becomes straightforward.
Everything becomes more flexible
Breaking files into chunks changes almost every aspect of the storage engine.
Large files are no longer constrained by the size of a single storage device. A file can span dozens of disks if necessary because no individual chunk is particularly large.
Balancing storage also becomes significantly easier. Instead of moving an entire 30 GB virtual machine image, PebbleFS can move a handful of 4 MB chunks from one device to another. The end result is the same, but the work required is dramatically smaller.
Failures become more manageable as well. Losing a disk no longer means automatically losing every file stored on it. It only means losing the chunks that happened to live there. If those chunks have replicas elsewhere, recovery is simply a matter of copying them back.
The unit of storage has changed.
Instead of managing files, PebbleFS manages chunks.
That distinction influences almost every other part of the system.
Choosing a chunk size
The next question is obvious.
Why 4 MB?
Why not 512 bytes?
Or 1 GB?
There isn’t one correct answer.
Smaller chunks make balancing extremely precise.
If a storage device becomes full, PebbleFS can move tiny amounts of data until the cluster is balanced again.
The downside is metadata.
Every chunk needs an entry somewhere.
A 100 GB file split into 4 KB chunks produces more than twenty-six million chunks.
That’s twenty-six million hashes.
Twenty-six million metadata records.
Twenty-six million placement decisions.
The metadata quickly becomes more expensive than the storage problem itself.
Going in the opposite direction has different tradeoffs.
Suppose every chunk is 1 GB.
Metadata becomes tiny.
Placement decisions become simple.
But balancing becomes coarse.
Moving a little bit of data now means moving an entire gigabyte.
Replication also becomes slower because every operation is much larger.
Somewhere in the middle is a reasonable compromise.
PebbleFS currently uses fixed-size chunks(16MB) because they’re simple, predictable, and easy to reason about. The exact size isn’t special. It’s simply a configuration value chosen to balance metadata overhead with operational flexibility.
Snippet of the pebbleFs splitter :-
// here, well be using stream processing with channels.
// instead of waiting for the ENTIRE file to be read before writing a single chunk, pebbleFs will be writing the chunk as soon as its read.
func Split(path string) (<-chan Chunk, <-chan error) {
chunkChan := make(chan Chunk)
errChan := make(chan error, 1)
go func() {
defer close(chunkChan)
defer close(errChan)
file, err := os.Open(path)
if err != nil {
errChan <- err
return
}
defer file.Close()
buffer := make([]byte, config.ChunkSize)
index := 0
for {
n, err := file.Read(buffer)
if err != nil {
if err == io.EOF {
break
}
errChan <- err
return
}
/*
file.Read() reuses the same buffer every iteration.
If we directly send `buffer`,
the next Read() would overwrite it.
So we create a new slice and copy
only the valid bytes.
*/
data := make([]byte, n)
copy(data, buffer[:n])
chunkChan <- Chunk{
ID: Hash(data),
Index: index,
Data: data,
Size: int64(n),
}
index++
}
}()
return chunkChan, errChan
}A chunk is just another file
One thing surprised me while implementing this.
After all the discussion about chunking, placement, and distributed storage, each chunk ultimately becomes… another file.
chunks/4a9d1c...71bf92...c84ef0...PebbleFS doesn’t invent a new on-disk format.
It simply stores each chunk as its own file on the underlying filesystem.
Linux handles writing bytes to disk.
PebbleFS handles deciding which bytes belong together and where they should be written.
That separation keeps the storage layer remarkably simple.
The next problem
Splitting files into chunks solves the problem of distributing data across multiple storage devices.
It immediately creates another one.
How do you identify those chunks?
Assigning random IDs works.
But there’s a much more interesting approach.
Instead of naming chunks arbitrarily, what if the data could name itself?
That single idea becomes the foundation for deduplication, integrity verification, immutable storage, and several other parts of PebbleFS.
The next chapter explores why every chunk in PebbleFS is identified by the SHA-256 hash of its contents rather than by a random identifier.
Part 3: Letting Data Name Itself
By this point, PebbleFS can split a file into fixed-size chunks.
That’s already a significant improvement over storing entire files. Chunks can be distributed across multiple storage devices, moved independently, and eventually replicated without treating the original file as one giant object.
But splitting data into chunks immediately creates another problem.
How do you identify them?
Every chunk needs some kind of identity. Without one, there’s no way to store it, find it later, or determine whether you’ve already seen it before.
At first, the answer seems obvious.
Give every chunk a random ID.
Chunk 1 → 8f12d91aChunk 2 → b72e9c44Chunk 3 → 45aef981This works.
It’s also how many systems identify objects internally.
The storage engine generates a UUID, stores the chunk under that identifier, and remembers the mapping somewhere in its metadata database.
Simple.
Except the identifier doesn’t actually tell us anything.
It doesn’t describe the data.
It doesn’t help verify the data.
It doesn’t tell us whether another identical chunk already exists.
It’s just a label.
The data already contains an identifier
Instead of inventing an identifier, what if we derived one from the data itself?
Cryptographic hash functions do exactly that.
Given any amount of input data, they produce a fixed-size output.
PebbleFS uses SHA-256.
CHUNK
↓SHA-256↓1f40fc92da241694750979ee6cf582f2...// this module implements CAS (content addressable storage)
// the entire idea of this is that the content determines the address.
func Hash(data []byte) string {
sum := sha256.Sum256(data)
return hex.EncodeToString(sum[:])
}
//usage iin splitter :
data := make([]byte, n)
copy(data, buffer[:n])
chunkChan <- Chunk{
ID: Hash(data),
Index: index,
Data: data,
Size: int64(n),
}That hexadecimal string becomes the chunk’s identity.
Notice what’s happened here.
The storage engine didn’t assign a name.
The chunk generated one.
If two chunks contain exactly the same bytes, they’ll produce exactly the same hash.
If even a single bit changes, the hash changes completely.
The identifier is now a direct property of the data instead of an arbitrary value stored alongside it.
This idea is known as content-addressed storage, and it ends up influencing almost every other part of PebbleFS.
Why not UUIDs?
Suppose two users upload exactly the same Linux ISO.
Using UUIDs, the storage engine has no idea they’re identical.
Upload #1
↓UUID
↓a1c93...Upload #2↓UUID
↓7fd21...Two different identifiers.
Two complete copies stored on disk.
Twice the storage consumed.
The storage engine happily duplicates everything because, from its perspective, these are two unrelated objects.
Now imagine using SHA-256 instead.
Both uploads are split into the same chunks.
Every chunk produces exactly the same hash.
PebbleFS computes the hash, checks whether that chunk already exists, and immediately discovers that it does.
There is nothing to write.
The metadata simply references the existing chunk.
Without adding any extra logic, we’ve implemented deduplication.
The chunk already existed.
There was no reason to store it again.
Deduplication falls out naturally
One of the nice things about content-addressed storage is that deduplication isn’t really a separate feature.
It’s simply a consequence of how data is identified.
Imagine two files.
backup-1.tar
↓Chunk A
Chunk B
Chunk Cand
backup-2.tar
↓Chunk A
Chunk B
Chunk CWith randomly generated identifiers, PebbleFS stores six chunks.
With content addressing, it stores three.
Both files point to the exact same data.
The storage engine doesn’t care which file created the chunk first.
It only cares whether a chunk with that hash already exists.
If it does, the upload skips the write.
As datasets grow larger, this becomes surprisingly powerful.
Backups.
Container images.
Virtual machine snapshots.
Build artifacts.
Many of these contain enormous amounts of repeated data.
Content addressing allows PebbleFS to reuse existing chunks automatically.
The hash verifies the data
Storage devices don’t always fail dramatically.Sometimes they fail quietly.
A single bit flips.
A memory error corrupts data before it’s written.
A failing drive returns incorrect bytes.
The file still exists.
It’s just wrong.
Without verification, PebbleFS has no way of knowing.
Content addressing changes that.
Suppose PebbleFS reads a chunk from disk.
Instead of blindly trusting it, the storage engine computes the SHA-256 hash again.
Read Chunk↓SHA-256↓Expected Hash?If the newly computed hash matches the stored hash, the chunk is identical to the one originally written.
If the hashes differ, the data has changed.
Somewhere between writing and reading, corruption occurred.
The beauty is that no separate checksum needs to be stored.
The identifier already serves that purpose.
Every read operation implicitly verifies integrity.
Why SHA-256?
There are many cryptographic hash functions.
SHA-1.
SHA-256.
SHA-512.
BLAKE2.
BLAKE3.
So why SHA-256?
The reason is that it’s extremely well understood.
It’s widely implemented, thoroughly analyzed, and supported almost everywhere.
For PebbleFS, hashing isn’t the bottleneck.
Storage devices are almost always slower than computing a SHA-256 hash.
Choosing a mature algorithm is worth far more than saving a few microseconds during hashing.
But what about hash collisions?
Whenever someone hears that a hash identifies data, they eventually ask the same question.
“What if two different chunks produce the same hash?”
Technically, it’s possible.
SHA-256 maps an effectively infinite number of possible inputs into a 256-bit output.
Mathematically, collisions must exist.
Practically, they don’t matter.
The probability of accidentally generating a collision is so small that you’re vastly more likely to experience hardware failures, cosmic-ray-induced memory corruption, or several disks dying simultaneously than encountering two unrelated chunks with the same SHA-256 hash.
For a storage engine, SHA-256 is effectively unique.
Systems far larger than PebbleFS rely on the same assumption.
Git.
Docker.
Many content-addressable systems depend on cryptographic hashes as object identifiers.
Part 4: Metadata — Teaching PebbleFS What a File Is
At this point, PebbleFS has learned how to split files into chunks and assign every chunk a deterministic identity using SHA-256. From the storage engine’s perspective, everything is working perfectly. Chunks are being written, duplicates are eliminated automatically, and every piece of data has a unique identity.
The problem is that we’ve accidentally destroyed the concept of a file.
Imagine uploading a video called vacation.mp4. After chunking and hashing, PebbleFS is left with something that looks like this:
8f13a...
4b8d2...
91cd7...
2ef19...Those hashes exist somewhere on different storage devices, but nothing tells us that they belong to vacation.mp4. Nothing says which chunk comes first. Nothing records the original file size. Nothing even says whether another file shares some of these chunks.
We’ve built a system that’s very good at storing chunks, but we haven’t built one that knows how to reconstruct a file.
That’s where metadata comes in.
Data and metadata are different things
One of the biggest architectural decisions in PebbleFS was separating data from metadata. The actual bytes always live on storage devices, while everything describing those bytes lives in a metadata database.
That distinction is important because they change at completely different rates. A chunk rarely changes after it’s written, while metadata changes constantly. A new upload creates metadata, deleting a file removes metadata, moving chunks between devices updates metadata, and adding another storage device creates new metadata.
The bytes themselves usually stay exactly where they are.
Keeping these concerns separate makes the storage layer much simpler. Chunks don’t know anything about files. They don’t know whether they’re shared, which device they’re on, or whether anyone is still using them. They’re simply immutable blobs of data.
All the intelligence lives somewhere else.
A file becomes a recipe
Instead of storing a file as one continuous object, PebbleFS stores enough information to rebuild it.
Imagine a file split into four chunks:
vacation.mp4↓Chunk A
Chunk B
Chunk C
Chunk D
PebbleFS doesn’t store the file itself. Instead, it stores something conceptually similar to this:
Filename:
vacation.mp4Size:
13,824,512 bytesChunks:0 → a18d...
1 → b62f...
2 → c91e...
3 → d47c...
Notice what’s missing.
There are no bytes here.
The metadata simply says: “If someone asks for this file, read these chunks in this order.”
The file is reconstructed on demand. That means the original file doesn’t physically exist anymore. Only its chunks do.
The file is recreated every time it’s read.
Order matters
Suppose two files contain exactly the same chunks:
File AA
B
CFile BC
B
AThe chunks are identical, but the files are not.
Without recording chunk positions, PebbleFS would have no way to distinguish between them. The storage engine therefore stores both the chunk hash and its position inside the file.
Conceptually, it looks like this:
0 → a18d...
1 → b62f...
2 → c91e...
3 → d47c...Reconstruction becomes extremely simple. Read position zero, then position one, position two, and continue until the final chunk. Concatenate the bytes and return the stream.
There is no guessing, scanning, or searching involved. Everything required to reconstruct the file is already described by metadata.
Metadata is tiny
One thing I found interesting while implementing PebbleFS is just how small metadata actually is.
Suppose someone uploads a 40 GB file. The data occupies 40 GB, but the metadata describing that file is usually only a few kilobytes.
Even though PebbleFS might eventually be storing terabytes of user data, the metadata database remains comparatively small.
That’s useful for several reasons. Backing up metadata becomes inexpensive, searching metadata remains fast, and updating metadata doesn’t involve touching the underlying chunks.
Most importantly, rebuilding the storage engine after a restart only requires loading the metadata database. The bulk of the storage remains untouched.
Choosing SQLite
Once I decided that metadata needed its own database, the obvious question became: which database?
The usual answers came to mind: PostgreSQL, MySQL, MongoDB, or some distributed key-value store.
I ended up choosing SQLite.
Not because it’s trendy. Not because it’s lightweight.
Because it fits the problem.
PebbleFS isn’t serving thousands of concurrent clients. The metadata layer performs mostly transactional operations: inserting files, inserting chunk mappings, updating reference counts, and reading chunk lists.
SQLite excels at exactly this kind of workload.
It’s battle-tested, supports transactions, requires no separate server, and turns the entire metadata layer into a single portable file. That simplicity is difficult to ignore.
Could PebbleFS eventually move to PostgreSQL? Certainly. The metadata layer is abstracted behind interfaces specifically so that changing databases doesn’t require redesigning the storage engine.
For now, SQLite keeps the architecture simple while still providing ACID guarantees.
Designing the schema
The easiest mistake would have been storing everything inside one enormous table. Instead, I modeled the relationships between the different pieces of the system.
Files reference chunks. Chunks exist independently. Storage devices store chunks. Those relationships naturally became separate tables.
At a high level, the schema looks like this:
files
│
│
▼
file_chunks
│
▼
chunks
│
▼
chunk_locations
│
▼
storage_devicesEach table has exactly one responsibility.
The files table stores information users recognize, such as the filename, size, and creation time.
The chunks table stores immutable chunk objects, including their hash, size, and reference count.
The file_chunks table connects files to chunks while preserving their order.
Finally, storage_devices and chunk_locations describe where each chunk physically lives.
Breaking the schema into relationships rather than giant records makes almost every operation easier.
Why relationships matter
Imagine deleting a file.
If every chunk belonged exclusively to one file, deletion would simply remove those chunks. But deduplication changed everything.
Suppose two files share the same chunk:
File A↓Chunk XFile B↓Chunk X
Deleting File A should not delete Chunk X because File B still depends on it.
The metadata layer solves this by keeping track of relationships rather than ownership.
Chunks exist independently. Files reference chunks. Removing a file only removes its references. The chunk itself remains until nothing points to it anymore.
That small distinction becomes the foundation for garbage collection later in the project.
Transactions matter more than speed
One thing I refused to compromise on was consistency.
Imagine uploading a file. PebbleFS successfully writes every chunk to storage, but then the system crashes before updating the metadata.
Now the chunks exist, but the metadata doesn’t.
Those chunks are effectively lost because nothing references them.
The opposite failure is equally problematic. Suppose metadata is written first and the process crashes before writing half the chunks.
Now the metadata references objects that don’t exist.
Both situations leave the storage engine inconsistent.
The solution is transactions.
Metadata changes either happen completely or not at all. PebbleFS never commits half a file. Either the upload succeeds and every relationship is recorded, or the upload fails and the metadata transaction is rolled back.
Consistency is worth far more than squeezing out a few extra writes per second.
Metadata drives the storage engine
By the time I finished implementing the metadata layer, I realized something: almost every operation inside PebbleFS starts with a metadata lookup.
Uploading a file means checking whether chunks already exist. Downloading a file means reading the chunk list. Deleting a file means removing its chunk references. Moving data means updating chunk locations. Adding another storage device means registering it in metadata.
The storage layer mostly moves bytes.
The metadata layer decides what those bytes mean.
Without metadata, PebbleFS is just a collection of hashed files scattered across different disks.
Metadata is what turns those independent chunks back into meaningful files.
Part 5: The Storage Layer
By this point, PebbleFS has learned quite a lot. It knows how to split files into chunks, how to uniquely identify every chunk using SHA-256, and how to reconstruct files through metadata. What it still doesn’t know is where those chunks should actually be written.
Until now, storage has been this vague idea of “some disk.” That’s no longer enough.
PebbleFS needs a way to discover storage devices, understand their capabilities, keep track of available capacity, and provide a consistent interface for reading and writing chunks regardless of what hardware sits underneath.
This is where the storage layer begins.
Everything starts with Linux
One design decision influenced almost every other part of PebbleFS: I didn’t want to talk to disks directly.
Linux already knows how to do that.
It knows how to communicate with SATA controllers, understands USB mass storage, and can mount ext4, XFS, Btrfs, exFAT, NTFS, NFS, and dozens of other filesystems.
Reimplementing any of that would mean rebuilding decades of work that the kernel already does exceptionally well. PebbleFS doesn’t gain anything from that.
Instead, it simply asks one question:
Has Linux mounted this storage somewhere?
If the answer is yes, PebbleFS can use it.
That means these are all perfectly valid storage locations:
/mnt/ssd/mnt/archive/mnt/usb/mnt/server-storage
PebbleFS doesn’t care whether one path points to an NVMe SSD and another points to a slow USB flash drive.
To the storage engine, they’re simply directories where chunks can be stored.
Storage becomes an abstraction
The more I thought about it, the more obvious it became that PebbleFS should never know how data is stored.
It should only know where it can store data.
That sounds like a subtle distinction, but it led to one of the core abstractions in the project.
Instead of writing directly to the filesystem everywhere in the codebase, PebbleFS talks to a storage backend.
Conceptually, every backend exposes the same operations:
type Backend interface {
Write(hash string, data []byte) error
Read(hash string) ([]byte, error)
Delete(hash string) error
Exists(hash string) bool
}The implementation behind those methods can vary.
Today, they’re backed by directories on a Linux filesystem.
Tomorrow, they could be backed by S3, NFS, or even another PebbleFS node.
The storage engine shouldn’t care.
As long as a backend knows how to store and retrieve chunks, everything above it continues to work exactly the same way.
This is one of the biggest advantages of building around interfaces instead of concrete implementations.
What is a storage device?
Internally, PebbleFS represents every storage location as a device.
A device isn’t necessarily a physical disk. It’s simply somewhere that chunks can live.
That distinction matters because multiple devices might exist on the same physical drive, and one device might even represent remote storage over the network.
Each device stores information such as its ID, mount path, total capacity, available capacity, and current status.
For example:
ID: ssd-01Path:
/mnt/ssdCapacity:
512 GBFree:
187 GBStatus:
Online
PebbleFS doesn’t care how Linux arrived at those numbers. It simply records them so placement decisions can be made later.
Organizing chunk storage
Earlier, we saw that chunks are named by their SHA-256 hash.
The next question is where those chunk files should actually be placed on disk.
The simplest approach would be storing everything directly under one directory:
chunks/ab123...91ef0...3dc91...
That works until the number of chunks becomes very large.
Most filesystems don’t appreciate directories containing millions of files.
PebbleFS instead shards chunks using the beginning of the hash.
A chunk whose hash begins with 7f ends up looking something like this:
/mnt/ssd/chunks/7f/7f4a09bc...
Another chunk beginning with 2a is stored separately:
/mnt/hdd/chunks/2a/2af17e91...
The hash itself determines the directory.
No additional bookkeeping is required.
The distribution is naturally even because SHA-256 outputs are uniformly distributed.
Storage shouldn’t know about files
One rule I kept coming back to during development was this:
The storage layer should never know what a file is.
Its responsibility ends at chunks.
When PebbleFS asks a backend to write data, it doesn’t say:
Store
movie.mp4.
It says:
Store the chunk whose hash is
7f4a09....
That’s it.
The backend has no knowledge of filenames, metadata, or chunk ordering. It simply writes bytes to the correct location.
This separation is intentional.
It keeps the storage layer extremely small, while most of the interesting logic remains in the layers above it.
The Storage Manager
Once multiple devices exist, something needs to coordinate them.
Rather than letting the rest of the system interact with devices directly, PebbleFS introduces another layer called the Storage Manager.
The Storage Manager owns every registered device:
Storage Manager┌─────────┬─────────┬─────────┐ │ │ │ SSD HDD USB Drive
Whenever another component needs to read or write a chunk, it doesn’t interact with a device directly. It talks to the manager.
The manager is responsible for finding devices, checking whether they’re online, and exposing a consistent API to the rest of the storage engine.
That means the metadata layer never needs to know whether a chunk lives on an SSD or a USB drive.
It simply asks the Storage Manager to retrieve it.
Registering devices
Every storage device must first be registered before PebbleFS can use it.
During registration, the manager records basic information about the device. Conceptually, the process looks like this:
Storage Path↓Validate↓Read Capacity↓Assign Device ID↓Store Metadata↓Ready
Once registration completes, the device becomes available for placement decisions.
From that point onward, new chunks may be written there.
Likewise, removing a device simply updates its state inside the manager. Future placement decisions ignore it, while existing chunks remain recorded in metadata until they’re migrated or recovered.
Health matters
Not every storage device behaves the same way.
A fast NVMe SSD might sustain several gigabytes per second. An old USB 2.0 flash drive might struggle to reach 20 MB/s.
Some devices may disappear unexpectedly. Others may become read-only. Some may simply run out of space.
The Storage Manager keeps track of these states so the rest of PebbleFS doesn’t have to.
Instead of asking:
Can I write to
/mnt/usb?
the placement layer asks:
Give me an available storage device.
The manager handles everything else.
Keeping those responsibilities separate makes both components significantly easier to reason about.
Preparing for the future
Although PebbleFS currently stores chunks on mounted Linux directories, the storage layer was intentionally designed to avoid depending on local disks.
Nothing prevents another backend from implementing the same interface.
It could be an object store, a remote node, cloud storage, or even another PebbleFS cluster.
The storage engine wouldn’t need to change.
Only the backend would.
That’s one of the reasons I spent extra time designing this abstraction before adding more features.
Distributed systems tend to grow in unexpected directions. Good abstractions make that growth manageable. Poor ones force rewrites.
Part 6: Deciding Where Data Lives
At this point, PebbleFS has everything it needs to store data. Files are broken into chunks, every chunk has a deterministic identity, metadata keeps track of how files are reconstructed, and the storage layer knows about every available storage device.
There’s only one question left before a chunk can be written:
Which device should receive it?
At first, this doesn’t sound like a difficult problem. Just pick a disk, write the chunk, and move on.
That works for exactly one upload.
Get berzzWrites’s stories in your inbox
Join Medium for free to get updates from this writer.
After a few thousand uploads, it starts falling apart.
The first idea
Suppose the storage pool contains three devices:
SSDHDDUSB
A new chunk arrives, and the easiest implementation is choosing one device at random.
Chunk↓Random Device↓Write
That’s only a few lines of code.
It also has one major flaw: randomness has no memory.
It doesn’t know that one device is already almost full. It doesn’t know that another device is much faster. It doesn’t know that a third device disappeared five minutes ago.
Random selection is easy to implement because it ignores information.
Unfortunately, that information matters.
Round robin isn’t much better
The next obvious solution is round robin:
Chunk 1 → SSDChunk 2 → HDDChunk 3 → USBChunk 4 → SSD
Every device receives approximately the same number of chunks.
For identical disks, this actually works surprisingly well. The problem is that storage devices are almost never identical.
Imagine these capacities:
SSD
500 GBHDD
4 TBUSB
32 GB
Round robin happily gives every device the same amount of data. The USB drive fills almost immediately while the hard drive still has terabytes available.
Treating every device equally only works when every device is actually equal.
PebbleFS is built around heterogeneous storage, so equal distribution doesn’t necessarily produce balanced storage.
Capacity matters
Instead of asking:
Which device received the previous chunk?
the storage engine should probably ask:
Which device has room for another one?
That immediately produces better results.
A device with hundreds of gigabytes free should receive more chunks than one that’s nearly full. Large storage devices naturally end up storing more data, while small devices remain useful without becoming bottlenecks.
This sounds obvious, but it’s an important shift in thinking.
Placement decisions should be based on the current state of the storage pool rather than a fixed algorithm.
Hardcoding isn’t scalable
While experimenting with placement algorithms, I kept running into the same problem.
Every new strategy required changing the storage engine itself.
If I wanted to test a different algorithm, I had to modify existing code.
That’s never a good sign.
Eventually, I realized that placement shouldn’t be part of the storage layer at all. It should be its own abstraction.
Instead of asking the Storage Manager to decide where chunks go, PebbleFS asks a placement strategy.
Conceptually, every strategy implements the same interface:
type Strategy interface {
Select(devices []*Device) (*Device, error)
}The storage engine doesn’t know which implementation it’s using.
It simply provides a list of available devices.
The strategy chooses one.
Everything else stays exactly the same.
Why an interface?
This might seem like overengineering for a small project.
It isn’t.
Distributed storage systems evolve constantly. Today’s placement algorithm may be completely different from tomorrow’s.
If placement logic is scattered throughout the storage engine, changing algorithms becomes painful.
If every algorithm satisfies the same interface, experimenting becomes trivial.
Replace one implementation. Run benchmarks. Measure results.
Nothing else changes.
The storage engine doesn’t even notice.
That’s exactly the kind of flexibility interfaces are designed to provide.
The first implementation
PebbleFS currently keeps the placement strategy intentionally simple.
When a chunk needs to be written, the strategy examines every available storage device. Offline devices are ignored, and devices without enough free space are ignored. From the remaining candidates, one is selected according to the configured strategy.
The chosen device is then returned to the Storage Manager, which performs the actual write.
The important point is that placement makes decisions.
Storage performs writes.
Those responsibilities never overlap.
Future strategies
One reason I wanted placement behind an interface is that there isn’t a universally correct algorithm. Different workloads benefit from different strategies.
For example, a homelab might simply want to maximize available space by choosing the device with the most free capacity.
A system optimized for performance might prefer SSDs until they reach a certain utilization.
Another system might distribute writes proportionally based on device capacity.
Clusters spanning multiple machines might avoid placing replicas on the same host.
Enterprise storage systems often consider rack awareness, network latency, failure domains, disk health, and historical performance before deciding where data should live.
All of those strategies solve the same problem. They simply optimize for different goals.
PebbleFS shouldn’t force one policy on everyone.
It should provide a framework where different policies can exist.
Placement isn’t permanent
Choosing a storage device doesn’t mean the chunk stays there forever.
That’s another misconception I had early on.
Placement decides where data starts its life. It doesn’t necessarily decide where it spends the rest of it.
Imagine adding another 8 TB drive to the storage pool six months later. Some chunks may eventually migrate there.
The original placement decision wasn’t wrong.
The storage pool simply changed.
Future versions of PebbleFS will include background rebalancing that continuously improves data distribution as devices are added or removed.
Placement handles new writes.
Rebalancing handles existing data.
They’re related problems, but they’re not the same problem.
Every write follows the same pipeline
By now, the upload path has become much more structured.
When a new file enters PebbleFS, every chunk follows roughly the same journey:
Read Chunk↓Compute SHA-256↓Already Exists?↓Yes ─────────────┐
│
No │
│ │
▼ │
Placement │
│ │
▼ │
Storage Manager │
│ │
▼ │
Write Chunk │
└────────────────┘Update Metadata
Every component has a single responsibility.
The chunker divides files. The hasher identifies data. The metadata layer tracks relationships. The placement strategy chooses a destination. The Storage Manager performs the write.
None of these components know how the others work internally.
That separation makes the system easier to extend and much easier to debug.
Part 7: The Write Pipeline
Up until now, we’ve been looking at the individual pieces that make up PebbleFS. We’ve talked about chunking, content-addressing, metadata, storage devices, and placement.
On their own, none of these components are particularly useful.
The interesting part is how they work together.
When someone uploads a file, PebbleFS doesn’t simply copy it to a disk. The file passes through a series of stages, with each stage responsible for solving one specific problem before handing the data to the next component.
Understanding this pipeline is probably the easiest way to understand how the storage engine actually works.
Step 1: Reading the file
Everything begins with a file on disk:
movie.mp4PebbleFS opens the file as a stream of bytes.
One thing I wanted to avoid from the beginning was reading the entire file into memory. That might work for a few megabytes, but it quickly falls apart for files that are tens or hundreds of gigabytes.
Instead, the file is processed incrementally. PebbleFS reads one chunk at a time, processes it, and immediately moves on to the next one.
Memory usage therefore remains almost constant regardless of how large the uploaded file is.
Whether the upload is 10 MB or 100 GB, the storage engine only needs enough memory to process a single chunk.
Step 2: Chunking
As bytes are read from the file, they’re grouped into fixed-size chunks.
Suppose the configured chunk size is 4 MB. A 10 MB file becomes three chunks:
movie.mp4↓Chunk 1 (4 MB)↓Chunk 2 (4 MB)↓Chunk 3 (2 MB)
Each chunk is treated completely independently from this point onward.
PebbleFS no longer thinks about the original file.
Everything happens one chunk at a time.
Step 3: Computing the hash
Before a chunk is written anywhere, PebbleFS computes its SHA-256 hash:
Chunk↓SHA-256↓a8f913...
This hash immediately becomes the chunk’s identity.
Every operation from this point onward refers to the chunk by its hash rather than by its contents.
That single decision simplifies almost every part of the upload process.
Step 4: Checking for duplicates
This is the first point where metadata becomes involved.
Once the hash is known, PebbleFS asks a simple question:
Does this chunk already exist?
The metadata database already contains every known chunk hash.
If the hash is found, there’s nothing to write. The storage engine already has those exact bytes.
Instead of writing another copy, PebbleFS simply records that the current file references the existing chunk:
Chunk↓SHA-256↓Exists?↓Yes↓Reuse existing chunk
This entire optimization happens before any storage device is touched.
There are no disk writes, no duplicate data, and no unnecessary movement of bytes. The only thing that changes is metadata.
Step 5: Choosing a destination
If the chunk doesn’t already exist, PebbleFS now has to decide where it should live.
This responsibility belongs entirely to the placement strategy.
The placement layer examines every available storage device. Offline devices are ignored, devices without sufficient free space are ignored, and the remaining candidates are evaluated according to the configured strategy.
Eventually, one device is selected:
Chunk↓Placement↓SSD
Notice that the placement layer never performs the write itself.
It only makes the decision.
Actually writing the chunk is someone else’s responsibility.
Step 6: Writing the chunk
Once a destination has been chosen, the chunk is handed to the Storage Manager.
The Storage Manager determines the final path where the chunk should be stored.
For example:
/mnt/ssd/chunks/a8/a8f913...The required directories are created if they don’t already exist, and the chunk is then written as a normal file.
Nothing particularly clever happens here.
Linux handles buffering, filesystem allocation, and block management.
PebbleFS simply writes a file.
Keeping the storage layer this simple was intentional.
The complexity belongs in the architecture, not in individual write operations.
Step 7: Updating metadata
Only after the chunk has been successfully written does PebbleFS update the metadata database.
Several things happen here. The chunk is inserted into the chunk table, its storage location is recorded, the file receives another chunk reference, and the chunk’s reference count is initialized or updated.
All of these operations occur inside a single transaction.
Either every metadata change succeeds together, or none of them do.
That prevents partially uploaded files from appearing inside the storage engine.
Consistency is far more valuable than squeezing out a few extra writes per second.
One chunk at a time
One thing I like about this pipeline is that every chunk follows exactly the same path.
It doesn’t matter whether it’s the first chunk of a video or the last chunk of a database backup.
Every chunk goes through the same stages:
Read↓Chunk↓Hash↓Already Exists?↓Placement↓Write↓Metadata
There are no special cases.
There are no different code paths for different file types or separate logic for images and videos.
Every upload becomes a repetition of the same sequence.
That predictability makes the system significantly easier to reason about.
What if something fails?
Uploads don’t always succeed.
A storage device might disappear halfway through a write. The system might lose power. SQLite might fail to commit a transaction.
Those failures have to be handled carefully.
Suppose a chunk is written successfully, but the metadata transaction fails.
The chunk now exists on disk, but nothing references it.
From PebbleFS’s perspective, it’s orphaned.
The opposite situation is even worse.
Suppose metadata is committed first and the write operation fails afterward.
Now the metadata references a chunk that doesn’t exist.
Neither situation is acceptable.
That’s why PebbleFS treats writing data and updating metadata as two closely coordinated operations.
The write isn’t considered complete until both have succeeded.
Maintaining consistency is one of the primary responsibilities of the write pipeline.
Why the pipeline is modular
If you look closely, every stage in the upload pipeline solves exactly one problem.
The chunker doesn’t know anything about metadata.
The metadata layer doesn’t know anything about storage devices.
The placement strategy doesn’t know how files are reconstructed.
The Storage Manager doesn’t know what a filename is.
Every component does one job.
That modularity wasn’t accidental.
It means each piece can evolve independently.
I can replace the placement algorithm without touching the metadata layer. I can move from SQLite to PostgreSQL without changing how chunks are written. I can add an S3 storage backend without rewriting the upload pipeline.
Each component communicates through well-defined interfaces.
That keeps the architecture flexible while preventing individual modules from becoming tightly coupled.
Part 8: Reading Data
Uploading a file is relatively straightforward. PebbleFS receives a stream of bytes, breaks it into chunks, stores those chunks, and records enough metadata to reconstruct the file later.
Reading works in the opposite direction.
The file no longer exists as one continuous object. All that remains are immutable chunks distributed across one or more storage devices.
When someone asks for a file, PebbleFS has to put all of those pieces back together without the user ever noticing they were separated in the first place.
Finding the file
Everything starts with metadata.
Suppose someone requests a file named movie.mp4. The first step isn't opening a storage device. It's querying the metadata database.
PebbleFS needs to answer a few basic questions: Does the file exist? How large is it? Which chunks belong to it? And in what order should those chunks be read?
Conceptually, the metadata might look like this:
movie.mp4Size:
13.8 MBChunks0 → a81fd...1 → c91ab...2 → 7f209...3 → d18ec...
Notice that no storage devices have been accessed yet.
Everything begins with metadata.
Without it, PebbleFS has no idea how to rebuild the file.
Locating every chunk
Knowing the chunk hashes isn’t enough.
PebbleFS also needs to know where those chunks live. That information comes from another metadata lookup.
For example:
a81fd...↓SSDc91ab...↓HDD7f209...↓USBd18ec...↓SSD
Each chunk may reside on a completely different storage device.
The storage engine doesn’t assume anything about locality. As long as metadata knows where each chunk is stored, reconstruction remains possible.
This is another reason separating metadata from storage was such an important architectural decision.
Finding data becomes a database lookup instead of a filesystem search.
Reading chunks
Once the locations are known, PebbleFS begins reading chunks one at a time.
The Storage Manager receives the request and determines which backend owns the chunk. The backend opens the corresponding file and returns its contents.
Conceptually, the process looks like this:
Chunk Hash↓Storage Manager↓Backend↓Read File↓Bytes
The Storage Manager doesn’t know which file is being reconstructed. The backend doesn’t know either.
They’re simply asked to retrieve a chunk identified by its hash.
Just like during uploads, every component stays focused on one responsibility.
Rebuilding the original file
At this point, PebbleFS has the individual chunks.
The only remaining task is assembling them in the correct order.
This is surprisingly simple.
PebbleFS reads the first chunk and writes its bytes to the output stream. It then reads the second chunk and appends those bytes. The process continues until every chunk has been processed.
Conceptually:
Chunk A+Chunk B+Chunk C↓Original File
The resulting byte stream is identical to the one originally uploaded.
From the user’s perspective, nothing unusual happened.
They requested a file.
They received a file.
The fact that PebbleFS rebuilt it from dozens or hundreds of independent chunks remains completely hidden.
Streaming instead of buffering
One design choice I made early on was avoiding large memory allocations during reads.
A naive implementation might read every chunk into memory before returning anything to the user. That works for small files, but it becomes impractical for multi-gigabyte uploads.
Instead, PebbleFS streams data as it reconstructs the file.
Each chunk is read and immediately written to the output. Once that chunk has been consumed, its memory can be released before the next chunk is processed.
Memory usage therefore remains almost constant regardless of file size.
Whether someone downloads a 5 MB image or a 100 GB backup archive, PebbleFS only needs enough memory to process a single chunk at a time.
That makes the storage engine much more predictable under heavy workloads.
Integrity verification
Earlier, we saw that every chunk is identified by the SHA-256 hash of its contents.
That identity also gives us a convenient integrity check.
Whenever a chunk is read from storage, PebbleFS can compute its hash again:
Read Chunk↓SHA-256↓Matches Metadata?
If the hashes match, the chunk is exactly the same as when it was written.
If they don’t, something modified the data.
Today, PebbleFS simply reports that error because there isn’t another copy of the chunk available.
Once replication is implemented, the storage engine will be able to retrieve another replica automatically and replace the corrupted copy in the background.
The integrity mechanism already exists.
Replication will simply make it useful.
Missing chunks
Another failure scenario is a missing chunk.
Perhaps a storage device was unplugged. Perhaps a file was accidentally deleted outside PebbleFS. Perhaps a disk failed completely.
If metadata references a chunk that can’t be found, reconstruction stops immediately.
The file is incomplete.
At the current stage of the project, PebbleFS cannot recover from that situation because every chunk exists in only one location.
That’s a deliberate limitation rather than an oversight.
Recovery mechanisms depend on replication, and replication hasn’t been implemented yet.
Rather than pretending failures don’t exist, PebbleFS exposes them honestly.
Supporting recovery will come later without requiring changes to the read pipeline itself.
Why reconstruction is inexpensive
One thing that surprised me while building PebbleFS is how little work reconstruction actually performs.
The storage engine isn’t decoding anything. It’s not decompressing data. It’s not rebuilding complex structures.
It’s simply concatenating byte streams.
The heavy lifting happened during upload.
Once metadata knows which chunks belong together, rebuilding the original file becomes little more than reading files in sequence.
That’s one of the advantages of keeping chunks immutable.
Nothing needs to be transformed before it’s returned.
The bytes written during upload are exactly the bytes returned during download.
The complete read path
By now, the entire read operation has a clear structure:
Filename↓Metadata Lookup↓Chunk List↓Chunk Locations↓Storage Manager↓Read Chunks↓Reassemble↓Return Stream
Every stage depends on the previous one.
Metadata determines what should be read.
The Storage Manager determines how it should be read.
The reconstruction layer determines how the chunks should be combined.
No component needs to understand the entire process.
Each one solves a small piece of the problem.
Part 9: Deleting Data
Writing data is relatively easy. Reading data is mostly a matter of following metadata. Deleting data, however, is where content-addressed storage starts behaving very differently from a traditional filesystem.
On most filesystems, deleting a file is straightforward. The filesystem removes the directory entry, marks its blocks as free, and eventually those blocks get reused. The file is gone.
PebbleFS can’t do that.
Not because deletion is difficult, but because the storage engine no longer owns chunks in the same way a traditional filesystem owns blocks.
Chunks can be shared.
Deleting one file doesn’t necessarily mean deleting its data.
The problem with shared chunks
Imagine two files:
ubuntu-24.04.iso↓Chunk A
Chunk B
Chunk C
and:
ubuntu-copy.iso↓Chunk A
Chunk B
Chunk C
Because PebbleFS uses content-addressed storage, only one copy of each chunk exists. Both files simply reference the same three chunks.
Now suppose the first file is deleted.
Should Chunk A be deleted?
No.
The second file still needs it.
From the storage engine’s perspective, deleting that chunk would silently corrupt another file. This is something a traditional filesystem never has to think about because blocks usually belong to exactly one file.
PebbleFS deliberately broke that assumption when it introduced deduplication.
Now it has to solve a different problem.
Ownership doesn’t exist anymore
One of the biggest conceptual shifts in PebbleFS is that chunks don’t belong to files.
Files reference chunks.
That sounds like a minor wording change, but it completely changes how deletion works.
Imagine this metadata:
movie.mp4↓Chunk XChunk Y
Deleting the file doesn’t delete Chunk X.
It only removes the relationship between the file and the chunk.
The chunk continues to exist independently. Whether it should remain on disk depends entirely on whether something else still references it.
The storage engine isn’t really deleting data.
It’s deleting references.
Reference counting
The simplest way to answer whether a chunk is still needed is to keep track of how many files reference it.
PebbleFS stores a reference count alongside each chunk.
Conceptually, it looks something like this:
Hasha81fd...References3
Every new file that uses the chunk increments the counter. Every deleted file decrements it.
Eventually, one of two things happens.
If the reference count is still greater than zero, the chunk remains on disk because at least one file still depends on it.
If the reference count reaches zero, PebbleFS knows that nothing in the system references those bytes anymore.
Only then can the chunk be safely removed.
Deletion becomes a metadata operation
Notice something interesting.
The first part of deletion doesn’t touch storage at all.
The storage engine begins by updating metadata. It removes the file record, removes every file-to-chunk mapping, and decreases the reference count for every affected chunk.
Only after those operations complete does PebbleFS even consider deleting physical data.
That makes deletion surprisingly inexpensive.
Most of the work happens inside SQLite.
Actual chunk files are only removed if they’re truly unused.
Immediate deletion or garbage collection?
Once a chunk’s reference count reaches zero, there are two possible approaches.
The first is deleting it immediately:
Reference Count1↓Delete File↓0↓Delete Chunk
This is simple.
The storage engine immediately reclaims disk space.
The second approach is delaying deletion. Instead of removing unused chunks right away, PebbleFS could simply mark them as eligible for cleanup. A background worker would periodically scan for unreferenced chunks and remove them.
This approach is known as garbage collection.
Many large storage systems prefer background cleanup because it keeps deletion operations fast while allowing cleanup work to happen independently.
PebbleFS currently performs cleanup immediately because the implementation is much simpler. As the project grows, background garbage collection may become a better choice.
What happens if deletion fails?
Failures don’t stop being interesting just because we’re deleting data.
Suppose PebbleFS removes the metadata successfully, but the process crashes before deleting the chunk file.
Now the chunk still exists on disk, but nothing references it.
It’s become an orphan.
Fortunately, this is one of the safer failure modes. The orphan consumes disk space, but it doesn’t corrupt data.
A future cleanup pass can safely remove it because the metadata already says nobody needs it.
The opposite failure is much worse.
Suppose PebbleFS deletes the chunk first and then the metadata update fails.
Now metadata still references data that no longer exists.
The next read operation fails because the chunk has disappeared.
That’s exactly why PebbleFS always updates metadata first.
Deleting an orphan is easy.
Recovering missing data isn’t.
The delete pipeline
By now, deletion follows a fairly predictable sequence:
File↓Find Chunk List↓Remove File Metadata↓Decrease Reference Counts↓Reference Count = 0?↓No ──────────────┐
│
Yes │
│ │
▼ │
Delete Chunk │
└────────────────┘
Like every other operation in PebbleFS, each stage has one responsibility.
Metadata determines which chunks are affected.
Reference counting determines whether they’re still needed.
The storage layer simply removes chunks when instructed.
No component needs to understand the entire process.
Part 10: Where PebbleFS Goes Next
At this point, PebbleFS is capable of storing files across multiple storage devices using content-addressed chunks.
Files are split into fixed-size pieces. Each chunk is identified by its SHA-256 hash. Metadata keeps track of how files are reconstructed. Storage devices are abstracted behind a common interface, and placement strategies determine where new chunks are written.
Uploads, downloads, and deletions all work as complete end-to-end operations.
That’s enough to call PebbleFS a functional storage engine.
It isn’t yet a distributed storage system in the way projects like Ceph or GlusterFS are, but the foundation is now in place.
More importantly, the architecture was designed with that future in mind.
Replication
Today, every chunk exists in exactly one location.
If that storage device disappears, the chunk disappears with it.
The next major addition will be replication.
Instead of writing a chunk to one storage device, PebbleFS will write multiple identical copies.
Chunk ┌───────────── SSD
│
──────┼───────────── HDD
│
└───────────── Remote NodeMetadata will record every replica, allowing the storage engine to continue serving data even if one device fails.
Replication also opens the door to automatic recovery.
If one copy becomes corrupted or disappears, PebbleFS can rebuild it from another replica without user intervention.
Smarter placement
The current placement strategy is intentionally simple.
It chooses an appropriate device from the available storage pool.
Future versions of PebbleFS will make those decisions much more intelligently.
Placement may consider storage capacity, device performance, health, replication requirements, or even whether two replicas are being placed on the same physical machine.
The architecture already supports this.
Only the strategy implementation needs to change.
Background workers
Right now, most operations happen synchronously.
Uploads write chunks immediately. Deletion removes unused chunks immediately.
As the storage engine grows, some of these tasks make more sense as background jobs.
Future workers may handle several responsibilities, including rebalancing data when new storage devices are added, repairing missing replicas, cleaning up orphaned chunks, verifying chunk integrity, and monitoring storage health.
Moving these tasks into background services keeps the critical path fast while allowing the storage engine to improve itself over time.
Beyond local storage
Although every example in this series has used mounted directories on a single Linux machine, nothing in PebbleFS fundamentally depends on local disks.
The storage layer already abstracts where chunks are stored.
That means another backend could store chunks on a remote machine just as easily as on a local SSD.
Eventually, multiple PebbleFS nodes will contribute storage to the same logical pool.
At that point, the project moves from a local storage engine to a genuinely distributed one.
Interestingly, very little of the architecture we’ve built so far needs to change.
Most of the work shifts from storage to networking.
Why I built PebbleFS
PebbleFS started with a simple observation.I had a collection of storage devices that individually weren’t very useful.What I really wanted wasn’t another filesystem.I wanted software that could treat all of those devices as one storage pool.
Along the way, the project became much more than that.It became an excuse to learn distributed systems by building one.Every concept in this series appeared because the previous design eventually stopped being good enough.
Chunking solved one problem and created another.Content addressing solved that problem and introduced metadata.Metadata required storage abstraction.
Storage abstraction required placement.Every decision naturally led to the next.Looking back, that’s probably my favorite part of the project.None of the architecture appeared fully formed.It evolved one problem at a time.
The road ahead
PebbleFS is still at the beginning of its journey.
There are better placement algorithms to explore, replication to implement, distributed metadata to experiment with, background repair workers to build, and eventually multiple machines cooperating as a single storage cluster.
But the hardest part is no longer writing code.It’s having an architecture that can grow without constantly being rewritten.
That was the goal from the beginning.Everything else is just another engineering problem waiting to be solved.
