How Discord Learned to Store Billions of Messages
A simple story about MongoDB’s growing pains, Discord’s real traffic,
why Cassandra was chosen, and why its writes are so fast.
Discord looks simple from the outside. We open a server, go to a channel, type a message, and it appears on the screen. But behind this small action, Discord has to save and read a huge number of messages every day.
This is the story of how Discord handled that problem. It started with MongoDB, faced serious speed issues, and then moved to Cassandra. I will keep the explanation simple and focus on the main ideas, not on deep database design.
First, What Is Discord?
A Discord server is like a large online community. One server can have many channels, such as general, memes, announcements, or study-help. People send messages inside these channels.
The most important thing to understand is how people read messages on Discord. A user normally opens one channel and asks for its latest messages. If the user scrolls up, Discord loads some older messages from the same channel.
So, Discord usually does not need to find every message from every server at once. Most of the time, the question is simple:
“Give me the latest 50 messages from this channel.”
This usage pattern became very important when Discord designed its new database system.
The Size of the Problem
Discord decided very early that it wanted to keep chat history forever. A message written years ago should still be available when someone opens the old conversation on another device.
This sounds useful, but it creates a big storage problem. The amount of data never becomes smaller. It only keeps growing.
In July 2016, Discord was receiving around 40 million messages every day. By December 2016, that number had reached 100 million messages per day. In January 2017, it was already more than 120 million messages per day.
That is about 1,389 new messages every second on average. During busy hours, the real number could be much higher. Discord also had almost the same amount of reading and writing. People were not only sending messages; they were also opening channels and reading them.
Discord needed a database that could keep growing without becoming slow or difficult to manage.
A Few Simple Database Ideas
Before looking at Discord’s old system, we need to understand a few basic ideas.
A database is a program that stores data and helps us find it quickly. We could save all messages in normal files, but finding one message inside billions of lines would be very slow. A database keeps the data in a more useful form.
Computers mainly use RAM and disk to work with data.
RAM is very fast, but it is expensive and limited. It also loses its data when the power goes off. A disk is slower, but it is cheaper, much larger, and keeps the data safely.
We can think of RAM as a study table and disk as a cupboard. A book on the table is easy to pick up. A book inside the cupboard takes more time because we have to get up, open the cupboard, and find it.
Databases also use indexes. An index works like the index at the back of a book. If we want to find a topic, we do not read every page. We look at the index and directly go to the correct page.
An index is fast when it fits in RAM. But as the index and data grow, some of them have to stay on disk. Then the database has to visit the disk more often, and the reply becomes slower.
Another important idea is a disk seek. A seek happens when the disk has to jump from one place to another to collect data. If 50 messages are kept in 50 different places, the disk may need many jumps. If those messages are kept together, the disk can read them in one smooth run. Reading nearby data is much faster than jumping around the disk.
The operating system also keeps recently used disk data in RAM. This is called a disk cache or page cache. If the same data is asked for again, it can be returned quickly from RAM.
But RAM has limited space. When old or rarely used data enters the cache, some useful data has to leave. If every new request asks for a different part of the disk, the cache keeps removing and loading data again and again. This is called cache thrashing. At that point, the cache is not helping much.
Discord’s First Database: MongoDB
Discord’s first version was built in about two months in early 2015. The team wanted to build quickly and find out whether people actually wanted the product. They did not want to spend months building a very large system before they even had users.
They stored everything in one MongoDB replica set. A replica set has one main copy of the data and some extra copies. If one machine fails, another copy can help keep the data safe.
Replication gives safety, but it does not automatically divide a growing data set across many machines. Every machine still keeps a copy of the complete data.
Discord used an index based on the channel ID and message creation time. This made it easy to ask for the newest messages from one channel.
Using MongoDB at the start was not a mistake. It helped a small team release the product quickly. The team also kept the code ready for a future database change. Their thinking was simple: first prove that the product is useful, but keep a clear path to a stronger system later.
They also chose not to use MongoDB sharding. Sharding means dividing data into parts and storing those parts on different machines. At that time, the team felt that MongoDB sharding was complicated to operate and was not stable enough for their needs.
What Went Wrong with MongoDB?
By November 2015, Discord had stored around 100 million messages. At this point, the data and its index could no longer fit inside RAM.
When the important data was in RAM, a request could get a quick answer. After the data became too large, more requests had to go to disk. Disk access was slower, and Discord’s reads were very random because different users were opening different channels at different times.
One request might be fast because its data was already in the cache. The next request might take much longer because it had to collect data from the disk. This made the response time unpredictable.
Unpredictable speed is a serious problem. A system that always takes a little longer can still be planned for. But a system that is fast now and suddenly hangs on the next request gives users a bad experience and is harder for engineers to understand.
The cache also started thrashing. A request for an old, inactive channel could load old data into RAM and push out recent, useful data. The next user would then need another disk read. This cycle kept repeating.
Discord knew that the problem would only become worse as more messages and users arrived. It was time to choose a new database.
Understanding the Traffic Before Choosing a Database
Discord did not begin by asking, “Which database is the most popular?” The team first studied how people were using the product.
This is a useful lesson. There is no database that is best for every app. We should first understand what kind of data we have, how often it changes, and how users ask for it.
Discord’s read and write traffic was close to 50/50. This was different from many websites where most work is reading and only a small part is writing.
For a mostly-read website, a cache such as Redis can solve many speed problems. Discord could not depend only on that idea because new messages were being written all the time. It needed a database that was very good at writing as well as reading.
The team found three common types of Discord servers.
The first type was a small gaming server that mostly used voice chat. It could have fewer than 1,000 text messages in a full year. This sounds easy, but those few messages might be spread across many places on the disk. Reading 50 of them could need many random disk seeks. Loading such old data could also remove more useful data from the cache.
The second type was a private text server for a group of friends. It might have fewer than 100 members but could create between 100,000 and one million messages in a year. Members normally wanted recent messages. However, the channel might not be opened often enough for its data to stay in the cache.
The third type was a large public server with thousands of members and millions of messages. Surprisingly, this was often the easiest case. Many people kept reading the same recent messages, so that data usually stayed in the cache.
This showed something interesting: the server with the most messages was not always the hardest one to support. A quiet server with old and scattered data could cause more random disk work.
Discord was also planning features such as mentions, pinned messages, and full-text search. These features would let users jump directly to older parts of a channel. That meant even more random reads in the future.
The new system had to handle both recent messages and sudden jumps into old history.
What Discord Needed from the New Database
After studying its traffic, Discord made a clear list of requirements.
1. It should scale by adding more machines
Get Ankit Raj’s stories in your inbox
Join Medium for free to get updates from this writer.
Discord did not want to keep buying one larger and more expensive machine. That method has a limit. At some point, one machine cannot be made any bigger.
The team wanted horizontal scaling. When the amount of data grew, they wanted to add another normal machine and increase the total capacity. They also did not want to manually divide and move all the old data every time a machine was added.
2. It should recover automatically when a machine fails
Machines can fail at any time. The new system had to keep working by using another copy of the data. The team had only some backend engineers, so they did not want an engineer to wake up at night and manually repair every failure.
3. It should need little maintenance
The database should work normally after it was set up. The main job should be adding more machines when the data grew, not tuning the system every week.
4. It should already be proven at a large scale
Discord liked new technology, but messages were too important for an unsafe experiment. The team wanted a database that large companies had already used successfully. Cassandra was already running on very large systems at companies such as Netflix and Apple.
5. Its speed should be predictable
Discord watched the slowest part of its traffic, not only the average speed. An alert was raised if 95 percent of API requests were not finishing within 80 milliseconds.
The team also wanted the database itself to be fast enough without putting a Redis or Memcached cache in front of all messages. A separate cache would add another system to manage. It could also create problems when a message was edited or deleted and the cached copy became old.
6. It should store each message separately
Discord did not want to save a whole channel as one large file or blob. Imagine that a channel is stored as one 50 MB file. To add one small message, the system might need to read the whole file, open its data, add the message, build the file again, and write all 50 MB back to disk.
That would be terrible when thousands of messages arrive every second. Discord wanted one new message to mean one small new write.
7. It should be open source
The team wanted control over an important part of its system. With an open-source database, engineers can study the code and fix a critical problem themselves. They are not fully dependent on one outside company, its price, or its future plans.
After checking these requirements, Discord found that Cassandra was the only database that matched all of them.
What Is Cassandra?
Cassandra is a distributed database. Instead of keeping all the work on one machine, it uses a group of machines called nodes.
All Cassandra nodes are equal. There is no single master node that controls everyone else. Any node can receive a request. This is called a masterless or peer-to-peer system.
We can imagine the nodes placed in a circle, which is often called a ring. Each node is responsible for one part of the data.
When a new message arrives, Cassandra uses a key, such as a channel ID, and changes it into a number using a hash function. That number tells Cassandra which node should keep the data.
This method is called consistent hashing. Its main benefit is that when a new node is added, Cassandra does not need to move every piece of data. Only some data moves to the new node. This makes it easier to grow the cluster.
Keeping Extra Copies of the Data
Cassandra can keep the same data on multiple nodes. This is called replication.
Discord used a replication factor of three. It means every message was kept on three different nodes.
If one node failed, two more copies were still available. Users could continue reading and sending messages while the failed machine was repaired or replaced.
The extra copies use more disk space, but they protect the service from a machine failure. For Discord, staying online was more important than saving that extra space.
Availability and Consistency
Distributed databases have to make difficult choices when machines cannot talk to each other because of a network problem.
Suppose two shops share the same stock information, but their phone connection breaks. One shop can stop selling until it confirms the latest stock with the other shop. This keeps the information fully correct, but customers cannot buy anything.
The other choice is to keep selling with the information already available. The shop stays open, but for a short time the two shops may show different stock numbers. They can fix the difference after the connection returns.
Cassandra prefers availability in this situation. It keeps accepting requests even when part of the network has a problem. This may mean that one user sees slightly older data for a short time.
This choice made sense for Discord. If a chat message appears a moment late, it is usually acceptable. If the whole chat service becomes unavailable, users are much more unhappy.
The correct choice depends on the business. A bank may prefer to stop a request rather than show an old account balance. A chat app can accept a tiny delay to stay online.
How Cassandra Keeps Related Data Together
A simple way to understand Cassandra is to think of boxes.
The first key chooses the box. For Discord, a channel can act like a box. The second key places messages in the correct order inside that box. The actual message is the value stored there.
This idea is sometimes called key-key-value, or KKV.
Keeping related messages together gives Cassandra an important advantage. When Discord asks for the latest 50 messages of one channel, Cassandra can read nearby data instead of jumping to 50 different places on the disk. The messages can also be kept in order, so the database does not need to sort a huge list every time.
The exact table design is a deeper topic. The main idea for now is enough: Cassandra spreads different groups of data across nodes, while keeping related data close together inside each group.
Why Are Cassandra Writes So Fast?
Cassandra is famous for fast writes. The reason is the way it accepts and stores new data. It uses an idea called an LSM Tree, which stands for Log-Structured Merge Tree.
The name sounds difficult, but the basic idea is simple.
When a new message comes in, Cassandra first adds it to a commit log on the disk. It adds the new data at the end of the log instead of searching for an old location and changing it. Writing at the end in one direction is called a sequential write, and disks can do it quickly.
Cassandra also places the message in a memtable. A memtable is an in-memory structure, so it lives in fast RAM.
After these two quick steps, Cassandra can say that the write has succeeded. It does not need to jump around the disk and rearrange old files before replying to the user.
When the memtable becomes full, Cassandra writes it to the disk as a sorted file called an SSTable. This also happens in a large sequential write.
SSTables do not change after they are created. As time passes, many SSTables are created. Cassandra later joins and cleans them in the background. This process is called compaction.
So the write path is fast because it mainly does two easy jobs:
- append the message to a disk log in order;
- put the message in RAM.
The harder work of sorting, joining, and cleaning files happens later in the background.
Reading can require Cassandra to check the memtable and more than one SSTable to find the newest value. That is why Cassandra writes are usually faster than its reads. Even then, its read speed was good enough for Discord’s needs.
The Main Lesson
Discord did not move away from MongoDB because MongoDB was simply a bad database. MongoDB helped the team build its first product quickly. The real problem was that Discord’s data size and usage pattern had changed.
The team studied the problem before choosing a new tool. It learned that the system needed fast writes, predictable random reads, easy horizontal growth, automatic recovery, and low maintenance.
Cassandra matched those needs. It spread data across equal nodes, kept three copies for safety, remained available during failures, stored related data together, and used sequential writes to accept new messages quickly.
The biggest lesson is not that every chat app should use Cassandra. The lesson is that database choices should come from the real behaviour of the product. A simple database can be the right choice at the beginning, and a distributed database can become the right choice when the scale and problems truly need it.
