CAP Theorem: Why Distributed Systems Can’t Have It All
Before becoming a software developer, I was simply a user of applications. Whenever I used an application, I expected everything to work perfectly. I wanted it to be available 24/7, my data to always be consistent, and every request to receive an immediate response without any network delays.
In my mind, that was how a good application should work.
But as I started learning about software development and, more importantly, distributed systems, I realised that achieving all these expectations simultaneously is not always possible.
When an application runs across multiple servers, communicates over a network, and handles thousands or even millions of requests, things become more complicated. Network failures can happen, servers can become unreachable, and data may not be immediately consistent across all nodes.
So, does this mean we have to compromise on some of these expectations?
Interestingly, there is a fundamental concept in distributed systems that explains exactly these trade-offs: the CAP Theorem.
Let’s understand what it means and why it matters when designing distributed systems.
According to the CAP Theorem, a distributed system cannot guarantee all three properties Consistency, Availability, and Partition Tolerance simultaneously when a network partition occurs.
However, CAP doesn’t mean that a distributed system permanently chooses any two out of these three properties. Instead, the real trade-off comes into play when a network partition occurs. At that point, the system must choose between prioritising Consistency or Availability, while continuing to tolerate the network partition.
A network partition occurs when nodes (servers) in a distributed system become unable to communicate with each other because of a network failure. In other words, the CAP Theorem helps us understand the trade-offs we need to make when designing reliable distributed systems.
Consistency: Every read receives the most recent write or an error.
Availability: Every request receives a response, even if the response may not contain the latest data.
Partition Tolerance: The system continues operating even when communication between some nodes is interrupted or delayed.
Understanding CAP Theorem Through a Real-World Example:
Imagine we are designing a banking application that maintains customer account balances. To handle a large number of requests, we have deployed our application across two servers: Node A and Node B.
Both nodes maintain a copy of the customer’s account balance, and they communicate with each other to keep the data synchronized.
Initially, the customer’s account balance is ₹10,000 on both nodes.
- Node A: ₹10,000
- Node B: ₹10,000
Under normal circumstances, both nodes can communicate with each other and keep their data consistent.
Now, let’s introduce a problem.
What happens when a network partition occurs?
Suppose a network failure occurs between Node A and Node B. Both servers are still running, but they can no longer communicate with each other.
A customer makes a withdrawal of ₹2,000 through Node A. Node A successfully processes the withdrawal and updates the balance to ₹8,000. However, Node B is unaware of this update because the network connection between the two nodes is interrupted.
- Node A: ₹8,000 (updated successfully)
- Node B: ₹10,000 (still holds the old balance)
This is where the CAP Theorem becomes interesting.
A customer now sends a request to Node B to check their account balance.
What should Node B do?
There are two possible approaches.
Approach 1: Prioritising Consistency (CP)
If we prioritise consistency, Node B cannot safely return the old balance of ₹10,000 because Node A has already processed the withdrawal.
Since Node B cannot communicate with Node A to verify the latest state, it may reject the request or wait until communication is restored.
In this approach:
- The system avoids returning potentially stale data.
- Some requests may fail or remain unavailable during the partition.
- Consistency is prioritised over availability.
This is known as a CP approach (Consistency + Partition Tolerance).
Approach 2: Prioritising Availability (AP)
Alternatively, we can prioritise availability.
Get Shreeprabha Bhat’s stories in your inbox
Join Medium for free to get updates from this writer.
Node B can continue responding to customer requests even though it cannot communicate with Node A.
It may return the balance of ₹10,000 based on its locally available data.
In this approach:
- The system continues responding to requests.
- Some responses may contain stale data.
- The nodes can synchronise their data once communication is restored.
This is known as an AP approach (Availability + Partition Tolerance).
What should we learn from this example?
The CAP Theorem does not tell us that one approach is universally better than the other. Instead, it helps us understand the trade-offs involved in designing distributed systems. For a banking application, returning an outdated account balance or accepting conflicting withdrawals could have serious consequences. Therefore, we may prefer consistency for critical financial operations, even if some requests temporarily become unavailable.
CP vs AP: Understanding the Trade-offs
Now that we understand how a network partition affects communication between nodes, let’s summarise the two approaches.
CP (Consistency + Partition Tolerance)
- Prioritises returning the latest consistent data.
- May reject or delay requests when the latest state cannot be verified.
- Sacrifices availability during a network partition when necessary to preserve consistency.
- Suitable for operations where returning stale data could cause serious problems.
AP (Availability + Partition Tolerance)
- Prioritises responding to requests even during a network partition.
- May return stale data because nodes cannot communicate with each other.
- Allows nodes to continue operating independently.
- Data can be synchronized after communication is restored, depending on the system’s conflict-resolution strategy.
Neither approach is universally better. The choice depends on the application’s requirements and the consequences of temporarily returning stale data or rejecting requests.
For example, a banking system may prioritise consistency for critical financial transactions, whereas a social media application may prioritise availability when displaying likes or comments.
What About CA? Why Can’t We Have All Three?
By now, we have discussed CP and AP systems. But what about CA (Consistency + Availability)? Why can’t a distributed system guarantee all three properties?
To understand this, let’s revisit the network partition scenario.
When a network partition occurs, nodes in a distributed system can no longer communicate with each other.
Now, consider a system that wants to guarantee both consistency and availability.
If a node receives a request, it must respond to that request to maintain availability. However, because it cannot communicate with the other nodes, it may not have the latest data.
If it responds using its local data, it risks returning stale information, violating consistency.
On the other hand, if it waits for confirmation from the other nodes or rejects the request, it may preserve consistency, but it sacrifices availability.
This is the fundamental trade-off explained by the CAP Theorem.
A CA system can provide consistency and availability when there is no network partition, but it cannot guarantee both during a partition while also tolerating that partition.
In other words, when a network partition occurs, a distributed system that continues to tolerate the partition must make a trade-off between consistency and availability.
Final Thoughts
When I first started learning about distributed systems, I assumed that a good application should always be available, always return consistent data, and respond to every request immediately.
But CAP Theorem helped me understand that distributed system design is not about achieving every ideal property simultaneously. It is about understanding the trade-offs and making decisions based on the application’s requirements.
As engineers, our responsibility is not simply to build systems that work under ideal conditions, but to design systems that behave predictably even when things go wrong.
And that is what makes distributed systems interesting.
