Understanding NoSQL Databases: Why Big Data Needed a Different Approach

So far in our, we’ve seen how data can be:

• Ingested from different sources
• Stored in Data Lakes
• Represented using different file formats
• Distributed across machines using Hadoop
• Processed using MapReduce

We also learned about CAP Theorem, which showed us that distributed systems have to deal with an important trade-off between Consistency, Availability, and Partition Tolerance.

Now let’s look at another major question:

If our data is becoming massive and distributed, are traditional relational databases always the best choice?

For many applications, relational databases are still an excellent choice. But when systems started dealing with:

• Huge volumes of data
• Rapidly changing data
• Unstructured and semi-structured information
• Millions of users
• Globally distributed applications
• Very high read/write traffic

developers began looking for different database architectures.

This led to the rise of: NoSQL Databases

What is NoSQL?

NoSQL generally refers to database systems that do not rely on the traditional relational, table-based model as their primary way of representing data.

The name is commonly interpreted as: “Not Only SQL”

This is important. NoSQL does not necessarily mean: “No SQL at all.”

Instead, it represents a broader family of databases that use different data models and access patterns depending on the problem.

Instead of forcing every dataset into:

Table
 ├── Rows
 └── Columns

NoSQL databases can represent data using models such as:

Key → Value
Document
Column Family
Graph

The goal is often to make it easier to build systems that require horizontal scalability, flexible schemas, high throughput, or distributed operation.

First, Let’s Understand Traditional Relational Databases

Before understanding why NoSQL exists, let’s quickly revisit the relational model. A relational database stores data in tables.

For example, an e-commerce application might have:

CUSTOMERS

ID | Name   | City
---|--------|------
1  | Aisha  | Delhi
2  | Rahul  | Noida
3  | Neha   | Jaipur

And another table:

ORDERS

Order_ID | Customer_ID | Amount
---------|-------------|-------
101      | 1           | 500
102      | 2           | 900
103      | 1           | 300

The relationship between the tables is maintained using keys.

This model is powerful because it provides:

• Structured data
• Relationships
• SQL querying
• Transactions
• Strong consistency options
• Mature tooling

For many applications, this is exactly what we need. But at very large scale, some challenges can appear.

Why Did NoSQL Become Popular?

The growth of the internet changed the amount and type of data applications had to handle.

Consider applications such as:

• Social media
• Search engines
• Online shopping
• Streaming platforms
• Gaming
• IoT
• Real-time analytics

These systems may have: Millions or billions of records and potentially: Thousands or millions of requests per second.

Traditional relational databases can scale significantly, but scaling a single relational system vertically can eventually become expensive or insufficient for certain workloads.

This encouraged architectures based heavily on: Horizontal Scaling

Vertical vs Horizontal Scaling

Let’s understand this difference.

Vertical Scaling

We make one machine more powerful.

For example:

Server
  ↓
More CPU
More RAM
More Storage

This is called scale up. It can work well, but the machine has physical and economic limits.

Horizontal Scaling

Instead of making one machine more powerful, we add more machines.

       Server
          ↓
   ┌──────┼──────┐
   ↓      ↓      ↓
Server  Server  Server

This is called scale out. Distributed NoSQL databases are often designed with horizontal scaling as a core requirement.

The Main Idea Behind NoSQL

The central idea isn’t simply: “NoSQL is better than SQL.”

Instead: Different applications have different data and scalability requirements, so different database models make sense for different problems.

For example: A banking application may strongly depend on transactions and consistency. A social media application may need massive scalability and flexible data structures. An IoT application may continuously generate huge amounts of time-series-like data. A recommendation system may need to represent relationships between users and products.

One database model doesn’t have to solve all of these problems equally well.

The Four Major Types of NoSQL Databases

NoSQL databases are commonly divided into four major categories:

1. Key-Value Databases

This is the simplest NoSQL model.

Data is stored as:

Key → Value

For example:

"user_101" → "Shreya"
"user_102" → "Rahul"
"user_103" → "Aisha"

The key uniquely identifies the value.

Another example could be:

"session_123" → User Session Data

The database can quickly find the value using the key.

Why Use Key-Value Databases?

They are useful when we already know the key and primarily need fast access to the associated value.

Common applications include:

• Caching
• Session management
• Shopping carts
• User preferences
• Real-time counters

A well-known example is: Redis

2. Document Databases

Document databases store data as documents, commonly represented using JSON-like structures.

For example:

{
    "id": 101,
    "name": "Shreya",
    "city": "Noida",
    "skills": [
        "C++",
        "Python",
        "SQL"
    ]
}

Instead of splitting this information across multiple relational tables, a document database can store the related information together in a single document.

Why Are Document Databases Useful?

One major advantage is schema flexibility.

Suppose one user has:

name
city
skills

while another user also has:

name
city
skills
experience
projects

A document-oriented database can often accommodate these differences more naturally than a rigid relational schema.

This makes document databases useful for:

• Content management
• Product catalogues
• User profiles
• Web applications
• Applications where data structures evolve frequently

A popular example is: MongoDB

3. Column-Family Databases

Column-family databases organize data around column families rather than traditional relational tables.

This model is particularly useful for:

• Massive datasets
• Distributed storage
• High write throughput
• Large-scale analytical or operational workloads

A well-known example is: Apache HBase

HBase is especially important in the Hadoop ecosystem because it can provide distributed, random, real-time read/write access over very large datasets.

4. Graph Databases

Some applications aren’t primarily about rows and columns.

They’re about: Relationships.

For example:

Shreya
   ↓ follows
Rahul
   ↓ follows
Aisha

A graph database represents entities as nodes and their relationships as edges.

Get Shireen Kachroo’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For example:

       Rahul
      /     \
 follows   works_with
    ↓          ↓
 Shreya     Aisha

Graph databases are useful for:

• Social networks
• Recommendation systems
• Fraud detection
• Network analysis
• Knowledge graphs
• Relationship-heavy applications

When relationships are the central part of the problem, graph databases can be a natural choice.

SQL vs NoSQL

Let’s compare the traditional relational model with the broader NoSQL approach.

But there is an important point:

SQL databases can also scale horizontally and run in distributed architectures.

And:

NoSQL databases can also provide strong consistency and transactions.

So we should avoid treating SQL and NoSQL as two completely rigid categories. The real difference is primarily about data models, system architecture, scalability patterns, and workload requirements.

Schema: Fixed vs Flexible

One of the biggest differences you’ll hear about is schema.

Relational Database

A relational database typically defines the structure beforehand.

For example:

STUDENT

ID
Name
Age
Department

Every row follows this structure.

Document Database

A document database can allow documents with different fields.

For example:

Document 1

{
    "name": "Shireen",
    "age": 21
}

Another document could be:

{
    "name": "Rahul",
    "age": 22,
    "skills": ["C++", "SQL"]
}

This flexibility can be useful when application requirements change frequently.

However, flexibility does not mean:

“No structure is required.”

Poorly designed document schemas can still create data-quality and maintenance problems.

NoSQL and Horizontal Scaling

One of the major reasons NoSQL became popular was its compatibility with distributed architectures.

Suppose we have:

              NoSQL Database
                   │
        ┌──────────┼──────────┐
        ↓          ↓          ↓
      Node 1     Node 2     Node 3

As the dataset grows, we can add more nodes:

Node 1
Node 2
Node 3
   ↓
Node 4
Node 5
...

The data can be distributed across the cluster.

This approach is called:

Scale Out

It is particularly useful for applications that need to handle massive amounts of data and traffic.

Sharding

Another important concept in distributed NoSQL databases is: Sharding

Sharding means dividing data across multiple machines.

For example, suppose we have:

Users 1–1,000,000

We could distribute the data:

Shard 1 → Users 1–250,000
Shard 2 → Users 250,001–500,000
Shard 3 → Users 500,001–750,000
Shard 4 → Users 750,001–1,000,000

Now different machines handle different portions of the dataset. This allows the system to scale horizontally.

Replication in NoSQL

Sharding distributes data. But what happens if a machine fails?

We can use: Replication

For example:

             Shard 1
            /       \
       Primary     Replica

If the primary node fails, another replica may be able to take over, depending on the database’s architecture.

So Sharding → Helps with scalability

Replication → Helps with availability and fault tolerance

Many distributed databases use both.

NoSQL and CAP Theorem

Remember our previous article?

We discussed: Consistency, Availability, Partition Tolerance

NoSQL databases often operate in distributed environments where network partitions are possible. Therefore, CAP-related trade-offs become important.

Depending on the database and configuration, a system may prioritise:

Stronger consistency

or

Higher availability with eventual consistency

The important thing is:

There is no single CAP behaviour shared by every NoSQL database.

Different systems make different design choices. This is why understanding the actual database architecture is more important than simply memorising: “NoSQL = AP.” That’s an oversimplification.

Eventual Consistency

Some distributed NoSQL systems use eventual consistency. This means replicas may temporarily contain different values but eventually converge.

For example:

Initially:

Node A → Value X
Node B → Value X

An update occurs:

Node A → Value Y
Node B → Value X

After replication:

Node A → Value Y
Node B → Value Y

This can allow the system to remain highly available while updates propagate through the cluster. Whether eventual consistency is appropriate depends entirely on the application’s requirements.

Polyglot Persistence

Using different databases for different requirements is called: Polyglot Persistence

For example:

                  Application
                       │
        ┌──────────────┼──────────────┐
        ↓              ↓              ↓
     SQL DB         Redis          MongoDB
  Transactions       Cache         Documents

Another application might use:

SQL + Redis + HBase + Graph Database

Each database is selected based on the workload it handles best.

This is a common architectural idea in modern distributed systems.

NoSQL Does NOT Mean “SQL Is Dead”

This is one of the biggest misconceptions. NoSQL did not replace relational databases.

Instead:

SQL and NoSQL solve different problems, and modern systems often use both.

For example:

=> A payment system may use a relational database because transactions and consistency are critical.

=> A caching layer may use Redis.

=> A product catalogue may use MongoDB.

=> A large distributed dataset may use HBase.

So the better question is not: “SQL or NoSQL?”

It is: “Which database model best fits this particular workload?”

Advantages of NoSQL Databases

1. Horizontal scalability

Many NoSQL systems are designed to scale by adding machines.

2. Flexible data models

Document and other NoSQL models can handle changing or semi-structured data naturally.

3. High throughput

Some NoSQL systems are designed to handle very high read/write workloads.

4. Distributed architecture

Many NoSQL databases are designed to operate across clusters.

5. Fault tolerance

Replication and distributed architectures can provide resilience against node failures.

6. Specialised data models

Instead of forcing every problem into tables, NoSQL provides models suited to specific workloads.

Limitations of NoSQL

NoSQL isn’t automatically better. It also introduces trade-offs.

1. Weaker relational capabilities

Many NoSQL databases don’t provide the same relational features as traditional SQL databases.

2. Transactions vary

Transaction support differs significantly between NoSQL systems.

3. Querying can be less standardised

There is no universal NoSQL query language equivalent to SQL across all NoSQL databases.

4. Data duplication

Some NoSQL designs intentionally duplicate data to make reads faster. This can make updates more complicated.

5. Application complexity

Distributed systems can introduce challenges involving:

• Replication
• Consistency
• Sharding
• Failure recovery
• Conflict resolution

6. Not ideal for every workload

For highly relational transactional applications, a relational database may still be the better choice.

Key Takeaways

If you remember only a few things from this article, remember these:

1. NoSQL generally means “Not Only SQL” and represents a family of non-relational database approaches.

2. NoSQL became popular as applications began handling massive, distributed, high-throughput, and flexible data workloads.

3. The four commonly discussed NoSQL categories are Key-Value, Document, Column-Family, and Graph databases.

4. Key-Value databases are useful for fast key-based access.

5. Document databases store flexible JSON-like documents.

6. Column-family databases are designed for large-scale distributed datasets and include systems such as HBase.

7. Graph databases focus on relationships between entities.

8. NoSQL systems often emphasise horizontal scalability, but not every NoSQL database works in exactly the same way.

9. Sharding distributes data across machines, while replication maintains additional copies for availability and fault tolerance.

10. NoSQL does not mean relational databases are obsolete.

11. Different workloads may require different database models.

12. Using multiple database technologies for different requirements is called polyglot persistence.

No SQL Summary