🚀 Processing 10 TB Data in Apache Spark in 3 Hours: A Cost-Effective Tuning Guide

Apache Spark is a beast when it comes to massive datasets, but not every organization has a 6,000-core cluster lying around to finish a job in 10 minutes. Often, the goal is to balance performance with cost.

In this post, we’ll dive into how to tune your spark-submit parameters—executor memory, cores, and count—to process 10 TB of data within a 3-hour (180-minute) window. This approach is significantly more budget-friendly and fits standard enterprise cluster quotas.

📦 Dataset and Partitioning Strategy

We are dealing with a massive scale:

• Data size: 10 TB
• Default Partition size: 128 MB

🧮 Calculate Number of Partitions

First, we need to know how many tasks Spark has to manage.

• Data size: 10 TB
• Partition size: 128 MB (default in Spark)

1 TB = 1,048,576 MB  
10 TB = 10,485,760 MB  
Partitions = 10,485,760 MB / 128 MB = 81,920

This means Spark needs to execute 81,920 individual tasks.

🔧 Step-by-Step Parameter Calculation