Apache Hadoop is an open-source framework designed to store and process large datasets across clusters of computers. It uses distributed storage and parallel processing to handle massive amounts of data efficiently, overcoming the limitations of traditional data processing systems.
- Handles structured, semi-structured and unstructured data.
- Provides scalability and fault tolerance across multiple machines.
Understanding Big Data
This section builds the foundation required to understand why Hadoop was created.
Fundamentals of Hadoop
This section introduces Hadoop as a solution to Big Data challenges.
- Introduction
- Architecture
- Hadoop 2.x vs Hadoop 3.x
- Ecosystem
- Install Hadoop in Linux
- Install Hadoop in Windows
- RDBMS vs Hadoop
Hadoop Ecosystem Tools
Hadoop consists of core components that manage storage, processing and resource allocation.
- Core Components: HDFS, YARN, MapReduce
- Storage and Query: HBase, Hive, Presto, Trino
- Data Processing: Spark, Flink
- Data Ingestion & Event Streaming: Kafka, Apache NiFi, Sqoop (Legacy)
- Coordination Tool: Zookeeper, Pig (Legacy)
Cluster, Rack & Schedulers
This section explains how Hadoop organizes machines and manages tasks efficiently.
HDFS
HDFS is Hadoop’s distributed file system designed for large-scale storage.
- Architecture
- Various Filesystems
- Block Size
- Daemons
- Replication Factor
- Data Operation
- Fault tolerance & high availability
Commands
MapReduce
MapReduce is Hadoop’s programming model for processing big data.
Programs
Hadoop Streaming & Security
This section covers Hadoop Streaming and managing HDFS file permissions and access controls.