Using DuckDB to Rewrite Your Parquet Files

A Simple Case of Less Is More

AI Generated Image by Author

Parquet is considered the gold standard file format for analytical engines because its physical design aligns directly with the hardware mechanics of modern CPU execution, memory layout, and cloud network I/O.

That said, too many small files cancel out the benefits of this file format.

The Small File Problem

In modern data engineering and analytical engines (such as DuckDB, Apache Spark, Trino, and Snowflake), a Parquet file is generally considered “small” if it is under 100 MB.

In high-throughput distributed pipelines, files under 10 MB or in the few-kilobyte range are categorized as severe “tiny files” that actively trigger the Small File Problem.

general guideline for classifying file sizes

Why Small Parquet Files Cause Performance Bottlenecks

Footer & Metadata Overhead
Parquet stores schema, column statistics (min/max), and dictionary offsets in the file’s footer. When files are tiny (e.g., a few kilobytes or megabytes), the ratio of metadata to actual data becomes disproportionately high.