Apache Parquet: The Columnar File Format for Big Data Analytics

Apache Parquet: The Columnar File Format for Big Data Analytics

What Is Apache Parquet?

Apache Parquet is an open-source, columnar storage file format designed for efficient analytical queries on large datasets. Originally developed by Twitter and Cloudera in 2013 and now a top-level Apache Software Foundation project, Parquet has become the de facto standard format for data lakes, cloud analytics (AWS Athena, Google BigQuery, Azure Synapse), and the data engineering ecosystem (Apache Spark, Dask, pandas, DuckDB, Polars).

Files carry the .parquet extension and are self-describing — schema information is embedded in the file's metadata footer, not in a separate schema registry.

Row-Oriented vs. Column-Oriented Storage

The fundamental insight behind Parquet is the difference between how data is stored versus how it is queried:

Row-oriented (CSV, JSON, Avro): all columns of row 1 are stored together, then all columns of row 2, etc.

[id=1, name="Alice", age=30, salary=75000]
[id=2, name="Bob",   age=25, salary=62000]
[id=3, name="Carol", age=35, salary=91000]

Column-oriented (Parquet): all values of column 1 are stored together, then all values of column 2, etc.

id:     [1, 2, 3, ...]
name:   ["Alice", "Bob", "Carol", ...]
age:    [30, 25, 35, ...]
salary: [75000, 62000, 91000, ...]

For analytical queries like SELECT AVG(salary) WHERE age > 28, a column-oriented format reads only the salary and age columns — skipping id and name entirely. On a table with 1,000 columns, this can reduce I/O by 99%.

Additionally, values within a column are of the same type and tend to be more similar to each other than values across columns — leading to much better compression ratios (often 5–10× better than row-oriented storage).

Parquet File Structure

File:
├── Row Group 0                  ← batch of rows (default: 128 MB)
│   ├── Column Chunk: id         ← compressed column data for rows in RG0
│   ├── Column Chunk: name
│   ├── Column Chunk: age
│   └── Column Chunk: salary
├── Row Group 1
│   ├── Column Chunk: id
│   └── ...
└── Footer                       ← file metadata (schema, statistics, offsets)
    ├── Schema                   ← Thrift-encoded schema (field names, types, nullability)
    ├── Row Group Metadata[]     ← for each row group: min/max stats per column chunk
    └── Key-Value Metadata       ← user-defined key/value pairs (pandas schema, etc.)

Row Groups

Row groups partition the file into horizontal slices (default 128 MB). They enable:

• Row group filtering: the footer contains min/max statistics per column per row group. If salary_min=50000 and salary_max=70000 for a row group, a query WHERE salary > 80000 skips the entire row group without reading it.
• Parallelism: different executors in Spark/Dask can read different row groups concurrently.

Pages

Within each column chunk, data is further divided into data pages (default 1 MB). Pages are the smallest unit of decompression — statistics and encodings are tracked per page.

Column Statistics

Each column chunk stores min/max values and null count. Combined with row group-level stats, this enables predicate pushdown — skipping entire row groups or column chunks before reading any data.

Data Types and Encodings

Parquet supports a rich type system:

Physical types (on-disk storage primitives):

• BOOLEAN, INT32, INT64, INT96 (deprecated 96-bit timestamp), FLOAT, DOUBLE
• BYTE_ARRAY (variable-length bytes), FIXED_LEN_BYTE_ARRAY

Logical types (semantic annotations on physical types):

• STRING (UTF-8 BYTE_ARRAY)
• DATE (INT32, days since epoch)
• TIME (INT64, microseconds since midnight)
• TIMESTAMP (INT64, microseconds since epoch; isAdjustedToUTC flag)
• DECIMAL (BYTE_ARRAY or fixed, with precision+scale)
• LIST, MAP, STRUCT — nested types via Dremel encoding (repetition + definition levels)
• UUID (FIXED_LEN_BYTE_ARRAY of 16 bytes)
• JSON, BSON (BYTE_ARRAY with logical type annotation)

Encodings (per column chunk, chosen adaptively):

• Plain: raw values, no transformation
• Dictionary: values replaced by dictionary indices (excellent for low-cardinality strings)
• RLE (Run-Length Encoding): for repetitive sequences
• Delta encoding: stores differences between consecutive integers (timestamps, sorted IDs)
• Bit packing: compact representation for small integers

Compression

Parquet supports per-column-chunk compression:

• SNAPPY (default in many tools) — very fast, ~60–70% of original size
• GZIP — smaller files (~50%), slower decompression
• BROTLI — best ratio (~45%), slowest
• ZSTD — excellent ratio (~50%) with fast decompression — recommended for new files
• LZ4 — fastest decompression, moderate ratio

Working with Parquet in Python

pandas + PyArrow

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# Create a DataFrame
df = pd.DataFrame({
    'id':     [1, 2, 3, 4, 5],
    'name':   ['Alice', 'Bob', 'Carol', 'Dave', 'Eve'],
    'age':    [30, 25, 35, 28, 32],
    'salary': [75000.0, 62000.0, 91000.0, 58000.0, 84000.0]
})

# Write to Parquet (ZSTD compression)
df.to_parquet('employees.parquet', compression='zstd', index=False)

# Read back
df2 = pd.read_parquet('employees.parquet')

# Read only specific columns (skips other columns entirely)
df_salary = pd.read_parquet('employees.parquet', columns=['name', 'salary'])

# Read with row filter (predicate pushdown to Parquet statistics)
df_senior = pd.read_parquet(
    'employees.parquet',
    filters=[('age', '>', 28)]
)

PyArrow native (more control)

import pyarrow as pa
import pyarrow.parquet as pq

# Write with partitioning (creates directory tree: country=US/, country=UK/, ...)
table = pa.Table.from_pandas(df)
pq.write_to_dataset(
    table,
    root_path='employees_partitioned/',
    partition_cols=['department'],
    compression='zstd'
)

# Read a partitioned dataset
dataset = pq.ParquetDataset(
    'employees_partitioned/',
    filters=[('department', '=', 'Engineering')]
)
table = dataset.read()

# Inspect file metadata (without reading data)
meta = pq.read_metadata('employees.parquet')
print(meta.num_rows)           # 5
print(meta.num_row_groups)     # 1
print(meta.schema)             # field names and types
print(meta.row_group(0).column(3).statistics)  # min/max for salary column

DuckDB (SQL over Parquet files)

-- Query Parquet directly with SQL (no loading into memory)
SELECT department, AVG(salary) as avg_salary
FROM read_parquet('employees_*.parquet')
WHERE age > 25
GROUP BY department
ORDER BY avg_salary DESC;

-- Read all files in a directory
SELECT * FROM read_parquet('data/year=2023/**/*.parquet');

import duckdb

conn = duckdb.connect()
result = conn.execute("""
    SELECT year, SUM(revenue) as total_revenue
    FROM read_parquet('sales/**.parquet')
    GROUP BY year
    ORDER BY year
""").df()  # returns pandas DataFrame

Polars (fast native Parquet support)

import polars as pl

# Lazy scan — no data loaded until collect()
lazy = pl.scan_parquet('large_dataset/*.parquet')

result = (
    lazy
    .filter(pl.col('age') > 28)
    .select(['name', 'salary', 'department'])
    .group_by('department')
    .agg(pl.col('salary').mean().alias('avg_salary'))
    .collect()
)

Hive Partitioning

Large Parquet datasets are typically organized using Hive-style partitioning — a directory naming convention where partition values are encoded in path components:

sales_data/
├── year=2022/
│   ├── month=01/
│   │   ├── part-00000.parquet
│   │   └── part-00001.parquet
│   └── month=02/
│       └── part-00000.parquet
└── year=2023/
    └── month=01/
        └── part-00000.parquet

Query engines (Spark, Athena, DuckDB, Polars) automatically recognize this structure and skip irrelevant partitions when WHERE year=2023 AND month=01 is in the query.

Parquet in the Data Ecosystem

• Apache Spark: native Parquet support; default format for DataFrames written to storage
• AWS Athena: SQL over Parquet files in S3; cost is proportional to bytes scanned
• Google BigQuery: supports external Parquet tables; columnar on-disk aligns with BigQuery's internal columnar storage
• Azure Synapse Analytics: Parquet is the recommended external table format
• Apache Hudi / Delta Lake / Apache Iceberg: all three ACID table formats use Parquet as their underlying data file format
• MLflow: stores model artifacts and experiment metadata alongside Parquet training data

Practical Tips

• Use ZSTD compression — better ratio than Snappy with fast-enough decompression for most workloads
• Row group size 128–256 MB is the sweet spot; too small → too many row groups; too large → poor parallelism
• Partition on high-cardinality but commonly filtered columns (date, region, category) — avoid partitioning on very-high-cardinality columns like user_id (creates millions of files)
• Use write_statistics=True (default) to store min/max statistics — essential for predicate pushdown
• Sort before writing: if data is sorted by a commonly filtered column (timestamp), predicate pushdown is dramatically more effective
• Avoid small files: many tiny Parquet files (< 32 MB) hurt query performance; use OPTIMIZE commands in Delta Lake/Iceberg or coalesce() in Spark before writing
• Schema evolution: Parquet supports adding nullable columns to existing datasets — readers that predate the new column get null for it

Parquet's columnar layout, predicate pushdown, per-column compression, and rich type system make it the universal format for analytical workloads at any scale.

Related conversions

Frequent conversions across the catalogue:

• JPG to PNG
• MP4 to MP3
• PDF to DOCX
• PNG to JPG
• MP3 to WAV
• DOCX to PDF