Apache Parquet: Columnar Data Format for Big Data Analytics

Apache Parquet: Columnar Data Format for Big Data Analytics

Apache Parquet is an open-source columnar storage format designed for use with big data processing frameworks in the Hadoop ecosystem and modern cloud data lakes. Developed jointly by Twitter and Cloudera in 2013 and donated to the Apache Software Foundation, Parquet has become the de facto standard for analytical workloads — used by Apache Spark, Apache Hive, Apache Drill, Presto, AWS Athena, Google BigQuery, Snowflake, DuckDB, and virtually every modern data platform.

The core insight behind Parquet — and all columnar formats — is that analytical queries typically access only a few columns out of many, and columnar storage makes these column-selective reads dramatically faster than row-oriented storage.

Row vs Columnar Storage: Why It Matters

Consider a table with 1 billion rows and 50 columns — a typical dimension table in a data warehouse. A query like SELECT AVG(revenue) FROM orders WHERE country = 'ES' needs only 2 columns out of 50.

Row-oriented storage (CSV, Parquet predecessor formats):

Row 1: [id=1, date=2024-01-01, country=US, product=A, revenue=99.99, ...]  ← 50 fields
Row 2: [id=2, date=2024-01-01, country=ES, product=B, revenue=149.00, ...]
...
Row 1B: [id=1000000000, ...]

To read country and revenue, the engine must scan every byte of every row — reading all 50 columns even though only 2 are needed. For 1 billion rows at ~200 bytes/row, that is 200 GB of I/O for a 2-column query.

Columnar storage (Parquet):

Column 'id':      [1, 2, 3, 4, ...]         ← 8 GB (int64)
Column 'date':    [2024-01-01, ...]          ← 8 GB (date32)
Column 'country': [US, ES, US, ES, ...]      ← 4 GB (dictionary-encoded string)
Column 'revenue': [99.99, 149.00, ...]       ← 8 GB (float64)
...

The engine reads only country (4 GB) and revenue (8 GB) = 12 GB of I/O — a ~17× reduction in data read. With compression (country has very high repetition, Snappy reduces it to ~0.5 GB), the actual I/O can be under 10 GB instead of 200 GB.

Parquet File Structure

A Parquet file is organized into a hierarchy of physical and logical structures:

Parquet File
├── Magic bytes: PAR1 (4 bytes)
├── Row Group 0 (e.g., 128 MB of rows)
│   ├── Column Chunk: 'id' (int64)
│   │   ├── Data Page 0
│   │   ├── Data Page 1
│   │   └── Dictionary Page (if dict encoding)
│   ├── Column Chunk: 'country' (string, dict-encoded)
│   │   └── ...
│   └── Column Chunk: 'revenue' (double)
│       └── ...
├── Row Group 1
│   └── ...
├── Row Group N
│   └── ...
├── File Footer (metadata)
│   ├── Schema (column names, types, logical types)
│   ├── Row group metadata (file offsets, min/max stats per column chunk)
│   └── Key-value metadata (custom application metadata)
└── Magic bytes: PAR1 (4 bytes)

Row Groups

A row group is a horizontal partition of the data — a set of consecutive rows stored together. The default row group size in PyArrow and Spark is 128 MB (after compression). Larger row groups improve compression ratios but increase memory requirements for writing; smaller row groups enable finer-grained predicate pushdown but add metadata overhead.

Column Chunks

Within each row group, each column is stored as a column chunk — all the values for that column across all rows in the row group, stored contiguously. Column chunks are what enable column-selective reads: to read only revenue, the reader seeks directly to each column chunk's byte offset (stored in the footer metadata) and reads only that data.

Pages

Column chunks are further divided into pages — the smallest unit of compression and encoding in Parquet. Pages are typically 1 MB by default. Each page has a header containing the page type, compressed/uncompressed size, encoding, and the number of values.

Page types:

• Data page (v1/v2): the actual column values
• Dictionary page: the dictionary lookup table for dictionary-encoded columns
• Index page: bloom filter or column index data for predicate pushdown

Encoding Schemes

Parquet uses several encoding schemes to reduce data size before compression:

Dictionary Encoding

For low-cardinality columns (like country, status, category), Parquet builds a dictionary of distinct values and encodes each value as a small integer index:

Dictionary: {0: "US", 1: "ES", 2: "DE", 3: "FR"}
Encoded values: [0, 1, 0, 0, 1, 2, 0, 3, 1, 0, ...]

Instead of storing repeated strings like "United States" (13 bytes each) thousands of times, each occurrence costs 1-2 bytes as a dictionary index. Dictionary encoding is applied automatically when the cardinality falls below a threshold (default: first 1 MB of data).

RLE/Bit-Packing Hybrid

For the dictionary indices and definition/repetition levels, Parquet uses a run-length encoding (RLE) + bit-packing hybrid. Runs of identical values are encoded as (value, count) pairs; non-runs are bit-packed at the minimum number of bits required for the maximum value.

For country indices (4 values, 2 bits each):

Run [0, 0, 0, 0, 0] (5 US) → RLE: (value=0, count=5)
Mixed [1, 2, 0, 3]          → Bit-packed: 01 10 00 11 (8 bits)

Delta Encoding

For sorted or nearly-sorted integer/timestamp columns, delta encoding stores only the differences between consecutive values:

Values:  [1000, 1001, 1003, 1007, 1015]
Deltas:  [1000, +1, +2, +4, +8]         ← first value + differences

Timestamps often have small, regular deltas (e.g., one millisecond apart), making delta encoding extremely effective.

BYTE_STREAM_SPLIT

For floating-point columns, BYTE_STREAM_SPLIT interleaves the byte streams of multiple values to improve compression:

For 4 float32 values (each 4 bytes):

Standard:  [B0_0 B0_1 B0_2 B0_3 | B1_0 B1_1 B1_2 B1_3 | ...]
Split:      [B0_0 B1_0 B2_0 B3_0 | B0_1 B1_1 B2_1 B3_1 | ...]

By grouping bytes at the same position across all values, similar bytes end up adjacent, dramatically improving Zstd and Snappy compression ratios for floating-point scientific data.

Compression Codecs

After encoding, Parquet compresses each page:

Zstandard (ZSTD) at level 3 is increasingly the default in modern tools (DuckDB, newer Spark versions) because it achieves GZIP-level compression at SNAPPY-level speed.

Python: PyArrow

PyArrow is the primary Python library for Parquet:

import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
import numpy as np

# Create a PyArrow table
table = pa.table({
    'id': pa.array(range(1000000), type=pa.int64()),
    'country': pa.array(['US', 'ES', 'DE'] * 333334, type=pa.string()),
    'revenue': pa.array(np.random.uniform(10, 1000, 1000000), type=pa.float64()),
    'timestamp': pa.array(pd.date_range('2024-01-01', periods=1000000, freq='s')),
    'is_active': pa.array([True, False] * 500000, type=pa.bool_()),
})

# Write Parquet with options
pq.write_table(
    table,
    'orders.parquet',
    compression='zstd',
    compression_level=3,
    row_group_size=100000,           # 100K rows per row group
    use_dictionary=['country'],       # Explicit dict encoding for low-cardinality
    write_statistics=True,            # Min/max stats per column per row group
    data_page_size=1024 * 1024,       # 1 MB data pages
    use_byte_stream_split=['revenue'], # Better float compression
)

# Read back
table2 = pq.read_table('orders.parquet')
print(table2.schema)
print(f"Rows: {table2.num_rows}, Columns: {table2.num_columns}")

# Column-selective read (reads only 2 columns, skips the rest)
revenue_by_country = pq.read_table(
    'orders.parquet',
    columns=['country', 'revenue'],
)

# Predicate pushdown (reads only matching row groups)
filtered = pq.read_table(
    'orders.parquet',
    columns=['id', 'revenue'],
    filters=[('country', '=', 'ES')],  # Pushes filter to row group stats
)

# Convert to/from Pandas
df = table2.to_pandas()
table3 = pa.Table.from_pandas(df, preserve_index=False)

# Inspect Parquet file metadata
meta = pq.read_metadata('orders.parquet')
print(f"Row groups: {meta.num_row_groups}")
print(f"Total rows: {meta.num_rows}")
print(f"Schema: {meta.schema}")
for i in range(meta.num_row_groups):
    rg = meta.row_group(i)
    print(f"Row group {i}: {rg.num_rows} rows, {rg.total_byte_size} bytes")

Partitioned Parquet Datasets

For large-scale analytics, Parquet files are typically partitioned by high-value filter columns:

# Write a partitioned dataset (Hive-style partitioning)
pq.write_to_dataset(
    table,
    root_path='orders_partitioned/',
    partition_cols=['country'],
    compression='zstd',
)
# Creates:
# orders_partitioned/country=US/part-0.parquet
# orders_partitioned/country=ES/part-0.parquet
# orders_partitioned/country=DE/part-0.parquet

# Read with automatic partition pruning
dataset = pq.ParquetDataset('orders_partitioned/', filters=[('country', '=', 'ES')])
es_table = dataset.read()  # Reads only ES partition files

Statistics and Predicate Pushdown

Parquet stores min/max statistics for each column in each row group. Query engines use these to skip entire row groups that cannot contain matching rows:

Row group 0: country min='DE', max='US'   → might contain 'ES'
Row group 1: country min='DE', max='DE'   → skip (no 'ES' possible)
Row group 2: country min='ES', max='US'   → might contain 'ES'

In a well-sorted dataset, only a fraction of row groups need to be read. This predicate pushdown can reduce I/O by 90% or more for highly selective queries on sorted columns.

Parquet with DuckDB

DuckDB can query Parquet files directly with SQL, with automatic predicate pushdown:

-- Query Parquet directly
SELECT country, AVG(revenue) as avg_revenue, COUNT(*) as orders
FROM read_parquet('orders.parquet')
WHERE timestamp >= '2024-06-01'
GROUP BY country
ORDER BY avg_revenue DESC;

-- Query partitioned dataset
SELECT * FROM read_parquet('orders_partitioned/**/*.parquet')
WHERE country = 'ES';

-- Convert CSV to Parquet
COPY (SELECT * FROM read_csv('orders.csv')) TO 'orders.parquet'
(FORMAT parquet, COMPRESSION zstd);

Parquet Schema Evolution

Parquet supports limited schema evolution:

• Add columns: new columns with null values for older files ✅
• Remove columns: columns absent in older files read as null ✅
• Rename columns: not supported without rewriting ❌
• Change types: limited (e.g., int32 → int64 may work; string → int does not) ⚠️

Conclusion

Apache Parquet has earned its dominant position in analytical data engineering because it solves the right problem with elegant engineering: by organizing data by column rather than row, it enables analytical queries to read only the data they need, applies highly effective compression to homogeneous column data, and provides statistics-based predicate pushdown to skip irrelevant data before any I/O occurs. Whether you are building a data lake on S3, querying data with DuckDB on a laptop, or running petabyte-scale Spark jobs, Parquet is the format that makes these workloads practical.

Related conversions

Frequent conversions across the catalogue:

• JPG to PNG
• MP4 to MP3
• PDF to DOCX
• PNG to JPG
• MP3 to WAV
• DOCX to PDF