Apache Parquet Format: Columnar Data Storage with Python

Apache Parquet is an open-source columnar storage format designed for analytical workloads at scale. Unlike CSV — which stores data row by row — Parquet organizes data by column, making it ideal for queries that touch only a subset of columns.

Parquet vs CSV vs JSON

For analytical datasets over 1 million rows, Parquet is typically 5–20× faster to read and 3–10× smaller than equivalent CSV.

Installation

pip install pandas pyarrow
# fastparquet is a lighter alternative engine
pip install fastparquet  # optional

Reading and writing with pandas

import pandas as pd

df = pd.DataFrame({
    'name':   ['Alice', 'Bob', 'Carol'],
    'age':    [25, 30, 28],
    'salary': [50_000.0, 65_000.0, 72_000.0],
    'active': [True, True, False],
})

# Write
df.to_parquet('employees.parquet', engine='pyarrow', compression='snappy')

# Read
df2 = pd.read_parquet('employees.parquet')
print(df2.dtypes)
# name      object
# age        int64
# salary   float64
# active      bool
# Types are preserved exactly — no manual casting

Column selection and predicate pushdown

import pyarrow.parquet as pq

# Read only needed columns — much faster on large files
df_partial = pd.read_parquet('employees.parquet', columns=['name', 'salary'])

# Push-down filters: PyArrow filters BEFORE loading into RAM
table = pq.read_table(
    'employees.parquet',
    columns=['name', 'salary'],
    filters=[('salary', '>', 60_000)],
)
df_filtered = table.to_pandas()
print(df_filtered)
#    name   salary
# 1   Bob  65000.0
# 2 Carol  72000.0

Compression codecs

# Snappy — speed/compression balance (recommended default)
df.to_parquet('data.parquet', compression='snappy')

# Gzip — better ratio, slower
df.to_parquet('data.parquet', compression='gzip')

# Zstd — better ratio than gzip at similar speed (pandas >= 1.5)
df.to_parquet('data.parquet', compression='zstd')

# No compression
df.to_parquet('data.parquet', compression=None)

import os
for codec in ['snappy', 'gzip', 'zstd']:
    df.to_parquet(f'data_{codec}.parquet', compression=codec)
    size = os.path.getsize(f'data_{codec}.parquet')
    print(f'{codec:8s}: {size:,} bytes')

Dataset partitioning

Partitioning splits data into separate files by column value — ideal for time-series or regional data:

import pandas as pd, numpy as np

n = 100_000
df_large = pd.DataFrame({
    'date':   pd.date_range('2024-01-01', periods=n, freq='1h'),
    'region': np.random.choice(['north', 'south', 'east', 'west'], n),
    'sales':  np.random.uniform(100, 10_000, n),
})
df_large['year']  = df_large['date'].dt.year
df_large['month'] = df_large['date'].dt.month

# Write partitioned (creates a directory tree)
df_large.to_parquet(
    'sales_partitioned/',
    partition_cols=['year', 'month'],
    engine='pyarrow',
)
# Structure: sales_partitioned/year=2024/month=1/xxx.parquet
#                                           /month=2/xxx.parquet  ...

# Read only March — loads only those partition files
df_march = pd.read_parquet(
    'sales_partitioned/',
    filters=[('month', '==', 3)],
)
print(f"March rows: {len(df_march):,}")

Inspecting schema and metadata

import pyarrow.parquet as pq

meta = pq.read_metadata('employees.parquet')
print(f"Rows:        {meta.num_rows}")
print(f"Row groups:  {meta.num_row_groups}")
print(f"Columns:     {meta.num_columns}")

schema = pq.read_schema('employees.parquet')
print(schema)
# name: string
# age: int64
# salary: double
# active: bool

CSV to Parquet conversion

import pandas as pd

# Direct conversion
df = pd.read_csv('data.csv')
df.to_parquet('data.parquet', compression='zstd')

# Chunked conversion for huge files (> 1 GB)
import pyarrow as pa
import pyarrow.parquet as pq

writer = None
for chunk in pd.read_csv('huge.csv', chunksize=100_000):
    table = pa.Table.from_pandas(chunk)
    if writer is None:
        writer = pq.ParquetWriter('huge.parquet', table.schema)
    writer.write_table(table)

if writer:
    writer.close()
print("Chunked conversion complete")

When to use Parquet

Use Parquet when:

• You have more than 100 000 rows and run analytical queries
• You need exact data type preservation without manual casting
• You work in data pipelines with Spark, Dask, or DuckDB
• Disk space or cloud storage costs matter

Prefer CSV / JSON when:

• Human readability or manual editing is required
• Sharing data with non-technical stakeholders
• The dataset is small (< 10 000 rows)
• Universal tool compatibility is the priority

Related conversions

Frequent conversions across the catalogue:

• JPG to PNG
• MP4 to MP3
• PDF to DOCX
• PNG to JPG
• MP3 to WAV
• DOCX to PDF