HDF5 Format: Scientific Big Data Storage Explained

HDF5 Format: Scientific Big Data Storage Explained

HDF5 (Hierarchical Data Format version 5) is a self-describing binary file format designed for storing and managing large, complex datasets. Developed by the HDF Group (originally at the National Center for Supercomputing Applications), HDF5 is the dominant format in scientific computing, aerospace engineering, genomics, climate modeling, particle physics, and increasingly in machine learning. When NASA stores satellite telemetry, CERN archives particle collision data, or a deep learning framework saves a trained model, HDF5 is frequently the format of choice.

The "hierarchical" in its name refers to its filesystem-like internal organization: an HDF5 file is a self-contained container in which datasets, metadata, and complex nested structures coexist in a tree of groups and datasets, all browsable without external documentation.

Why HDF5 Exists

Before HDF5, scientific data was typically stored in one of three ways:

1. Flat binary files — fast but completely opaque without accompanying documentation describing the data layout, dimensions, units, and provenance
2. Text/CSV files — human-readable but orders of magnitude slower to read and write, with no support for multi-dimensional arrays
3. Format-specific solutions — NetCDF (climate), FITS (astronomy), proprietary vendor formats — each requiring specialized tools

HDF5 solved this fragmentation by providing a general-purpose container format that is self-describing (all metadata is embedded alongside the data), supports arbitrarily large arrays of any numeric type, and is accessible through libraries in every major scientific programming language.

HDF5 File Structure

An HDF5 file is organized as a directed acyclic graph (DAG) that resembles a filesystem:

/                              ← root group
├── experiment/                ← group
│   ├── metadata               ← group
│   │   └── parameters         ← dataset (structured dtype)
│   ├── raw_data               ← dataset (3D float32 array)
│   └── processed/             ← group
│       ├── spectra            ← dataset (2D float64 array)
│       └── timestamps         ← dataset (1D int64 array)
├── calibration/               ← group
│   ├── dark_frame             ← dataset (2D uint16 array)
│   └── flat_field             ← dataset (2D float32 array)
└── results/                   ← group
    ├── classifications        ← dataset (1D string array)
    └── confidence_scores      ← dataset (1D float32 array)

The two fundamental building blocks are groups (containers, like directories) and datasets (arrays of data, like files). Both can carry attributes — small metadata items attached directly to the object.

Datasets: Multi-dimensional Arrays

An HDF5 dataset is an N-dimensional array of a defined datatype. The datatype can be:

• Integer types: int8, int16, int32, int64, uint8, uint16, uint32, uint64
• Floating point: float32 (single), float64 (double), float128 (quad, platform-dependent)
• Complex numbers: complex64, complex128
• Strings: fixed-length or variable-length UTF-8
• Compound types: C-struct-like records with named fields of mixed types
• Enumerated types: integer codes mapped to named values
• Arrays and variable-length types: arrays of arrays, variable-length sequences
• References: pointers to other datasets or groups within the same file

Chunking

By default, HDF5 stores datasets as contiguous blocks on disk. For large datasets — especially those that will be read in slices — chunking breaks the dataset into fixed-size chunks that are stored independently:

import h5py
import numpy as np

with h5py.File('experiment.h5', 'w') as f:
    # Create a large 3D dataset, chunked in 64×64×64 blocks
    ds = f.create_dataset(
        'raw_data',
        shape=(1024, 1024, 512),
        dtype='float32',
        chunks=(64, 64, 64),
        compression='gzip',
        compression_opts=4,
    )
    # Write a slice
    ds[256:320, 256:320, 128:192] = np.random.rand(64, 64, 64).astype('float32')

Chunking enables several important capabilities:

• Partial reads — read only the slices you need without loading the entire dataset into memory
• Compression — each chunk is compressed independently, enabling on-the-fly decompression of accessed chunks
• Parallel I/O — different processes can read/write different chunks simultaneously
• Resizable datasets — chunked datasets can be extended in any dimension (using maxshape=(None, None, None))

Compression Filters

HDF5 supports a pipeline of data transformation filters applied to each chunk before storage:

The shuffle filter reorders bytes in multi-byte values (e.g., the four bytes of a float32 are grouped by position across all values in the chunk), dramatically improving compression ratios for numeric data — often 2-5× better than without shuffling.

Python h5py Usage

The h5py library is the standard Python interface to HDF5:

import h5py
import numpy as np
from datetime import datetime

# Create a new HDF5 file
with h5py.File('results.h5', 'w') as f:

    # File-level attributes
    f.attrs['creator'] = 'experiment_pipeline v2.3'
    f.attrs['created'] = datetime.utcnow().isoformat()
    f.attrs['hdf5_version'] = h5py.version.hdf5_version

    # Create groups
    raw = f.create_group('raw_data')
    proc = f.create_group('processed')
    meta = f.create_group('metadata')

    # Store a large array with chunking + compression
    signal = raw.create_dataset(
        'signal',
        data=np.random.randn(10000, 256).astype('float32'),
        chunks=(500, 256),
        compression='gzip',
        compression_opts=6,
        shuffle=True,
    )
    signal.attrs['units'] = 'microvolts'
    signal.attrs['sample_rate_hz'] = 1000
    signal.attrs['channel_count'] = 256

    # Store variable-length strings
    channel_names = [f'CH{i:03d}' for i in range(256)]
    dt = h5py.special_dtype(vlen=str)
    raw.create_dataset('channel_names', data=channel_names, dtype=dt)

    # Store a compound (structured) dataset
    dtype = np.dtype([
        ('timestamp', 'int64'),
        ('event_type', 'S16'),
        ('value', 'float64'),
    ])
    events = np.array([
        (1710000000000, b'stimulus', 1.0),
        (1710000001500, b'response', 0.7),
        (1710000003200, b'stimulus', 1.0),
    ], dtype=dtype)
    meta.create_dataset('events', data=events)

    # Store a resizable dataset (append-friendly)
    ds = f.create_dataset(
        'streaming_data',
        shape=(0, 64),
        maxshape=(None, 64),
        dtype='float32',
        chunks=(1000, 64),
    )

# Read back selectively
with h5py.File('results.h5', 'r') as f:
    # Read only rows 1000-2000 (1 GB data, only reading 8 MB)
    subset = f['raw_data/signal'][1000:2000, :]
    print(f"Shape: {subset.shape}, dtype: {subset.dtype}")

    # Read attributes
    sr = f['raw_data/signal'].attrs['sample_rate_hz']
    print(f"Sample rate: {sr} Hz")

    # Traverse the entire file tree
    def print_tree(name, obj):
        indent = '  ' * name.count('/')
        if isinstance(obj, h5py.Dataset):
            print(f"{indent}DATASET {name}: shape={obj.shape}, dtype={obj.dtype}")
        else:
            print(f"{indent}GROUP   {name}/")

    f.visititems(print_tree)

Appending to a Resizable Dataset

with h5py.File('streaming.h5', 'a') as f:
    ds = f['streaming_data']
    new_batch = np.random.randn(500, 64).astype('float32')
    current_size = ds.shape[0]
    ds.resize(current_size + 500, axis=0)
    ds[current_size:current_size + 500, :] = new_batch
    print(f"Dataset now has {ds.shape[0]} rows")

HDF5 in Deep Learning

HDF5 is used extensively in machine learning:

Keras/TensorFlow: the .h5 or .keras model format uses HDF5 to store layer weights, architecture JSON, and optimizer state:

# Save a Keras model
model.save('model.h5')

# Load a Keras model
from tensorflow import keras
model = keras.models.load_model('model.h5')

# Inspect the HDF5 structure
with h5py.File('model.h5', 'r') as f:
    print(list(f.keys()))  # ['model_config', 'model_weights', 'training_config']
    f.visititems(lambda n, o: print(n) if isinstance(o, h5py.Dataset) else None)

Large training datasets: when datasets are too large for memory, HDF5 datasets serve as memory-mapped arrays:

with h5py.File('imagenet_train.h5', 'r') as f:
    images = f['images']   # shape: (1281167, 3, 224, 224), float32
    labels = f['labels']   # shape: (1281167,), int32
    # Iterate in mini-batches without loading all 500 GB
    for i in range(0, len(labels), 32):
        batch_x = images[i:i+32]
        batch_y = labels[i:i+32]
        # train on batch...

Parallel HDF5

For high-performance computing (HPC), HDF5 supports parallel I/O via MPI (Message Passing Interface):

import h5py
from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

# Each MPI rank writes its own portion of a shared dataset
with h5py.File('parallel_output.h5', 'w', driver='mpio', comm=comm) as f:
    ds = f.create_dataset('results', shape=(size * 1000, 256), dtype='float32')
    # Each rank writes rows [rank*1000 : (rank+1)*1000]
    ds[rank*1000:(rank+1)*1000, :] = np.random.randn(1000, 256).astype('float32')

HDF5 vs NetCDF vs Zarr

NetCDF4 is actually built on top of HDF5 — it adds conventions for geophysical data (coordinate variables, CF conventions) while using HDF5 as its storage engine. Zarr is a newer format designed for cloud object stores, where HDF5's single-file model creates contention bottlenecks.

Conclusion

HDF5's combination of self-describing structure, arbitrary-type multi-dimensional arrays, flexible compression, and proven scalability to petabyte-scale datasets has made it indispensable in scientific computing for over two decades. Its hierarchical organization makes complex experimental datasets navigable without external documentation, its chunking and compression features make it practical for datasets far larger than available RAM, and its parallel I/O support enables HPC workflows that process terabytes in minutes. Whether you are storing neural network weights, satellite imagery, genomic sequencing data, or particle physics collision records, HDF5 provides the foundation for reliable, efficient, and self-documenting data storage.

Related conversions

Frequent conversions across the catalogue:

• JPG to PNG
• MP4 to MP3
• PDF to DOCX
• PNG to JPG
• MP3 to WAV
• DOCX to PDF