HDF5: Hierarchical Data Format for Scientific Computing

HDF5: Hierarchical Data Format for Scientific and High-Performance Computing

What Is HDF5?

HDF5 — Hierarchical Data Format version 5 — is a versatile, self-describing file format and library designed for storing and managing large, complex collections of scientific data. Developed by the National Center for Supercomputing Applications (NCSA) in the 1990s and now maintained by The HDF Group, HDF5 is the de facto standard for scientific data in fields ranging from computational physics and genomics to machine learning and satellite remote sensing.

Files use the .h5 or .hdf5 extension (occasionally .he5 for HDF-EOS satellite data). Unlike simple CSV or binary blob formats, HDF5 files are self-describing: the format carries its own metadata — dimension sizes, data types, units, coordinate systems, custom attributes — alongside the raw data, making files interpretable without external documentation.

The HDF5 Data Model

HDF5 organizes data in a POSIX-like filesystem hierarchy:

file.h5
├── /                           ← Root group
│   ├── simulation/             ← Group (like a directory)
│   │   ├── velocity            ← Dataset (array of float32, shape 1000×100×100)
│   │   ├── pressure            ← Dataset (array of float64, shape 1000×100×100)
│   │   └── temperature         ← Dataset
│   ├── metadata/
│   │   ├── authors             ← Dataset (array of strings)
│   │   └── created             ← Attribute on group: "2024-03-15"
│   └── mesh/
│       ├── x_coordinates       ← Dataset (float64, shape 101)
│       ├── y_coordinates       ← Dataset (float64, shape 101)
│       └── connectivity        ← Dataset (int32, shape N×4)

Groups

Groups are containers (analogous to directories). They hold other groups and datasets, and can carry attributes — small pieces of metadata (strings, scalars, small arrays) stored directly on the group.

Datasets

Datasets are the primary data containers — N-dimensional arrays of a single datatype. Properties:

• Shape: any number of dimensions, any sizes (a scalar, a 1D vector, a 5D array…)
• Datatype: fixed-width integers (int8 to int64, uint8 to uint64), IEEE floats (float16, float32, float64), complex numbers, strings (fixed or variable length), compound types, enumerated types, array types
• Chunking: datasets can be divided into fixed-size chunks stored independently — critical for efficient partial I/O on large arrays
• Compression: per-dataset, applied at chunk level (zlib/DEFLATE, LZW, SZIP, Blosc, LZ4, Bitshuffle)
• Fill value: default value for unwritten regions
• Extendable dimensions: datasets can grow along marked dimensions after creation

Attributes

Attributes are small metadata items attached to any group or dataset: units, coordinate reference systems, creation timestamps, instrument parameters, calibration coefficients. They are not chunked or compressed and should be small (< 64 KB).

Working with HDF5 in Python: h5py

h5py is the standard Python interface:

import h5py
import numpy as np

# Create an HDF5 file
with h5py.File('simulation.h5', 'w') as f:
    # Create a group
    grp = f.create_group('velocity')

    # Create a chunked, compressed dataset
    data = np.random.randn(1000, 256, 256).astype(np.float32)
    dset = grp.create_dataset(
        'u_component',
        data=data,
        chunks=(100, 64, 64),   # chunk shape
        compression='gzip',
        compression_opts=6      # gzip level 1-9
    )

    # Attach attributes
    dset.attrs['units'] = 'm/s'
    dset.attrs['long_name'] = 'Zonal velocity component'
    dset.attrs['missing_value'] = -9999.0

    # Create a variable-length string dataset
    f.create_dataset(
        'authors',
        data=np.array(['Dr. Smith', 'Prof. Jones'], dtype=h5py.string_dtype())
    )

# Read an HDF5 file
with h5py.File('simulation.h5', 'r') as f:
    # List top-level groups
    print(list(f.keys()))           # ['velocity']

    # Access a dataset (lazy — data not loaded yet)
    dset = f['velocity/u_component']
    print(dset.shape, dset.dtype)   # (1000, 256, 256) float32

    # Slice-read: only read a chunk, not the full array
    subset = dset[0:10, 100:200, 100:200]   # shape: (10, 100, 100)

    # Read attributes
    print(dset.attrs['units'])       # b'm/s'

    # Walk the entire file tree
    def print_structure(name, obj):
        print(name, type(obj).__name__)
    f.visititems(print_structure)

Chunking and Compression: Performance Critical

Chunking is the most important HDF5 configuration choice. Rules of thumb:

• Chunk shape should match access pattern: if you read time slices [t, :, :], chunk as (1, H, W) not (T, 1, 1); if you read spatial subsets [:, y, x], use (T, 1, 1) chunks
• Chunk size ~1 MB: the sweet spot for I/O — big enough to amortize overhead, small enough to avoid reading unnecessary data
• Compression is applied per-chunk: only compressed chunks are decompressed at read time
• gzip/DEFLATE: universal, ~40–70% size reduction on floating-point data with predictor shuffling
• LZ4 / Blosc: much faster decompression, slightly less compression — better for interactive workloads
• Bitshuffle + LZ4: optimal for scientific floating-point data (rearranges bit planes before compressing)

# Optimal chunking for time-first access (reading full spatial slices)
dset = f.create_dataset('temperature',
    shape=(10000, 1024, 1024),
    dtype=np.float32,
    chunks=(1, 1024, 1024),      # one full 2D slice per chunk
    compression='lzf',            # fast compression
    shuffle=True                  # byte shuffle improves ratio
)

HDF5 in Major Scientific Ecosystems

NetCDF-4

NetCDF-4 (a superset of NetCDF-3) uses HDF5 as its underlying storage format. Opening a .nc NetCDF-4 file with h5py works — you see the same group/dataset hierarchy.

Machine Learning: Keras / TensorFlow

Keras saves model weights in HDF5 (.h5/.keras):

model.save('model.h5')             # saves architecture + weights
model = tf.keras.models.load_model('model.h5')

Astronomy: FITS Alternative

Many modern large-survey pipelines (Euclid, DESI, Vera Rubin Observatory/LSST) use HDF5 instead of FITS for intermediate data products because of chunking, compression, and parallel I/O capabilities.

Genomics: HDF5 in Bioinformatics

• 10x Genomics single-cell RNA-seq data: .h5 files (Feature-Barcode Matrix HDF5)
• HDF5-backed AnnData (anndata/scanpy): .h5ad files for single-cell analysis
• Variant call format: GVCF data in HDF5 for large cohort genomics

Materials Science: NOMAD, NXS

• NeXus (.nxs): HDF5-based format for neutron and X-ray scattering data (ISIS, SNS, ESRF)
• NOMAD repository: DFT calculation results in HDF5

Parallel HDF5

HDF5 supports parallel I/O via MPI (Message Passing Interface) through the parallel HDF5 library (h5py with mpio driver):

from mpi4py import MPI
import h5py

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

with h5py.File('parallel_output.h5', 'w', driver='mpio', comm=comm) as f:
    dset = f.create_dataset('data', shape=(1000, 1000), dtype=np.float32)
    # Each MPI rank writes its portion
    dset[rank*100:(rank+1)*100, :] = np.random.randn(100, 1000)

This allows petabyte-scale simulations (climate models, cosmological N-body, CFD) to write data from thousands of compute nodes simultaneously to a single HDF5 file on a parallel filesystem.

Command-Line Tools

# Inspect file structure
h5ls -r simulation.h5            # recursive listing
h5dump -n simulation.h5          # names/structure only
h5dump -d /velocity/u_component simulation.h5   # dump a dataset

# Check for errors
h5check simulation.h5

# Repack (recompress/rechunk)
h5repack -i input.h5 -o output.h5 \
  -f /velocity/u_component:CHUNK=1x256x256 \
  -f /velocity/u_component:GZIP=6

Practical Tips

• Always use chunking for datasets > 1 MB — unchunked contiguous datasets cannot be compressed and require reading the full array for any access
• Match chunk shape to read pattern — wrong chunking is the #1 HDF5 performance problem
• Use h5py.File as a context manager (with statement) — ensures proper file closing and flushes I/O buffers
• String encoding: h5py ≥ 3.0 returns bytes by default; use dset.asstr()[...] for Unicode strings
• SWMR (Single Writer Multiple Reader): HDF5 1.10+ supports SWMR mode for reading while writing (useful for live instrument data streams)
• Virtual Datasets (VDS): HDF5 1.10+ allows mapping multiple source datasets into a single virtual dataset — useful for aggregating per-timestep files into a single time-series view
• Inspect with HDFView (free GUI from The HDF Group) or h5web (web-based, no install)

HDF5's combination of hierarchical organization, self-describing metadata, chunking, compression, and parallel I/O makes it the universal container for scientific data at any scale — from a single researcher's experiment to exascale supercomputer simulations.

Related conversions

Frequent conversions across the catalogue:

• JPG to PNG
• MP4 to MP3
• PDF to DOCX
• PNG to JPG
• MP3 to WAV
• DOCX to PDF