HDF5 format: scientific hierarchical container — OSM gridded analytics export

HDF5 (Hierarchical Data Format version 5) is a binary self-describing container for scientific data, where a single .h5 file holds a full filesystem-like hierarchy of groups (folders), datasets (n-dimensional arrays), and attributes (metadata). The format was born in 1998 at NCSA (National Center for Supercomputing Applications) at the University of Illinois as a complete rewrite of the earlier HDF4, and since 2006 it has been developed by the non-profit HDF Group under a BSD-style license. Today HDF5 is the standard container for CMIP6 climate models, NASA EOSDIS satellite data (MODIS, ICESat-2, GRACE), LHC offline data, LAMMPS molecular dynamics, BIDS-HDF5 neuroimaging. In geoinformatics HDF5 serves multidimensional raster cubes, DEM models in 3D datasets, gridded climate analytics. On osm2cdr.ru hdf5_exporter.py takes gridded analytics from PostGIS (POI density, livability index, scoring grids), folds them via numpy into multidimensional arrays, and writes a .h5 file with /city/{name}/metric/{name} hierarchy — ready for downstream analysis in xarray, MATLAB, IDL, R.

History: from NCSA 1988 to HDF Group 2026

HDF5's story starts in 1988 at NCSA — the same supercomputing lab at the University of Illinois that later birthed Mosaic, the first graphical web browser. The team faced a prosaic problem: scientists on a Cray Y-MP generated terabytes of numerical results in many binary formats, and data exchange between research groups turned into converter hell. The solution was HDF4 (Hierarchical Data Format version 4) — a single self-describing container that could hold arrays, tables, images, and metadata in one file.

HDF4 lived for a decade — NASA picked it as the primary format for EOS (Earth Observing System), and MODIS Terra/Aqua products still ship in HDF4 today (.hdf files with the MODIS-EOS profile). But by the mid-1990s it became clear the architecture didn't scale: 32-bit offsets capped files at 2 GB, the API was clunky (separate function families per object type — SD, VG, AN), and parallel I/O wasn't supported. NCSA made a radical call — rewrite everything from scratch.

HDF5 1.0 shipped in November 1998 and instantly fixed three predecessor pain points. First, 64-bit offsets removed the 2 GB limit — files could now grow into the exabyte range. Second, the API was unified: instead of parallel interfaces per object type, a single "groups + datasets + attributes" model emerged, working through uniform H5O_* functions. Third, parallel I/O via MPI arrived — a mandatory feature for supercomputing where hundreds of nodes simultaneously write to a single file.

In 2006 the development team spun off from NCSA into the independent non-profit HDF Group headquartered in Champaign, Illinois. That secured stable long-term development — NCSA as an academic body couldn't guarantee multi-decade support, and NASA, NOAA, and DOE needed exactly those guarantees. Since then the HDF Group has released major versions every 2-3 years; as of 2026 the active line is 1.14, with an experimental 2.0 in development.

In parallel with its own popularity, HDF5 became the technological foundation for other standards. NetCDF-4 (2008) is essentially HDF5 with mandatory CF-Conventions for climatology. MATLAB has stored its .mat files as HDF5 since version 7.3. Keras long used HDF5 (.h5) as the canonical neural-network weights format before the industry shifted to the safer TensorFlow SavedModel and PyTorch state_dict.

Inside the file: groups, datasets, attributes

HDF5 is essentially a filesystem inside a file. The root group / contains subgroups and datasets; subgroups can hold more subgroups and datasets — hierarchy of arbitrary depth. Each object (group, dataset) has attributes — key-value pairs for metadata.

Group. A container for other objects, analogous to a folder in a filesystem. Holds datasets, other groups, soft-links and hard-links to objects elsewhere in the file. The root group is / (like Unix root). A path to a dataset looks like /climate/temperature/2024 — a direct filesystem-path analog.

Dataset. The main data carrier — an n-dimensional array of one type (int8/16/32/64, float32/64, complex, compound struct, variable-length string). Each dimension size is set at creation; dimensions can be unlimited (extendable). Inside a dataset, data is stored in one of three layouts: contiguous (one block), chunked (square/cubic chunks for compression and random access), compact (for small <64 KB datasets inlined in the header).

Attribute. Small key-value pairs attached to a group or dataset. Used for metadata — units = "meters", crs = "EPSG:4326", created_by = "osm2cdr". Attributes are essentially small datasets that HDF5 stores specially for fast access.

Dataspace. Describes the dataset's dimensions — shape (e.g. (360, 720, 12) for monthly globe-grid temperature), maxshape (for extendable), and logical structure. Hyperslab selection allows reading arbitrary slices — "Russia only + January 2024 only".

Chunking. Large datasets break into square/cubic chunks (typically 1024×1024 for 2D, 64×64×64 for 3D), which are stored and compressed independently. That's the key to efficient operation: reading an arbitrary slice decompresses only the touched chunks, not the whole dataset.

Compression. Each chunk is compressed independently by one of the filters: gzip (universal, slow), szip (fast, historically proprietary, now open), lzf (fast lossless for h5py), blosc (multi-algorithm). Plus shuffle filter (reorders bytes for better gzip efficiency on float data).

Parallel I/O. HDF5 supports MPI-IO: dozens or hundreds of supercomputer processes write into one file simultaneously, each to its own dataspace region. That's critical for climate models — CMIP6 runs on 1000+ cores, with each core delivering its slice of a 4D grid.

HDF5 vs NetCDF-4 vs Zarr

Three formats for scientific multidim arrays often get confused, but the differences are rigid.

HDF5 — the most general. Any hierarchy, any dtypes, any names. Single binary file. The most mature — 28 years in production, best-in-industry parallel I/O.

NetCDF-4 — HDF5 with constraints. Under the hood, a genuine HDF5 file (h5py opens it), but on top sit CF-Conventions: mandatory dimension variables (time, lat, lon), standardized variable names (standard_name), fixed unit notations. That makes NetCDF-4 a strictly climate-tooling format.

Zarr — cloud-native alternative. Not a single file, but a directory with chunks as separate files on object storage (S3, R2). HDF5 in the cloud needs serialization via Kerchunk (HDF5 → JSON virtual zarr), Zarr is cloud-native by design. But Zarr is younger (2017), parallel I/O weaker, ecosystem smaller.

The choice is simple: if data is "in HPC center, parallel writes from 100 nodes, 30-year archive" — HDF5. If "climate data with CF-Conventions" — NetCDF-4. If "S3 bucket for cloud analytics" — Zarr.

Strengths and weaknesses

Strengths. Hierarchy of arbitrary depth inside one file — pack the whole project dataset (dozens of arrays of various sizes) into one .h5. Industrial-grade parallel I/O — the only format for PB-scale supercomputing applications. Per-chunk compression — random access without unpacking the whole dataset. Self-describing — dtype, shape, attributes all inside the file, no external JSON schemas needed. 28 years of stability — files from 1998 read by current libraries. Any dtype support — int/float/complex/compound struct/variable-length string. Open-source BSD-style — no vendor lock-in.

Weaknesses. Not cloud-native — a single binary file maps poorly onto object storage without serialization (Kerchunk). Binary, not human-readable — need h5py / HDFView / specialized tools, text diff doesn't work. Big and heavy — the HDF5 C library is 6 MB, with dependencies ~50 MB in a Docker image. Complex concurrent-write model — without MPI you can't write to one file from multiple processes. Overkill for simple 2D rasters — use GeoTIFF for those. Legacy from szip licensing — old files may use szip compression, which until 2017 was under a commercial license for production use (now open, but legacy still surfaces).

Use cases

Climate models. CMIP6 (Coupled Model Intercomparison Project Phase 6) — 100+ global climate models from MPI, NCAR, NOAA centers. Each model run produces petabytes of 4D data (time × level × lat × lon × variables). All stored as NetCDF-4 on top of HDF5. ECMWF's ERA5 reanalysis — 5 PB of historical meteorology, all NetCDF-4/HDF5.

Satellite remote sensing. NASA EOSDIS (Earth Observing System Data and Information System) — petabytes of data from MODIS, ICESat-2 (laser altimetry of polar caps), GRACE (gravity field), ATLAS. Most in HDF5 or HDF4-EOS (legacy variant). ESA's Sentinel partially in SAFE-format with XML+JP2, partially NetCDF.

Particle physics. LHC experiments (Large Hadron Collider) — ATLAS, CMS — generate ~50 PB of raw data per year. Reconstructed data sits in ROOT format (CERN-specific), but secondary analysis format is HDF5 for portability.

Molecular dynamics. LAMMPS, GROMACS, AMBER — protein and materials simulators. Trajectory files (atom positions across 10⁹ timesteps) stored as HDF5 (.h5md — HDF5 for Molecular Data subset).

Neuroimaging. BIDS (Brain Imaging Data Structure) — the fMRI / MEG / EEG data standard. The BIDS-HDF5 extension stores multi-subject experiments in one hierarchy /subject-001/run-01/....

Machine learning (legacy). Until 2020 Keras used model.save('weights.h5') as the canonical format. TensorFlow has since moved to SavedModel (protection against arbitrary code execution in HDF5), PyTorch — to state_dict in its own .pt. But HDF5 still appears in legacy projects.

Geo analytics (our case). Gridded analytics on top of OSM — POI density heatmaps, scoring rasters (Walk Score, Green Score), DEM cubes (elevation+slope+aspect). HDF5 gives a self-describing container for all layers of one study.

Software for HDF5

h5py (Python). Free, BSD-licensed. The canonical Python binding to HDF5. NumPy-native — f['/dataset'][:] returns a numpy array. Install with pip install h5py. Basic usage: f = h5py.File('out.h5', 'w'); f.create_dataset('grid', data=arr, compression='gzip'). Mature, actively maintained, used in every Python scientific project.

HDFView. Free (open-source). The official GUI from HDF Group for browsing and lightweight editing of HDF5. Tree-view of hierarchy, table-view for 2D datasets, graph-view for 1D, image-view for image datasets. Cross-platform (Java-based). Ideal for quick file inspection.

MATLAB. Paid. Since version 7.3 (2006) saves .mat files as HDF5. Functions h5create, h5write, h5read. Convenient hierarchical browser in the Workspace pane.

R rhdf5. Free (Bioconductor package). R binding for HDF5. Used in bioinformatics for single-cell RNA-seq (.h5ad AnnData files — HDF5 subset).

IDL. Paid (NV5 Geospatial, ex Harris/Exelis). The standard in remote sensing — most NASA EOS workflows are written in IDL. Functions H5F_OPEN, H5D_OPEN, H5D_READ.

NCO / CDO. Free CLI tools for NetCDF (and HDF5 through the netcdf-4 backend). ncdump -h file.h5 — shows structure. cdo info file.nc — for NetCDF-4.

GDAL. Free. HDF5 raster driver — opens HDF5 datasets as raster bands (for image-like data). Doesn't work with arbitrary hierarchy — only with specific scientific products (NASA HDF-EOS, JAXA HDF5). QGIS via GDAL can open some HDF5 datasets.

PyTables. Free (BSD). An alternative Python binding to HDF5 with a table-oriented API (lookup, indexing, queries). Handy for tabular datasets, whereas h5py is array-oriented.

Panoply. Free from NASA GISS. A visualizer for NetCDF-4 (HDF5-based) climate datasets. Maps, time-series, slicing 4D cubes.

Pipeline in OSM2CDR

Our hdf5_exporter.py takes gridded analytics from PostGIS and folds them into a hierarchical HDF5 file. Flow:

Step 1: PostGIS gridded query. For the given bbox, a regular grid is built (default 256×256 cells). For each cell, SQL aggregations run: COUNT(poi.id) for POI density, AVG(walk_score) for accessibility, SUM(building.area) / cell_area for density. The result is a numpy array of the same shape as the grid.

Step 2: Multi-metric assembly. If the user picked several metrics (POI density + Walk Score + Green Score + Livability), results are stacked into a 3D array (metrics, height, width) — for convenient slicing.

Step 3: h5py write. A .h5 file is created with the hierarchy: - / (root group) — attributes: bbox, crs, grid_resolution, created_at, source - /city/{slug}/ — group per city (if bbox covers several) - /city/{slug}/density_poi — 2D float32 dataset, chunked 64×64, gzip-compressed - /city/{slug}/walk_score — 2D uint8 dataset, chunked 64×64 - /city/{slug}/heatmap_temporal — 3D dataset (12 months × height × width) if temporal analytics is selected

Each dataset gets attributes: units, valid_min, valid_max, description. That makes the file self-describing — the user via h5py.File('export.h5').visit(print) immediately sees the whole structure with no external doc.

Step 4: Validation. The file is validated via h5dump -H export.h5 — checks the header is parseable — and h5stat export.h5 — statistics on datasets, attributes, sizes. If the file is assembled correctly, it ships; otherwise the exporter returns 500 with a detailed error.

The limit — gridded analytics for a bbox up to 100×100 km on the free tier, up to 500×500 km on pro. Beyond that, performance degrades due to PostGIS query time, not HDF5 itself (HDF5 calmly handles PB-scale).

FAQ

How is HDF5 different from NetCDF-4? NetCDF-4 is HDF5 with extra constraints: mandatory dimension variables, CF-Conventions for metadata, standardized names. A NetCDF-4 file is a valid HDF5 (h5py opens it), but not every HDF5 is a valid NetCDF-4. Use NetCDF-4 if you work in climatology/oceanography (interop with CDO, NCO, xarray). Use HDF5 if you need arbitrary hierarchy for general scientific computing.

How do I open an .h5 file? The simplest path — HDFView GUI from HDF Group (free, cross-platform). In Python — h5py.File('file.h5', 'r'). In MATLAB — h5info('file.h5') for structure, h5read('file.h5', '/path/to/dataset') for data. On the CLI — h5dump -H file.h5 for headers, h5ls -r file.h5 for recursive listing.

Can I use HDF5 as an S3-friendly cloud format? Technically — no. An HDF5 file is a single binary blob; on read the client makes many random reads to different file regions. Object storage (S3, R2) handles range requests, but the HDF5 library isn't tuned for high-latency operations. The fix — Kerchunk: a tool that indexes an HDF5 file on S3 and produces virtual Zarr metadata (JSON), letting fsspec/xarray read HDF5 chunks as Zarr chunks. Alternative — convert to Zarr at cloud upload time.

HDF5 for machine learning — still OK? It used to be. Keras saved weights as .h5 until 2020. Now the industry has moved away: HDF5 supports arbitrary Python pickles in attributes, which posed a security risk (loading someone else's model → arbitrary code execution). TensorFlow shifted to SavedModel (Protocol Buffers + assets directory), PyTorch — to state_dict in .pt. For intermediate ML data (datasets, embeddings) HDF5 is still used via PyTables, but new projects start on Parquet/Arrow.

Which compression is best for HDF5? gzip level 4-6 — the golden middle: 2-4× compression on climate floats with reasonable CPU cost at write and read. lzf (h5py-specific) — 2-3× faster than gzip, but weaker compression. blosc — best-in-class for float arrays (5-10× compression + fast decompression), but needs the blosc-hdf5 plugin. szip — historical NASA standard, fast, but poor cross-tool support. For geodata we use gzip + shuffle filter — typically 2-3× compression on POI density arrays.

Related

• NetCDF — scientific climate format since 1989
• Zarr — cloud-native multidimensional arrays
• GeoTIFF — raster geodata format since 1995
• GeoArrow — columnar geodata on Apache Arrow

Sources

• HDF Group official site — hdfgroup.org
• HDF5 User Guide — docs.hdfgroup.org/hdf5/v1_14/_u_g.html
• h5py documentation — docs.h5py.org
• NASA HDF-EOS portal — hdfeos.org
• Kerchunk (HDF5 → virtual Zarr) — github.com/fsspec/kerchunk
• HDFView GUI — hdfgroup.org/downloads/hdfview