NetCDF format: climate + ocean + gridded geo — OSM analytics export to .nc

2026-05-0617 min read

scientificnetcdfclimategriddedcf-conventionsunidata

Export formats: NETCDF

NetCDF (Network Common Data Form) is a self-describing binary format for multidimensional scientific arrays, created in 1989 at the Unidata Program Center (UCAR, Boulder, Colorado). The original goal — machine-independent exchange of atmospheric and oceanographic data between Cray, VAX, Sun, and the microcomputers of the era: one file readable identically everywhere, no endianness surprises. Over 36 years NetCDF became the de facto standard of climate science (CMIP6, ERA5, NCEP reanalysis), satellite remote sensing (MODIS, MERRA-2, Sentinel atmospheric products), and gridded oceanography (ROMS, HYCOM). On osm2cdr.ru netcdf_exporter.py takes OSM analytics (POI density, building density, road density per cell) and packages them into a .nc4 file with CF-Conventions attributes — lat/lon/time axes, units, standard_name. The output is a gridded cube ready for import into xarray / CDO / Panoply for urban research.

Map generation of Novosibirsk city center

History: from Unidata 1989 to NetCDF-4 on HDF5

NetCDF's story starts in 1988 at the Unidata Program Center — a division of UCAR (University Corporation for Atmospheric Research) in Boulder, Colorado. Glenn Davis, Russ Rew, and Steve Emmerson were working on a problem: atmospheric models produced output on a Cray supercomputer (big-endian), while analysis happened on Sun workstations (little-endian) and VAX (mixed). Every lab wrote its own parser for its own format; data exchange between universities was painful.

In 1989 the first public NetCDF-1 shipped — architecturally borrowing from NASA CDF (Common Data Format) but with a focus on machine-independent representation: XDR (eXternal Data Representation) for every numeric type, self-describing metadata inside the file (dimensions, variables, attributes), and an open-source C library under a liberal license. By 1997 NetCDF-3 (Classic format) was out — the stable version still in use today: 2 GB file limit (32-bit offsets), one root-level namespace, simple structure without groups. A 64bit-Offset variant followed shortly for files > 2 GB.

In 2006 Unidata partnered with the HDF Group to redesign the backend on HDF5. NetCDF-4 shipped in 2008. A NetCDF-4 file is an HDF5 file with an extra convention: special _NCProperties / _Netcdf4Dimid attributes, plus restrictions on some HDF5 features. That brought unlimited file size, groups (hierarchical structure inside the file), compound types (record-like structures), variable-length types (vlen, ragged arrays), chunking + compression (gzip/szip/zstd), and parallel I/O via MPI.

Alongside format evolution, a convention layer grew on top: CF-Conventions (Climate and Forecast Metadata Convention) — the standard for attributes in climate / oceanographic / atmospheric NetCDF files. CF-1.0 shipped in 2003; the current CF-1.11 (2024) is dozens of pages of rules: which attributes are required (units, standard_name, long_name), how to describe axes (axis="T" for time, axis="X" for longitude), how to declare CRS (grid_mapping_name), how to encode time as "days since 1970-01-01". CMIP6, ERA5, every NASA atmospheric product, ECMWF reanalysis — all strictly CF-compliant. That's what makes NetCDF not just a format but an entire interoperability ecosystem.

Inside the file: dimensions + variables + attributes + CF

NetCDF's data model — three fundamental abstractions:

Dimensions. Named axes with size. A typical climate cube — time × level × lat × lon. Size is fixed at creation (exception — UNLIMITED dimension, typically time, for appending new timesteps). For example: time=365, level=37, lat=720, lon=1440 for daily ERA5 at 0.25°.

Variables. Multidimensional arrays over dimensions. Each variable has a name (temperature, pressure), dtype (float32, int16, double), an ordered list of dimensions (temperature(time, level, lat, lon)), and attributes. There are data variables (the actual data) and coordinate variables (axis values — e.g. lat has dimension lat and contains [-89.875, -89.625, ..., 89.875]).

Attributes. Key-value metadata attached either to a variable or to the file globally. That's where CF-conventions live:

global attributes:
    Conventions = "CF-1.11"
    title = "OSM POI density grid, Berlin, 2026"
    institution = "osm2cdr.ru"
    source = "OpenStreetMap via osm2pgsql + PostGIS aggregation"
    history = "2026-05-13: created from OSM extract"

variables:
    float lat(lat)
        units = "degrees_north"
        standard_name = "latitude"
        axis = "Y"
    float lon(lon)
        units = "degrees_east"
        standard_name = "longitude"
        axis = "X"
    int time(time)
        units = "days since 2020-01-01"
        standard_name = "time"
        axis = "T"
        calendar = "gregorian"
    float poi_density(time, lat, lon)
        units = "count km-2"
        long_name = "OpenStreetMap POI density per square kilometer"
        _FillValue = -9999.0
        grid_mapping = "crs"
    char crs
        grid_mapping_name = "latitude_longitude"
        longitude_of_prime_meridian = 0.0
        semi_major_axis = 6378137.0
        inverse_flattening = 298.257223563

That set is enough for any CF-aware tool (xarray, CDO, NCO, Panoply, IDV, QGIS) to understand that poi_density is a [time × lat × lon] grid in WGS84, units = count per km², time decoded into datetime objects. From there — slicing, plotting, regridding, climate operators all work out of the box.

Map generation of Sochi city center

NetCDF-3 vs NetCDF-4: classic binary vs HDF5

The main practical split is the format version. They are binary-incompatible, though the APIs are similar.

NetCDF-3 (Classic / 64bit-Offset). The original binary format Unidata designed from scratch. Simple structure: header at file start (XDR-encoded dimensions, variables, attributes), then data in sequential blocks. Strengths — maximum compatibility (read by anything, including Fortran 77 legacy tools, ancient MATLAB R2008, old IDL), small header overhead, simple parser. Weaknesses — 2 GB limit (Classic) or ~16 EB via 64bit-Offset, no compression, no chunking, no groups, no parallel I/O.

NetCDF-4. An HDF5 file with a NetCDF semantic layer on top. Gets all HDF5 features: unlimited size, chunking + compression (gzip default, szip / zstd optional), hierarchical groups (like directories inside the file), compound types, parallel I/O via MPI. Extension .nc4 or just .nc. The file is a valid HDF5 — h5py opens it. Weaknesses — larger overhead (~few KB header), unreadable by NetCDF-3-only tools, slower on small files.

In modern (2026) workflows, NetCDF-4 is the default. The climate community migrated long ago: CMIP6, ERA5, MODIS Level-3, MERRA-2 — all NetCDF-4. NetCDF-3 sticks around for backward compatibility and for very small datasets where compression overhead isn't worth it. Our netcdf_exporter.py writes NetCDF-4 with zlib compression level=4 (balanced).

Ecosystem: ncdump, NCO, CDO, xarray, Panoply

NetCDF is not just a format but a mature ecosystem of tools built up over 36 years.

ncdump / ncgen. Standard CLI utilities from Unidata. ncdump -h file.nc shows the header (dimensions, variables, attributes) in human-readable CDL form. ncdump file.nc — full data dump. ncgen is the reverse: text CDL → binary .nc. Used for quick inspection and generating test files.

NCO (NetCDF Operators). Charlie Zender (UC Irvine), free, BSD-like. A CLI toolkit: ncks (NetCDF Kitchen Sink — extract / subset / convert), ncrcat (concatenate along the record dimension), ncea (ensemble average), ncwa (weighted average), ncap2 (arithmetic with an expression language). Indispensable for batch processing millions of files in climate workflows: ncks -v temperature -d time,0,30 input.nc output.nc — extract only the first 30 temperature timesteps.

CDO (Climate Data Operators). Max Planck Institute for Meteorology, free, GPLv2. NCO's competitor with a focus on climate analysis: cdo remapbil,r360x180 input.nc output.nc — bilinear regrid to a 1°×1° grid; cdo yearmean input.nc output.nc — annual mean; cdo selname,temperature input.nc output.nc — variable selection. 600+ operators, chainable through pipes.

xarray (Python). The modern Python interface for multidimensional labeled arrays, a natural wrapper over NetCDF-4. xr.open_dataset('era5.nc') → a Dataset with all variables, coords, attrs. Label-based slicing: ds.sel(time='2024-01-15', lat=slice(50, 60)). Lazy loading via Dask for terabyte-scale data. Today's de facto standard in the Python climate stack.

Panoply (NASA GISS). Free Java GUI viewer for NetCDF / HDF / GRIB. Plot 2D slices, animations through the time dimension, colormaps, contours. Indispensable for quick-look — open the file, see what's inside, sanity-check units. Runs on Windows / macOS / Linux.

NCL (NCAR Command Language). A high-level interpreted language from NCAR for climate analysis with native NetCDF support. Gradually being replaced by Python+xarray, but there's a huge legacy codebase in the climate community.

netcdf4-python. Low-level Python binding over the libnetcdf C library. Used inside xarray, but also accessible directly for fine control over compression, chunking, and parallel writes.

MATLAB, IDL, R. All have native NetCDF readers. MATLAB ncread('file.nc', 'temperature'), R ncdf4::nc_open(), IDL NCDF_OPEN(). Legacy but still used in academia.

GDAL. Supports NetCDF as a raster driver (for 2D georeferenced subsets). gdal_translate -of GTiff NETCDF:"file.nc":temperature output.tif — extract one time slice as GeoTIFF. Not for full N-D workflows, but for integration with classical GIS.

OPeNDAP. Not a library but a protocol — lets you read NetCDF files from a remote server by URL without downloading. THREDDS Data Server, ERDDAP — standard backends. xr.open_dataset('https://thredds.../era5.nc') transparently works as a local file, fetching only the requested slices.

NetCDF vs HDF5 vs Zarr: when to pick which

Three formats for multidimensional scientific arrays — each fills a niche.

NetCDF. Domain-specific (climate, ocean, atmosphere, gridded geo). Mature CF-conventions ecosystem, specialized tools (CDO, NCO, NCL), all major datasets from meteorological agencies published this way. Best fit when working with climate models, satellite atmospheric products, oceanographic reanalysis, or when you need interoperability with the CMIP6 / ERA5 / NCEP ecosystems. Single-file, desktop-era storage model.

HDF5. A generic hierarchical container. Any structure — groups + datasets + arbitrary attributes. NetCDF-4 is a formal subset of HDF5 with CF semantics. HDF5 is used widely in machine learning (older models were saved as .h5), bioimaging (Imaris, BigDataViewer), NASA EOS data (MODIS L1/L2). If you need maximum structural flexibility and don't need CF semantics — HDF5. If the data is climate — NetCDF with CF.

Zarr. Cloud-native heir to HDF5 / NetCDF. Each chunk — a separate S3 object, parallel read/write without a global lock. NumPy-compatible API. CF-conventions over Zarr (xarray engine='zarr') work identically to NetCDF. Best fit for cloud workflows on S3/GCS, parallel writes from a Dask cluster, terabyte/petabyte datasets. Downsides — a directory tree, not a single file (ZipStore is a workaround), no mature CLI tooling at the CDO/NCO level.

Practical rule: data generated once and read often, single laptop / HPC node, needs CDO/NCO operations → NetCDF. Data updated in parallel from many writers, cloud-hosted, needs HTTP range streaming → Zarr. Generic non-geo scientific data with arbitrary hierarchy → HDF5.

NetCDF vs Zarr: one format, two storage models

Worth calling out: at the data-model level NetCDF-4 and Zarr are nearly identical. Same dimensions, same variables, same CF-conventions on top. xarray switches between them in one line:

ds.to_netcdf('output.nc')    # NetCDF-4 (HDF5) — single file
ds.to_zarr('output.zarr')    # Zarr — directory of chunks

The difference is the storage model. NetCDF-4 is a monolithic HDF5 file: one file, a global B-tree index, POSIX file API for reads. Ideal for locally-mounted disk, poor on S3 (range reads expensive, no parallel writes — file lock). Zarr is chunks as separate S3 objects, JSON metadata, parallel writes are trivial (each chunk = an independent PUT).

The Pangeo project in the early 2020s essentially cloned NetCDF semantics into Zarr precisely for the cloud migration: same CF-conventions, same xarray API, but it now runs on S3 without pain. CMIP6 publishes data in both formats in parallel. For archive download — NetCDF; for in-cloud analysis — a Zarr copy.

In osm2cdr we ship both: netcdf_exporter.py writes NetCDF-4 for classical climate / GIS workflows, zarr_exporter.py (beta) writes Zarr for cloud-native experiments. Output semantics are identical — only the storage layout differs.

Use cases

Climate model output. CMIP6 (Coupled Model Intercomparison Project) — the global benchmark for climate models. ~100 models, ~30 PB of output, all in NetCDF-4 / Zarr with CF-conventions. A scientist via xarray + Dask opens a petabyte-scale ensemble, runs .sel(experiment='ssp585', time=slice('2050', '2100')) — and gets end-of-century warming projections from multiple models at once.

Atmospheric reanalysis. ERA5 from ECMWF — the best global atmospheric reanalysis (1940 — present, hourly, 0.25° grid). 5 PB of NetCDF-4 in the Copernicus Climate Data Store. NCEP/NCAR Reanalysis, MERRA-2 (NASA), JRA-55 (Japan) — all similar. This is the baseline data for climate research, weather forecasting, renewable energy resource assessment.

Satellite remote sensing. MODIS Level-3 products (Terra/Aqua, 2000—present): aerosol optical depth, vegetation indices, sea surface temperature — all NetCDF-4 / HDF-EOS5. Sentinel-3 OLCI/SLSTR atmospheric products, ICESat-2 ATL06/ATL08, Aura OMI ozone — NetCDF-4 ubiquitous. NASA Earthdata Cloud / ESA Copernicus are the entry points.

Oceanography. ROMS (Regional Ocean Modeling System), HYCOM (HYbrid Coordinate Ocean Model), NEMO — all produce NetCDF output. Argo float profiles, GLORYS reanalysis, OISST — all standard NetCDF with CF. Oceanographers live in xarray / CDO / Panoply.

Hydrology. USGS National Water Model output, GLDAS (Global Land Data Assimilation System) — gridded soil moisture, runoff, evapotranspiration in NetCDF-4. NWS forecast products through NOMADS / THREDDS.

Urban gridded analytics (our case). Imagine a researcher analyzing how POI density / building density evolved across Moscow over the last 5 years, by season, in comparison with air-quality reanalysis. The natural structure is a time × variable × lat × lon cube. Through osm2cdr netcdf_exporter.py we aggregate OSM data via PostGIS into grid cells and package them into NetCDF-4 with CF-conventions. The result opens directly in xarray, easily joins with ERA5 air quality via xr.merge() — and becomes a ready dataset for an urban climate paper.

Map generation of Berlin city center

Strengths and weaknesses

Strengths. Self-describing — the file carries its own metadata, no external sidecars required. Machine-independent — a single binary file reads on any architecture since creation (a 1989 NetCDF-1 file opens today unchanged). A huge ecosystem (ncdump, NCO, CDO, xarray, Panoply, NCL, GDAL) built up over 36 years. CF-Conventions are the de facto standard for climate metadata — units, axes, time encoding, CRS unified. Used in literally every major climate / atmospheric / ocean dataset (CMIP6, ERA5, MODIS, MERRA-2). Open spec (Unidata), open library (libnetcdf, BSD-like). NetCDF-4 brings chunking + compression + parallel I/O via MPI for HPC workflows. OPeNDAP lets you work with remote files without download.

Weaknesses. Desktop-era storage model — a single binary file fits poorly into the cloud (S3 range reads expensive, parallel writes need file locking). Cloud-native replacement — Zarr. Not human-readable (binary) — inspection needs ncdump -h. NetCDF-4 requires an HDF5 dependency — installing libhdf5 in a headless Docker is sometimes non-trivial. Not optimized for tabular data (POI lists) — for that go GeoParquet / SQL. CF-Conventions are complex (10+ pages of rules), easy to forget a required attribute — then tools start warning. For editing — chunks must be rewritten (especially NetCDF-3). 2 GB limit in NetCDF-3 Classic, worked around via 64bit-Offset or NetCDF-4.

FAQ

How does NetCDF differ from HDF5? NetCDF-4 is an HDF5 file with additional restrictions and conventions: special attributes (_NCProperties), prohibition of some HDF5 features (e.g., hard links between datasets), CF-conventions for climate metadata. Any NetCDF-4 file is a valid HDF5 file (h5py opens it), but not vice versa: an arbitrary HDF5 may not follow NetCDF semantics. NetCDF-3 is a completely separate format, unrelated to HDF5. More in our HDF5 article.

NetCDF-3 or NetCDF-4 — which to pick? In 2026 — NetCDF-4 by default. Compression cuts size 3–5× for typical climate data, unlimited file size removes the 2 GB limit, groups give logical structure. NetCDF-3 only makes sense when compatibility with very old tools (Fortran 77 legacy, ancient MATLAB) is required. Our netcdf_exporter.py writes NetCDF-4 with zlib level=4.

What are CF-Conventions and are they mandatory? CF (Climate and Forecast Metadata Convention) is the attribute standard for NetCDF files in climate / forecast / earth system data. It specifies required variable attributes (units, standard_name, long_name), how to encode time ("days since 1970-01-01"), how to describe CRS (grid_mapping), how to label axes (axis="T"). Technically — not mandatory (a file without CF is still a valid NetCDF), but without them xarray / CDO / Panoply can't auto-recognize axes and do proper plotting. All professional datasets are CF-compliant. Our exporter writes CF-1.11.

How do I open a .nc file without programming? Panoply (NASA GISS, free Java GUI) is the best choice for quick-look: open, inspect dimensions / variables, draw a 2D map. Alternatives — IDV (NCAR), ncBrowse, or QGIS itself through the GDAL NetCDF driver (Layer → Add Raster Layer → .nc file → pick a subdataset). For command-line quick inspection — ncdump -h file.nc.

How does NetCDF handle time? Time in NetCDF is stored as a numeric value + a units string of the form "<unit> since <reference>": for example "days since 1850-01-01 00:00:00" or "hours since 2024-01-01". An extra calendar attribute specifies the calendar (gregorian, noleap, 360_day for climate models). When read via xarray / netCDF4-python, time auto-decodes into datetime objects. That lets you store time-series from 1850 to 2300 without a single Unix-timestamp limit.

Can NetCDF store vector data (points, lines, polygons)? Technically — yes, through the CF-Conventions Discrete Sampling Geometries (DSG) extension: timeSeries, profile, trajectory features are described via a special instance dimension + coordinate variables. Used for measurements from buoys, weather stations, ship tracks. But for typical GIS vector layers (POI, buildings, roads) NetCDF is unnatural. Pick GeoJSON, Shapefile, GeoParquet, GeoArrow. NetCDF — for rasterized / gridded data.

Pipeline in OSM2CDR

Our netcdf_exporter.py is a four-step pipeline turning an OSM area into a CF-compliant gridded cube:

Step 1: PostGIS aggregation. Take the user's bbox, split into a grid (default 100×100 cells, configurable). For each cell PostGIS computes aggregations: COUNT POI per category, SUM building footprint area, SUM road length, density per km². The SQL query is parameterized and uses ST_Within with a spatial index across grid cells.

Step 2: Stack into a numpy array. Aggregation results land in a 3D numpy array [variable × lat × lon] (or 4D [time × variable × lat × lon] if multiple snapshots are requested — for a future temporal feature). The lat/lon axes are the actual coordinates of the cell centers in WGS84.

Step 3: Wrap in an xarray Dataset. Create an xr.Dataset with coordinates (lat, lon, optionally time) and data variables (poi_density, building_density, road_density, etc.). Attach CF-conventions attributes: standard_name, units, long_name per variable; axis, standard_name for coordinates; grid_mapping for CRS (WGS84 latitude_longitude); global attrs (Conventions="CF-1.11", title, institution="osm2cdr.ru", source, history).

Step 4: Write NetCDF-4. Through ds.to_netcdf('output.nc4', format='NETCDF4', engine='netcdf4', encoding={var: {'zlib': True, 'complevel': 4} for var in ds.data_vars}). The result is ~1–5 MB for a typical city-bbox 100×100×4 variables grid, ready to import into any CF-aware tool.

Validate output:

ncdump -h output.nc4 | head -30
# Should show Conventions = "CF-1.11", correct axes, units

cdo info output.nc4
# CDO recognizes the grid and variables

python -c "import xarray; ds = xarray.open_dataset('output.nc4'); print(ds)"
# xarray decodes time, attaches coords, shows Dataset structure

Conclusion

NetCDF is the best choice if your work lives in climate science, atmospheric / oceanographic research, satellite remote sensing, or gridded environmental data. 36 years of a mature ecosystem (ncdump, NCO, CDO, xarray, Panoply), CF-conventions as the de facto standard, every major climate dataset published exactly this way (CMIP6, ERA5, MODIS, MERRA-2). For cloud-native scaling — Zarr with the same semantics. For generic hierarchical scientific data — HDF5 without CF overhead. For typical OSM maps (vector POI/roads/buildings) — NetCDF is overkill, use GeoJSON or Shapefile.

On osm2cdr.ru netcdf_exporter.py aggregates OSM features into a gridded cube with CF-1.11 attributes — a natural bridge between OpenStreetMap and the climate research ecosystem. If you're writing an urban climate paper and need a POI density grid to join with ERA5 air quality — our NetCDF export plugs straight into xarray.

Try NetCDF export from an OSM area — /formats/netcdf/.

Sources

Unidata NetCDF documentation — docs.unidata.ucar.edu/netcdf-c/current/
CF Metadata Conventions — cfconventions.org
NCO (NetCDF Operators) — nco.sourceforge.net
CDO (Climate Data Operators) — code.mpimet.mpg.de/projects/cdo
Panoply (NASA GISS) — giss.nasa.gov/tools/panoply
xarray documentation — docs.xarray.dev
ERA5 reanalysis (Copernicus) — cds.climate.copernicus.eu
CMIP6 datasets — esgf-node.llnl.gov/projects/cmip6