OSM PBF — OpenStreetMap's bulk-exchange binary format: specification and applications

2026-06-2810 min read

OSMPBFProtocol Buffersplanet.osmbulk exchangeOpenStreetMapGeofabrik

Export formats: OSM-PBF

OSM PBF (OpenStreetMap Protocol Buffers Format) is the primary binary format in which the OpenStreetMap project distributes full planetary dumps and regional extracts. A single planet.osm.pbf file in May 2026 weighs about 80 GB and contains every node, way, and relation ever drawn on the OSM map since the project's founding in 2004. It is the primary feed for all downstream services: osm2pgsql fills PostGIS on osm2cdr.ru servers, Nominatim builds its geocoding index from the same dumps, Overpass API holds copies for thematic queries, planetiler — Cloudflare's vector-tile generator — turns planet.osm.pbf into ready-to-serve PMTiles in a couple of hours. PBF is not for user-facing visualization — it's a bulk-exchange container between OSM-processing servers.

Map generation of Krasnodar city center — Krasnodar city center, 200×200 m fragment. Every pixel of this image traces back to nodes and ways once stored in planet.osm.pbf — then passed through osm2pgsql into PostGIS and rendered by the osm2cdr engine.

History: escape from XML and the birth of the format

Before 2010 OpenStreetMap distributed full planetary dumps as OSM XML — text XML with <node>, <way>, <relation> tags. By the end of 2009 the planet.osm file weighed about 200 GB bzip2-compressed, and parsing took hours even on powerful servers. Every node carried dozens of bytes of XML markup just to store two lat/lon numbers plus a tag set — which for a 100-million-element dataset meant monstrous overhead.

In 2010 Scott Crosby (then a distributed-systems researcher) designed and proposed to the OSM community a format based on Google Protocol Buffers. The idea was simple: instead of textual XML, use a binary format with tightly packed integer fields, delta-encoded coordinates, and group-level compression. Result: planet.osm.pbf turned out 30× smaller than uncompressed XML and 7× smaller than bzip2-compressed XML. Parsing sped up 5–10× thanks to the binary structure.

In December 2010 Geofabrik — the German company running the main OSM-dump mirror — switched to PBF as the primary distribution format. By 2012 all major toolchains (osm2pgsql, Osmosis, Osmium, imposm) already supported PBF, and XML dumps remained as a secondary fallback for legacy scripts.

By 2026 PBF is the only format the OSM community seriously considers for dumps. XML lives on in the OSM API (single-element queries), but bulk exchange is exclusively PBF.

What's inside an OSM PBF file

OSM PBF is a sequence of compressed blocks, and each block is a Protocol Buffers message. At the top level the file consists of alternating pairs of BlobHeader + Blob. BlobHeader is a small header with block type and size, Blob is the payload itself, optionally compressed with zlib deflate (at block level, not whole-file — so random access by offset is possible).

There are two block types. HeaderBlock comes first and contains the dataset's bounding box, required-features (properties the decoder must support, e.g. OsmSchema-V0.6 and DenseNodes), optional-features (e.g. Has_Metadata). PrimitiveBlock is the main data carrier — it contains an array of PrimitiveGroup, and each group holds either nodes, or ways, or relations (but not mixed).

Nodes in modern PBF files are almost always serialized as DenseNodes — a structure where coordinates, IDs, and timestamps are stored as int64 arrays with delta-encoding between neighbors. If nodes lie close in geospace (and they do, because OSM numbering is historically locality-aware), the delta between adjacent IDs is a small number, which Protocol Buffers varint encoding compresses into 1–2 bytes. Coordinates are stored as nano-degrees (multiplied by 10⁷ and rounded) — for a typical map fragment this gives deltas of tens of meters, yielding 2–3 bytes per coordinate instead of 8.

Ways store a list of node references as a delta-encoded int64 array — again exploiting ID clustering. Tag keys and values live in a stringtable — a block-wide dictionary of strings referenced by index. If the file has thousands of highway=residential, the string is stored once. Relations are similar to ways but references are typed members with roles (outer, inner, via, from, to).

Who needs OSM PBF: five roles

Mapping service deploying its own OSM copy. If you're building a competitor to osm2cdr.ru or a specialized service for navigation, real estate, logistics — you need your own PostGIS with current OSM data. The initial bootstrap is a planetary or regional .osm.pbf from Geofabrik, run through osm2pgsql. Then — daily updates via osmosis --rri or pyosmium-get-changes, also in PBF. Without PBF this pipeline is impossible: XML is simply too slow.

Urban-data researcher. If the task is to count how many buildings with building=residential Russia has, how many meters of two-lane road Sverdlovsk Oblast has, or to map courtyard-playground coverage — that's analysis on raw OSM. A regional PBF from Geofabrik (100 MB to 5 GB depending on region) downloads in minutes, opens via pyosmium or osmium-tool, and processes without a DBMS — streaming from disk.

Vector-tile developer. The modern tile-generation toolchain (planetiler, tilemaker, openmaptiles) takes planet.osm.pbf as input and in 1–4 hours on a modern CPU builds a ready MBTiles or PMTiles with vector tiles for the whole planet. Without PBF this process would stretch into days.

OSM-history archivist. Full historical OSM dumps (.osm.pbf with Has_Metadata and Has_History_Versions) include every edit of every node over 22 years. Urban-change researchers, geo-historians, OSINT journalists — all use historical PBFs for temporal analysis.

Corporate consumer of OSM with air-gapped infrastructure. Military, intelligence, banks, telecom — some corporate clients can't constantly hit the public OSM API. They subscribe to weekly PBF snapshots, run internal QA, and import into their PostGIS instances. PBF is the only format that supports such batch delivery.

Software for OSM PBF: seven programs

osm2pgsql — the main OSM-to-PostGIS importer. Used on OpenStreetMap.org's own servers for tile.openstreetmap.org rendering, and on osm2cdr.ru servers. Written in C++, planet.osm.pbf import speed is 4–8 hours on an NVMe server with 64 GB RAM. osm2pgsql.org

Osmium Tool — CLI utility for all kinds of PBF manipulation: bbox extraction, tag filtering, format conversion, merge, diff, statistics. Written in C++ on top of libosmium. Indispensable for scripting. osmcode.org/osmium-tool

Osmosis — Java tool of similar purpose, historically older. Supports --rri (replicate region intervals) — the canonical path for daily updates. github.com/openstreetmap/osmosis

imposm3 — alternative import to PostGIS in Go, optimized for serving vector tiles. Selective import (only tags from a mapping file), 2–3× faster than osm2pgsql on large datasets. github.com/omniscale/imposm3

pyosmium — Python bindings for libosmium. Convenient for research scripts and Jupyter notebooks. Streaming API, minimal RAM. github.com/osmcode/pyosmium

planetiler — Java MVT/PMTiles generator from OSM PBF. One of the fastest: planet.osm.pbf → ready planetary vector map in 1.5 hours. Used by Cloudflare and OnTheGoMap. github.com/onthegomap/planetiler

Geofabrik download server — primary distributor of regional PBFs. Slices by country, federal district, oblast, US state. Daily updates with 24-hour lag from planet.osm. download.geofabrik.de

Pitfalls and tips

Don't confuse OSM PBF and MVT PBF. These are two different formats, both using Google Protocol Buffers as the low-level encoding, but semantics and use are completely different. OSM PBF is a full-fidelity dump with nodes/ways/relations, complete tag set, edit metadata, optionally with history. Regional file sizes range from megabytes to gigabytes, planetary — tens of GB. MVT PBF (Mapbox Vector Tile) is a per-tile, rendering-optimized format storing only geometries visible in one tile (256×256 or 512×512 pixels) with a minimal attribute set needed for styling. A single MVT tile is typically 5–500 KB. You can't tell them apart by extension — both use .pbf; by context: a file called russia.osm.pbf is OSM PBF; a directory tree tiles/{z}/{x}/{y}.pbf is MVT.

Geofabrik vs planet.osm.pbf. If you need whole-planet data — take planet.osm.pbf from planet.openstreetmap.org (updates weekly on Wednesdays). If you need a country or regional slice — Geofabrik is more convenient: they slice and update daily. Geofabrik works by politically recognized borders — you can't download "Crimea separately" or "Transnistria separately". For arbitrary bboxes, use osmium extract.

Flatnodes file size is critical for import speed. osm2pgsql with the --flat-nodes flag creates a flatnodes.bin file — an array of all node-ID coordinates indexed by ID. For the planet this is an 80–100 GB file. Without --flat-nodes import finishes the read faster but requires 250+ GB RAM for the in-memory node cache. On the production osm2cdr.ru server --flat-nodes on NVMe is used — a tradeoff between speed and available memory.

Daily updates via minutely diffs. OSM publishes minutely changesets at planet.openstreetmap.org/replication/minute/. The utility pyosmium-get-changes --server URL --start-date YYYY-MM-DD output.osc.gz collects all edits for a period into one file. Then osm2pgsql --append output.osc.gz applies them to PostGIS. On osm2cdr.ru this runs via osm-replication-watchdog.service (systemd service, restarts on failure). Without this mechanism, data goes stale within a week.

PBF is write-once, append-many. The format itself doesn't allow convenient modification of an existing file. To update — either regenerate completely, or store diff files (.osc.gz) separately. So the workflow is always: bootstrap PBF → import into DB → from then on work with the DB and forget the PBF.

Streaming reads are mandatory for large files. Loading planet.osm.pbf entirely into RAM is impossible — you need a streaming pass. In libosmium this is the base case (handler pattern with callbacks per node/way/relation). In pyosmium — osmium.SimpleHandler with overridden methods. Never try to read the whole PBF as one array — OOM guaranteed.

Author metadata is optional. Standard Geofabrik dumps don't include edit metadata (user, version, timestamp, changeset). If you need it — take russia-internal.osh.pbf (history dump) from the same server. They weigh 3–5× more and require a commercial license from Geofabrik for commercial use.

How to use OSM PBF together with osm2cdr.ru

osm2cdr.ru does not produce OSM PBF as an output format — we deliver 124 other formats (DXF, SVG, GeoJSON, MBTiles, PMTiles, etc.), but PBF is our input format and our infrastructure is built on it. If your task is to export a chunk of OSM in PBF for a bbox to feed into your own tool, use this workflow.

Step 1. Download a regional PBF from Geofabrik:

wget https://download.geofabrik.de/russia/central-fed-district-latest.osm.pbf

Step 2. Cut your bbox with osmium-tool:

osmium extract -b 37.5,55.7,37.8,55.85 central-fed-district-latest.osm.pbf \
  -o moscow-center.osm.pbf

Step 3. Run your analysis pipeline:

import osmium

class BuildingCounter(osmium.SimpleHandler):
    def __init__(self):
        super().__init__()
        self.buildings = 0
        self.residential = 0
    def way(self, w):
        if 'building' in w.tags:
            self.buildings += 1
            if w.tags.get('building') == 'residential':
                self.residential += 1

handler = BuildingCounter()
handler.apply_file('moscow-center.osm.pbf')
print(f"Buildings: {handler.buildings} (residential: {handler.residential})")

Step 4. When analysis is complete and it's time to visualize the result — go to osm2cdr.ru, select the same bbox, pick the desired output format (e.g., DXF for AutoCAD or PMTiles for a web map), and get the file. osm2cdr reads data from the same Geofabrik PBF you just used for counting — but processes it through QGIS rendering and styling with 20 built-in map styles.

Map generation of Vladivostok city center — Vladivostok city center, 200×200 m. A render at the eastern edge of the same russia.osm.pbf dump — geography spanning 8000 km, one PBF file as the source.

Sources

OSM Wiki, PBF Format Specification — wiki.openstreetmap.org/wiki/PBF_Format
Scott Crosby, "PBF design notes" — github.com/openstreetmap/OSM-binary
Geofabrik regional downloads — download.geofabrik.de
planet.openstreetmap.org weekly dumps — planet.openstreetmap.org
libosmium documentation — osmcode.org/libosmium
osm2pgsql manual — osm2pgsql.org/doc/manual.html
planetiler benchmarks, OnTheGoMap — github.com/onthegomap/planetiler/blob/main/BENCHMARKS.md