GeoJSON-Seq (RFC 8142) format: streaming features — OSM export for big-data pipelines

GeoJSON-Seq is the format invented precisely because plain GeoJSON turned out to be a poor choice for big data. A single FeatureCollection object weighing hundreds of megabytes can't be streamed properly, can't be processed line-by-line, can't be appended to — it's only valid JSON when fully closed with a curly brace. RFC 8142 (August 2017, Sean Gillies and Daniel Holm Stahl) solved it the most direct way: "we ditch FeatureCollection, write one Feature per record, separators between records are either ASCII Record Separator 0x1E plus newline, or just newline (NDJSON style)." The result is a format ideally suited for Apache Spark, BigQuery GIS, Snowflake, Kafka, and tippecanoe pipelines — wherever data flows as a stream and client memory is bounded. On osm2cdr.ru GeoJSON-Seq is exported via the ogr2ogr GeoJSONSeq driver: a PostGIS query becomes a file that you can cat into tippecanoe, load via bq load, stream-upload to S3 in chunks, and read with gdal without resident-set bloat.

History: why RFC 8142 was needed

By 2016 GeoJSON (RFC 7946) was the dominant web cartography format, but at hundreds of millions of records it became clear: the format is built against streaming. To read the first Feature, a parser must reach the closing ] of the features array — otherwise it doesn't know if the document is valid. Fine for a 5 MB block-level map, broken for a 50 GB dump of all of North America.

Sean Gillies (same author of Fiona, Rasterio, and one of the six original creators of GeoJSON 1.0 in 2008) and Daniel Holm Stahl raised the idea of "text sequences for GeoJSON" at the IETF. The reference point was RFC 7464 — the "JSON Text Sequences" standard describing a general mechanism: prepend each JSON document with an RS byte (Record Separator, 0x1E ASCII), terminate with LF (0x0A). Any invalid JSON between separators is unambiguously detected as a bad record and can be skipped — this is robust to embedded newlines inside pretty-printed JSON.

In August 2017 RFC 8142 was published under the name GeoJSON Text Sequences. The spec is only seven pages, but its adoption opened the door for the whole big-data geo stack: Mapbox tippecanoe made GeoJSON-Seq its preferred input format, Google BigQuery began loading it via bq load --source_format=NEWLINE_DELIMITED_JSON, ESRI and FME added drivers. In GDAL the corresponding GeoJSONSeq driver appeared in version 2.4 (December 2018) and became a standard component from 3.0 (May 2019).

Technically: two variants of one format

The RFC 8142 spec describes the strict variant, but in practice two are used:

Variant 1 — RS-prefixed (strict per RFC 8142). Each record starts with one RS = 0x1E byte, followed by the Feature JSON (pretty-print allowed, with inner newlines), and closes with one LF = 0x0A. The parser searches for RS as the record boundary, making the format robust against embedded newlines inside JSON. File extension — usually .geojsons or .geojsonseq.

Variant 2 — LF-only (NDJSON style, de facto). No RS, only \n (LF) serves as separator. That means each Feature must be minified — no inner newlines inside JSON. Extension — .geojsonl or .ndjson. This variant technically doesn't comply strictly with RFC 8142, but it's what tippecanoe, jq pipelines, BigQuery, and most practical tools use. It's sometimes called "NDGeoJSON" (Newline-Delimited GeoJSON).

The GDAL GeoJSONSeq driver supports both: writes RS-prefixed by default, switches to LF-only via the RS=NO option. On read it autodetects — if the first byte is 0x1E, reads strict, otherwise as NDJSON.

# RS-prefixed (proper RFC 8142):
<RS>{"type":"Feature","geometry":{"type":"Point","coordinates":[30.31,59.93]},
     "properties":{"amenity":"cafe"}}<LF>
<RS>{"type":"Feature","geometry":...}<LF>

# LF-only (NDGeoJSON, de facto):
{"type":"Feature","geometry":{"type":"Point","coordinates":[30.31,59.93]},"properties":{"amenity":"cafe"}}
{"type":"Feature","geometry":...}

The key architectural difference from GeoJSON RFC 7946 is no FeatureCollection wrapper. The whole file isn't valid JSON, but every line/record is a valid JSON Feature object. That little decision yields a huge practical effect: you can tail -f a file (for real-time GPS logs), head -100 for debugging, wc -l for record count, split -l 10000 for sharding. The entire UNIX toolkit works on GeoJSON-Seq natively.

Pipeline: PostGIS → ogr2ogr → tippecanoe / BigQuery / Kafka

This is the classic "GeoJSON-Seq as universal pipe" pattern: one ogr2ogr command produces a stream that four different downstream tools consume without intermediate conversions. Tippecanoe bakes MVT vector tiles, BigQuery runs SQL analytics with PostGIS-compatible ST_* functions, Kafka distributes messages to Spark/Flink, S3 accepts stream upload via multipart. Same file everywhere.

GeoJSON-Seq vs GeoJSON: key differences

Regular GeoJSON is a single FeatureCollection JSON object requiring full in-memory load before processing. To read the first Feature, the parser must reach the end of the file (syntactically). At 100 MB it's tolerable, at 50 GB it's impossible.

GeoJSON-Seq changes three things: - Memory pattern: O(1) constant memory instead of O(n). The parser reads one record, processes, forgets, reads the next. - Appendability: new Features can be added at the end of the file — pure concatenation. With plain GeoJSON you'd rewrite the whole wrapper. - Splittability: the file easily splits into chunks by lines without knowing the structure. Spark takes it as textFile with N-line block partitioning.

What's lost in the migration: top-level metadata (FeatureCollection-level bbox, crs, name — nowhere to put them), representation compactness (each Feature repeats "type":"Feature", accumulating ~10 bytes overhead per record), validity of the whole file as JSON (no JSON.parse(file)).

GeoJSON-Seq vs CSV: geometry vs flat table

Engineers often write "let's just use CSV with lat/lon columns — why do we need GeoJSON-Seq". That works for points but breaks for anything more complex:

For billion-point GPS tracks CSV wins on parsing speed. For OSM data with buildings, MultiPolygon lakes, and nested OSM tags CSV becomes hell: WKT strings in one column are unreadable, nested tags collapse into string-serialization. GeoJSON-Seq is the compromise — structural enough for complex data, linear enough for streaming.

Use case: tippecanoe for MVT vector tiles

Tippecanoe is an open-source tool by Mapbox, built by Eric Fischer (2016) for generating MVT (Mapbox Vector Tiles) from large geodata volumes. It's the de facto standard for preparing vector basemaps and thematic layers in pmtiles/mbtiles.

Tippecanoe takes GeoJSON-Seq in LF variant (NDGeoJSON) as input:

ogr2ogr -f GeoJSONSeq -lco RS=NO \
        osm_buildings.geojsonseq \
        PG:"host=localhost dbname=osm" \
        -sql "SELECT way, building, name FROM planet_osm_polygon WHERE building IS NOT NULL"

tippecanoe -o buildings.pmtiles \
           --layer=buildings \
           --maximum-zoom=16 \
           --minimum-zoom=10 \
           osm_buildings.geojsonseq

Full pipeline: PostGIS query for buildings → GeoJSON-Seq stream → tippecanoe compiles into PMTiles → ready for MapLibre web map. All stages are memory-bounded, can run on 10 million buildings on a laptop with 8 GB RAM.

Use case: BigQuery GIS bulk load

Google BigQuery has supported geo queries via ST_* functions (PostGIS-compatible syntax) since 2019. For bulk data load, the correct format is Newline-Delimited JSON, which is GeoJSON-Seq in LF variant:

ogr2ogr -f GeoJSONSeq -lco RS=NO osm_pois.geojsonseq ...

bq load \
  --source_format=NEWLINE_DELIMITED_JSON \
  --autodetect \
  osm_dataset.pois \
  osm_pois.geojsonseq

BigQuery imports in parallel via Dataflow, no single-master bottleneck. From there — regular SQL with ST_DWithin, ST_Intersects on petabyte-scale datasets.

Use case: Apache Spark / Sedona

Apache Sedona (formerly GeoSpark) is a geo extension for Spark. GeoJSON-Seq reads directly via spark.read.format("geojson") or through textFile with line-by-line parsing:

from sedona.spark import *

sedona = SedonaContext.create(spark)
df = sedona.read.format("geojson").load("hdfs://path/osm_buildings.geojsonseq")
df.createOrReplaceTempView("buildings")
sedona.sql("SELECT name, ST_Area(geometry) FROM buildings WHERE ST_Area(geometry) > 1000").show()

Each Spark partition takes its N-line block, parses independently, computes aggregations. Linear scalability to hundreds of nodes.

Use case: Kafka real-time GPS streams

For real-time tracking (delivery, fleet, IoT) it's convenient to put one Feature per message into a Kafka payload. The producer formats a GeoJSON-Seq line, the consumer reads line by line:

import json
from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')

feature = {
    "type": "Feature",
    "geometry": {"type": "Point", "coordinates": [30.31, 59.93]},
    "properties": {"vehicle_id": "TX-42", "timestamp": "2026-05-13T12:30:00Z"}
}
producer.send('gps-stream', json.dumps(feature).encode())

Consumer-side: every message parses directly via json.loads() without a FeatureCollection wrapper. Audit dump — simply kafkacat -C -t gps-stream > audit.geojsonseq and you get a valid GeoJSON-Seq file.

Software for GeoJSON-Seq

GDAL / ogr2ogr. Free. Since version 2.4 (December 2018) the GeoJSONSeq driver is built-in. Reads and writes both variants (RS and LF). RS=NO option for NDGeoJSON mode.

tippecanoe. Free (Mapbox, BSD license). De facto standard for generating MVT/PMTiles from GeoJSON-Seq. Memory-bounded streaming, handles tens of gigabytes.

jq. Universal utility for JSON stream processing. cat file.geojsonseq | jq 'select(.properties.amenity=="cafe")' > cafes.geojsonseq — filter millions of records in a pipe.

pyogrio. Python binding to GDAL, 5-20x faster than GeoPandas on read. Supports GeoJSONSeq via the same driver.

GeoPandas. gpd.read_file('file.geojsonseq') works transparently via GDAL/fiona. For large files prefer pyogrio or DuckDB.

DuckDB Spatial. SELECT * FROM ST_Read('file.geojsonseq') — columnar parser, excellent for aggregations.

Apache Sedona. Distributed processing via Spark.

FME (Safe Software). Commercial ETL — has GeoJSONSeqReader/Writer transformer.

Mapbox/Tilemaker. Open-source alternative to tippecanoe (more memory-frugal).

Pipeline in OSM2CDR

Our GeoJSON-Seq export runs through the GDAL GeoJSONSeq driver. The pipeline:

Step 1: PostGIS query. Based on the bbox and selected layers, a SQL query is built against the osm tables (planet_osm_polygon, planet_osm_line, planet_osm_point). WKB fast-path is used for direct binary transfer without intermediate GeoJSON parsing.

Step 2: ogr2ogr streaming. The GeoJSONSeq driver writes the file feature-by-feature, no in-memory accumulation. RS-prefixed (strict RFC 8142) by default. A format option toggles to LF-only for tippecanoe/BigQuery compatibility.

Step 3: Output. The file with extension .geojsonseq (or .geojsonl for the LF variant) is delivered via downloadable URL. On large exports the size reaches tens and hundreds of megabytes, while the process memory footprint stays O(1) — around 50-100 MB regardless of output size.

Because the format is streaming-friendly, we can serve it via HTTP chunked transfer — the user can start the tippecanoe pipeline before the export has fully finished on the server.

Strengths and weaknesses

Strengths. O(1) memory streaming — handles files of any size on any machine. Appendable — adding new Features = cat >>. Splittable — Spark/Flink split by lines into partitions. UNIX-friendly — head, tail, wc -l, split, grep work natively. Supported by all major big-data platforms (BigQuery, Spark, tippecanoe, GDAL 2.4+). IETF standard (RFC 8142, 2017) — formally open spec. Backward-compatible with GeoJSON Feature syntax — same libraries parse at the record level.

Weaknesses. Two variants (RS vs LF) create confusion — some tools handle only one. No top-level metadata (FeatureCollection-level bbox, crs — nowhere to put them). JSON overhead per record (~10 bytes "type":"Feature" repeats). Not for the web — browsers can't parse line-by-line out of the box. Larger than binary formats (FlatGeobuf on the same data is 3-5x more compact).

Comparison with similar formats

GeoJSON — for web, small/medium datasets, simple integration. GeoJSON-Seq — for big-data pipelines, streaming, tippecanoe input. FlatGeobuf — when you need spatial index and HTTP range reads (lazy loading in browser). MBTiles — pre-rendered tiles in SQLite container for offline maps.

FAQ

What's the difference between GeoJSON-Seq and GeoJSON? GeoJSON is a single FeatureCollection JSON document. GeoJSON-Seq is a sequence of separate Feature objects, delimited either by RS (0x1E) + LF or just LF. The main benefit — streaming processing with O(1) memory.

Which file extension to use? Canonical per RFC 8142 — .geojsons or .geojsonseq. In practice for the LF variant (NDGeoJSON) .geojsonl or .ndjson are common. GDAL accepts any, looking at content.

Can QGIS open GeoJSON-Seq? Yes, with QGIS 3.4+ (GDAL 2.4+) the GeoJSONSeq driver is built-in — drag-and-drop works. For large files QGIS may lag — prefer pyogrio via Python.

Does GeoJSON-Seq follow the right-hand rule from RFC 7946? Each individual Feature must follow all RFC 7946 rules (including WGS84 CRS, polygon right-hand rule, antimeridian splitting). GeoJSON-Seq doesn't cancel those rules, only changes the packaging.

RS vs LF — which variant to pick? If the file is for tippecanoe, BigQuery, Spark — LF-only (NDGeoJSON). For strict spec-compliant processing and robustness to pretty-printed JSON — RS-prefixed (RFC 8142 strict). On osm2cdr.ru the default is RS, an option switches to LF.

Can two GeoJSON-Seq files be concatenated? Yes, that's the main feature: cat file1.geojsonseq file2.geojsonseq > merged.geojsonseq. With plain GeoJSON you'd have to parse, merge features arrays, and rewrite.

Related

• GeoJSON — the main geodata format for the web, RFC 7946
• FlatGeobuf — binary streaming format with R-tree
• MBTiles — SQLite container for raster/vector tiles
• PMTiles — cloud-native single-file vector format

Sources

• RFC 8142 — GeoJSON Text Sequences, IETF, August 2017
• RFC 7464 — JSON Text Sequences — base text-sequences spec
• GDAL GeoJSONSeq driver
• tippecanoe (Mapbox)
• BigQuery GIS load NEWLINE_DELIMITED_JSON
• Apache Sedona