NDJSON & JSON Lines: The Complete Guide to Streaming JSON Format
Standard JSON is a beautiful format for representing structured data — but it has a fundamental problem for large datasets and streaming: the entire document must be parsed at once. To read the last record in a 10 GB JSON file, you have to parse all 10 GB. NDJSON (Newline-Delimited JSON), also known as JSON Lines (.jsonl) or JSON Streaming, solves this by storing one valid JSON value per line — enabling true streaming, parallel processing, and incremental reading of massive datasets.
What Is NDJSON?
NDJSON (Newline-Delimited JSON) stores multiple JSON values separated by newlines. Each line is a complete, self-contained, valid JSON value (most commonly an object):
{"id":1,"name":"Alice Smith","email":"alice@example.com","age":32,"city":"New York"}
{"id":2,"name":"Bob Jones","email":"bob@example.com","age":28,"city":"London"}
{"id":3,"name":"Carol Williams","email":"carol@example.com","age":45,"city":"Toronto"}
Each line can be parsed independently with a standard JSON parser. To process 3 records from a 10 GB file, you read 3 lines — no need to load the entire file.
The Multiple Names Problem
NDJSON goes by several names that all describe essentially the same format:
- NDJSON (Newline-Delimited JSON): the format as specified at ndjson.org; requires UTF-8, each line must be a valid JSON value
- JSON Lines (
.jsonl): a closely related specification at jsonlines.org; the most common file extension in practice - JSON Streaming: informal term for the concept
- ldjson (Line-Delimited JSON): another informal alias
- LDJSON: same as ldjson
All of these formats are functionally identical: one JSON value per line, newline-separated. The choice of name/extension often reflects the tool or ecosystem you are using:
- Machine learning datasets:
.jsonl(most common) - Elasticsearch bulk API:
.ndjson - Log aggregation tools:
.jsonlines - MongoDB mongodump: ndjson internally
NDJSON vs Regular JSON
| Feature | JSON | NDJSON |
|---|---|---|
| File structure | Single JSON document | One JSON value per line |
| Streaming | ❌ Parse entire file | ✅ Process line by line |
| Append records | ❌ Must rewrite file | ✅ Append a line |
| Parallel processing | ❌ Requires full parse first | ✅ Split by lines and process |
| Random access | ❌ Must scan from start | ✅ Line offset for indexing |
| Human readability | Better (indented JSON) | Acceptable (one object per line) |
| Schema | Optional | Optional |
| Comments | ❌ Not in JSON spec | ❌ Not supported |
| Compression | Standard | Gzip line-by-line or whole file |
Why NDJSON Dominates Big Data and ML
Machine Learning Datasets
HuggingFace datasets, OpenAI's training data, and most academic NLP datasets distribute data in JSONL format because:
- Each example is one line — trivially parallelizable
- Lines can be shuffled by line number without loading into memory
- New examples can be appended without rewriting the file
- Dataset size doesn't matter — streaming processing handles terabyte files
Example ML training data (one JSONL record per training example):
{"instruction":"Translate to French:","input":"Hello, world","output":"Bonjour, monde"}
{"instruction":"Summarize:","input":"The quick brown fox jumps...","output":"A fox jumps over a dog."}
Log Aggregation
Structured logging systems (Elasticsearch, Splunk, Datadog, Loki) use NDJSON for log shipping because each log event is an independent JSON object:
{"timestamp":"2025-04-25T14:00:00Z","level":"INFO","service":"api","message":"Request processed","duration_ms":45}
{"timestamp":"2025-04-25T14:00:01Z","level":"ERROR","service":"db","message":"Connection timeout","retry":1}
Elasticsearch Bulk API
Elasticsearch's bulk indexing API uses a two-line NDJSON structure:
{"index":{"_index":"users","_id":"1"}}
{"name":"Alice","email":"alice@example.com","age":32}
{"index":{"_index":"users","_id":"2"}}
{"name":"Bob","email":"bob@example.com","age":28}
Data Pipeline Interchange
Tools like jq, Apache Spark, AWS Athena, and BigQuery all support JSONL natively — it is the preferred format for data pipeline steps because each stage can process records as they arrive.
Reading and Writing NDJSON
Python — Streaming (Memory Efficient)
# Read NDJSON line by line (O(1) memory regardless of file size)
with open('data.jsonl', 'r', encoding='utf-8') as f:
for line in f:
record = json.loads(line.strip())
if record: # skip empty lines
process(record)
# Write NDJSON
with open('output.jsonl', 'w', encoding='utf-8') as f:
for record in records:
f.write(json.dumps(record, ensure_ascii=False) + '\n')
Python with pandas
import pandas as pd
# Read entire JSONL into a DataFrame
df = pd.read_json('data.jsonl', lines=True)
# Write DataFrame as JSONL
df.to_json('output.jsonl', orient='records', lines=True, force_ascii=False)
jq — Command Line Processing
# Process NDJSON with jq (streaming)
cat data.jsonl | jq '.name' # extract name field
cat data.jsonl | jq 'select(.age>30)' # filter records
cat data.jsonl | jq -c '{n:.name,e:.email}' # reshape records
# Aggregate (loads all into memory)
cat data.jsonl | jq -s 'map(.age) | add / length' # average age
JavaScript / Node.js
// Streaming read with readline
const readline = require('readline');
const fs = require('fs');
const rl = readline.createInterface({
input: fs.createReadStream('data.jsonl'),
terminal: false
});
rl.on('line', (line) => {
if (line.trim()) {
const record = JSON.parse(line);
console.log(record.name);
}
});
Gzipped NDJSON
Large NDJSON files are typically gzip-compressed (.jsonl.gz or .ndjson.gz):
# Compress
gzip data.jsonl
# Stream-decompress without extracting
zcat data.jsonl.gz | jq '.name'
python3 -c "import gzip,json,sys; [print(r['name']) for r in [json.loads(l) for l in gzip.open(sys.argv[1],'rt')]]" data.jsonl.gz
Converting NDJSON
NDJSON → JSON Array
# Wrap lines in a JSON array
jq -s '.' data.jsonl > data.json
Or in Python:
records = [json.loads(line) for line in open('data.jsonl') if line.strip()]
json.dump(records, open('data.json', 'w'), indent=2)
JSON Array → NDJSON
# Explode JSON array to NDJSON (jq -c for compact)
jq -c '.[]' data.json > data.jsonl
NDJSON → CSV
# Using jq + csvkit
cat data.jsonl | jq -r '[.name,.email,.age] | @csv' > data.csv
Or pandas:
df = pd.read_json('data.jsonl', lines=True)
df.to_csv('data.csv', index=False)
CSV → NDJSON
import csv, json
with open('data.csv', encoding='utf-8') as f:
reader = csv.DictReader(f)
with open('data.jsonl', 'w') as out:
for row in reader:
out.write(json.dumps(row) + '\n')
NDJSON in the Cloud
- AWS S3 + Athena: Athena can query JSONL files in S3 directly using SQL — no loading into a database required
- Google BigQuery:
bq load --source_format=NEWLINE_DELIMITED_JSONingests JSONL - Azure Data Factory: native JSONL support in data flows
- Snowflake:
COPY INTOfrom JSONL files in S3/Azure/GCS storage
NDJSON is the format that makes big data practical: simple enough that any script can generate it, flexible enough that each line can have a different schema, and stream-friendly enough that terabyte datasets can be processed on a laptop with constant memory usage. For any dataset with more than a few thousand records, NDJSON is almost always the right choice over regular JSON.
Related conversions
Common video conversions that pair well with this guide: