BSON: Binary JSON — MongoDB's Native Data Format

BSON: Binary JSON — MongoDB's Native Data Format

BSON (Binary JSON) is a binary-encoded serialization format created by MongoDB Inc. (then 10gen) in 2009 as the native data format for MongoDB. BSON extends JSON with additional data types (binary data, ObjectId, Date, Decimal128, regular expressions, JavaScript code) and encodes them in a binary format that prioritizes traversal speed and in-place field updating over raw compactness.

BSON is to MongoDB what Parquet is to Spark: not a general interchange format, but the internal storage format optimized for the specific access patterns of the database. Understanding BSON is essential for working with MongoDB at a level deeper than the driver API — for schema design, index planning, aggregation pipeline optimization, and diagnosing document size limit issues.

BSON Design Goals

BSON was designed with two primary goals that sometimes conflict with compactness:

1. Fast traversal — every element is prefixed with its type and size, so the BSON parser can skip elements without decoding them. This enables field lookups by name in O(n) without a parse tree.

2. In-place updating — because MongoDB's storage engine (WiredTiger) may update documents in place, BSON reserves space for size-prefixed elements. This means some BSON documents are actually larger than their JSON equivalents — a deliberate trade-off for write performance.

BSON Type System

BSON defines 20 data types beyond JSON's 6:

BSON Binary Encoding

A BSON document is encoded as:

document ::= int32 e_list "\x00"
e_list ::= element e_list | ""
element ::= "\x10" e_name int32                   # int32
           | "\x12" e_name int64                   # int64
           | "\x01" e_name double                  # float64
           | "\x02" e_name string                  # string
           | "\x03" e_name document               # embedded doc
           | "\x08" e_name ("\x00" | "\x01")      # boolean
           | "\x09" e_name int64                   # datetime
           | "\x07" e_name (byte*12)               # ObjectId
           | "\x0A" e_name                         # null
           | "\x05" e_name binary                  # binary
e_name ::= cstring
string ::= int32 (byte*) "\x00"
cstring ::= (byte*) "\x00"

For example, the document {"x": 1}:

\x0C\x00\x00\x00   ← total document size: 12 bytes
\x10               ← type: int32
x\x00              ← key "x" (cstring, null-terminated)
\x01\x00\x00\x00   ← value: 1 (int32, little-endian)
\x00               ← document terminator

Note that BSON uses little-endian byte order for all multi-byte integers and floats — unlike CBOR and network protocols which use big-endian.

Why BSON Can Be Larger Than JSON

For a document with mostly short string values:

{"status": "active", "tier": "premium", "region": "us-east"}

JSON (minified): 52 bytes

BSON:

Total doc size: 4 bytes
Per field: type(1) + key_length + null(1) + string_length(4) + content + null(1)
"status": 1 + 7 + 4 + 6 + 1 = 19 bytes
"tier": 1 + 5 + 4 + 7 + 1 = 18 bytes
"region": 1 + 7 + 4 + 7 + 1 = 20 bytes
Terminator: 1 byte
Total: 4 + 19 + 18 + 20 + 1 = 62 bytes

BSON: 62 bytes (vs 52 bytes JSON) — strings have 4-byte length prefix overhead.

But for a document with integers and dates:

{"count": 1500000, "ts": 1710500000000, "active": true}

JSON: 54 bytes (numbers as text) BSON: 28 bytes (int32=4B, int64=8B, bool=1B, keys+overhead)

BSON wins decisively for numeric and binary data.

ObjectId: MongoDB's Primary Key

The ObjectId is BSON's most distinctive type — a 12-byte globally unique identifier generated by the MongoDB driver without coordinating with the server:

ObjectId structure (12 bytes):
[4-byte Unix timestamp][3-byte machine ID][2-byte process ID][3-byte counter]
  TTTTTTTT               MMMMMM             PPPP               CCCCCC

Example: ObjectId("507f1f77bcf86cd799439011")

50 7F 1F 77   ← Unix timestamp: 1350393207 (2012-10-16)
BC F8 6C       ← Machine ID
D7 99           ← Process ID
43 90 11        ← Counter

Properties of ObjectId:

• Monotonically increasing within a second — ObjectIds sort chronologically when created at different times
• Globally unique with astronomically low collision probability — machines + PID + counter ensures uniqueness across a distributed cluster
• Embeds creation time — ObjectId.generation_time gives the creation timestamp without a separate field
• 12 bytes / 24 hex chars — more compact than UUID (16 bytes / 36 chars with hyphens)

from bson import ObjectId
from datetime import datetime

oid = ObjectId()
print(oid)                          # 65f4a2b3c1d8e7f902a3b4c5
print(oid.generation_time)          # 2024-03-15 10:30:43+00:00
print(oid.binary)                   # b'\x65\xf4\xa2\xb3...' (12 bytes)

# Convert string to ObjectId
oid2 = ObjectId("507f1f77bcf86cd799439011")
print(oid2.generation_time)         # 2012-10-16

# Use in MongoDB query
from pymongo import MongoClient
client = MongoClient("mongodb://localhost:27017/")
db = client["mydb"]
doc = db.users.find_one({"_id": ObjectId("507f1f77bcf86cd799439011")})

Binary Data Subtypes

BSON binary data carries a subtype byte:

The UUID subtype distinction matters: subtype 0x03 stored UUIDs in a non-standard byte order (used by the old Ruby and Java drivers), while subtype 0x04 uses the standard RFC 4122 byte order. Cross-driver UUID comparisons can fail if the subtypes don't match.

Python: pymongo and bson

from pymongo import MongoClient
from bson import ObjectId, Decimal128, Binary, Regex
from bson.codec_options import CodecOptions
import datetime
import re

client = MongoClient("mongodb://localhost:27017/")
db = client["ecommerce"]

# Insert a document with rich BSON types
order = {
    "customer_id": ObjectId("507f1f77bcf86cd799439011"),
    "status": "pending",
    "amount": Decimal128("149.99"),          # Exact decimal (no float rounding)
    "created_at": datetime.datetime.utcnow(), # Stored as int64 ms
    "items": [
        {"sku": "WIDGET-A", "qty": 2, "price": Decimal128("49.99")},
        {"sku": "GADGET-B", "qty": 1, "price": Decimal128("50.01")},
    ],
    "tags": ["electronics", "premium"],
    "avatar": Binary(b'\x89PNG...', subtype=0x00),  # Binary data
    "notes_pattern": Regex("urgent|priority", "i"),  # Regex stored natively
}

result = db.orders.insert_one(order)
print(f"Inserted: {result.inserted_id}")

# Query with ObjectId
doc = db.orders.find_one({"_id": result.inserted_id})
print(f"Amount: {doc['amount']}")

# Aggregation
pipeline = [
    {"$match": {"status": "pending"}},
    {"$group": {
        "_id": None,
        "total": {"$sum": "$amount"},
        "count": {"$sum": 1},
    }},
]
for result in db.orders.aggregate(pipeline):
    print(f"Pending orders: {result['count']}, Total: {result['total']}")

# Serialize/deserialize BSON directly
import bson

document = {"x": 1, "y": ObjectId(), "ts": datetime.datetime.utcnow()}
bson_bytes = bson.encode(document)
print(f"BSON size: {len(bson_bytes)} bytes")
decoded = bson.decode(bson_bytes)

Document Size Limits

MongoDB enforces a 16 MB maximum document size. This is a BSON-level constraint, enforced by the server. Common causes of hitting this limit:

• Arrays that grow unbounded (e.g., appending events to a document instead of using a separate events collection)
• Embedding large binary blobs in documents
• Deeply nested structures with many duplicate field names

The GridFS specification handles files larger than 16 MB by splitting them into 255 KB chunks stored in a fs.chunks collection, with metadata in fs.files.

BSON vs JSON vs MessagePack

Conclusion

BSON is not a general-purpose serialization format — it is an operational format optimized for MongoDB's specific requirements: fast field-level traversal, in-place updates, and rich type support for database use cases (ObjectId for primary keys, Decimal128 for financial data, Regex for pattern matching, Binary with subtypes for encrypted fields). Understanding BSON's type system, ObjectId structure, and size implications is essential for designing efficient MongoDB schemas and diagnosing performance issues at the storage layer. When working with MongoDB, you are always working with BSON, even when the driver presents a convenient JSON-like API.

Related conversions

Document conversions that follow this topic naturally:

• PDF to DOCX
• DOCX to PDF
• MD to PDF
• HTML to PDF
• ODT to PDF
• PDF to TXT