MessagePack: Binary JSON Serialization Explained

MessagePack: Binary JSON Serialization Explained

MessagePack is a binary serialization format that encodes JSON-like data structures — maps, arrays, strings, integers, floats, and booleans — into a compact binary representation. The project tagline describes it well: "It's like JSON, but fast and small." Unlike Protocol Buffers, MessagePack requires no schema definition — you serialize any data structure directly, just as you would with JSON, but the output is binary and typically 2-5× smaller.

MessagePack was created by Sadayuki Furuhashi in 2008 and is used by Redis (for cluster communication), Fluentd (log routing), Neovim (remote plugin protocol), and many real-time systems that need JSON-level flexibility with binary-level performance.

MessagePack vs JSON: The Core Trade-off

JSON's design prioritizes human readability and universal compatibility. MessagePack's design prioritizes space efficiency and parse speed while retaining JSON's schema-free flexibility. Consider a simple user object:

{"id": 42, "name": "Alice", "active": true, "score": 98.6}

JSON (minified): 46 bytes

MessagePack binary (hex):

84                  ← fixmap, 4 entries
a2 69 64            ← fixstr "id" (2 chars)
2a                  ← positive fixint 42
a4 6e 61 6d 65      ← fixstr "name" (4 chars)
a5 41 6c 69 63 65   ← fixstr "Alice" (5 chars)
a6 61 63 74 69 76 65← fixstr "active" (6 chars)
c3                  ← true
a5 73 63 6f 72 65   ← fixstr "score" (5 chars)
cb 40 58 99 99 99 99 99 9a  ← float64 98.6

MessagePack: 35 bytes (~24% smaller for this tiny example; savings grow for larger payloads with more numeric data)

The MessagePack Type System

MessagePack defines a fixed set of types with efficient binary encodings:

Integers

MessagePack uses range-based encoding to minimize bytes for common integer values:

Values in the range 0–127 encode in a single byte (the value itself, with the MSB clear). Values -32 to -1 also encode in one byte using a 3-bit type marker. This means the vast majority of typical integer API values (user IDs, counts, status codes) encode in 1-2 bytes.

Strings and Bytes

MessagePack distinguishes between UTF-8 strings and raw byte sequences:

Short strings (≤31 bytes) — typical for field names, status values, and short identifiers — use a 1-byte header. This means a 5-character string costs 6 bytes total, versus 7 bytes in JSON ("Alice" includes the two quote characters).

Arrays and Maps

Small arrays and maps (≤15 elements) get a 1-byte header, making typical API response objects very compact.

Floats

MessagePack encodes IEEE 754 floats without any text conversion overhead:

• float 32: 5 bytes (1 format byte + 4 data bytes)
• float 64: 9 bytes (1 format byte + 8 data bytes)

Compare this to JSON: the value 3.141592653589793 requires 17 bytes as text, versus 9 bytes as a float64 in MessagePack.

Extension Types

MessagePack's most powerful feature for applications is the ext type system, which allows custom type definitions using a 1-byte type identifier:

fixext 1:  1 format byte + 1 type byte + 1 data byte  = 3 bytes total
fixext 2:  1 format byte + 1 type byte + 2 data bytes = 4 bytes total
fixext 4:  1 format byte + 1 type byte + 4 data bytes = 6 bytes total
fixext 8:  1 format byte + 1 type byte + 8 data bytes = 10 bytes total
fixext 16: 1 format byte + 1 type byte + 16 data bytes = 18 bytes total
ext 8:     variable length, up to 255 bytes
ext 16:    variable length, up to 65535 bytes
ext 32:    variable length, up to 4 GB

The MessagePack specification reserves ext types -128 to -1 for official use and 0 to 127 for application-defined types. Official ext types include:

• Timestamp (type -1): nanosecond-precision timestamps in 4, 8, or 12 bytes — far more compact than ISO 8601 strings
• Application-defined types: UUIDs, Decimal types, Binary UUIDs, geographic coordinates — anything you need

import msgpack
import uuid
from datetime import datetime, timezone

# Custom encoder for UUID
def encode_uuid(obj):
    if isinstance(obj, uuid.UUID):
        return msgpack.ExtType(1, obj.bytes)  # type=1, 16 bytes
    return obj

# Custom decoder for UUID
def decode_uuid(code, data):
    if code == 1:
        return uuid.UUID(bytes=data)
    return msgpack.ExtType(code, data)

user = {
    'id': uuid.uuid4(),
    'name': 'Bob Smith',
    'created': datetime.now(timezone.utc),
}

packed = msgpack.packb(user, default=encode_uuid, use_bin_type=True)
unpacked = msgpack.unpackb(packed, ext_hook=decode_uuid, raw=False)
print(unpacked['id'], type(unpacked['id']))  # UUID object

Python msgpack

import msgpack

# Basic serialization
data = {
    'users': [
        {'id': 1, 'name': 'Alice', 'score': 99.5, 'active': True},
        {'id': 2, 'name': 'Bob', 'score': 87.3, 'active': False},
    ],
    'total': 2,
    'page': 1,
}

# Pack (serialize)
packed = msgpack.packb(data, use_bin_type=True)
print(f"MessagePack: {len(packed)} bytes")

import json
json_str = json.dumps(data)
print(f"JSON: {len(json_str.encode())} bytes")

# Unpack (deserialize)
unpacked = msgpack.unpackb(packed, raw=False)  # raw=False returns str, not bytes
assert unpacked == data

# Streaming pack/unpack (for large datasets)
with open('data.msgpack', 'wb') as f:
    packer = msgpack.Packer(use_bin_type=True)
    for record in large_record_iterator():
        f.write(packer.pack(record))

# Streaming unpack
with open('data.msgpack', 'rb') as f:
    unpacker = msgpack.Unpacker(f, raw=False)
    for record in unpacker:
        process(record)

# Fallback encoder for non-serializable types
def fallback_encoder(obj):
    if isinstance(obj, datetime):
        return {'__datetime__': obj.isoformat()}
    raise TypeError(f"Unknown type: {type(obj)}")

packed = msgpack.packb({'ts': datetime.now()}, default=fallback_encoder, use_bin_type=True)

Use Cases

Redis Caching

MessagePack is commonly used to serialize Python objects before storing them in Redis, where the compact binary format reduces memory usage and network transfer:

import redis
import msgpack

r = redis.Redis()

def cache_set(key: str, value, ttl: int = 3600):
    packed = msgpack.packb(value, use_bin_type=True)
    r.setex(key, ttl, packed)

def cache_get(key: str):
    raw = r.get(key)
    if raw is None:
        return None
    return msgpack.unpackb(raw, raw=False)

# Store a complex object
cache_set('user:42', {'id': 42, 'name': 'Alice', 'roles': ['admin']})
user = cache_get('user:42')

WebSocket Real-Time APIs

MessagePack is a popular binary frame format for WebSocket APIs that need to send many small messages at high frequency:

# FastAPI WebSocket with MessagePack
from fastapi import FastAPI, WebSocket
import msgpack

app = FastAPI()

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    async for raw in websocket.iter_bytes():
        msg = msgpack.unpackb(raw, raw=False)
        response = process_message(msg)
        await websocket.send_bytes(msgpack.packb(response, use_bin_type=True))

Neovim Remote Plugin Protocol (RPC)

Neovim uses MessagePack-RPC as the protocol for communicating between the Neovim core and remote plugins (including LSP clients, GUI frontends, and editor extensions):

import pynvim

# Connect to a running Neovim instance via msgpack-rpc
nvim = pynvim.attach('socket', path='/tmp/nvim.sock')
nvim.command('echo "Hello from Python"')
current_buf = nvim.current.buffer[:]  # Returns list of lines

MessagePack vs Alternatives

When to Choose MessagePack

Choose MessagePack when:

• You need JSON flexibility (no schema, dynamic types) but want binary efficiency
• The serialized data will not be human-inspected in production (logs, debugging aside)
• You are storing serialized objects in Redis, Memcached, or similar key-value stores
• You are building a real-time system (WebSockets, gaming, IoT) where message size and parse speed matter
• You need nanosecond-precision timestamps without a custom encoding

Choose JSON instead when:

• Human readability of serialized data is required
• Browser consumption without a JavaScript library is needed
• Debugging requires inspecting raw wire data

Choose Protobuf instead when:

• Maximum performance and minimum payload size are critical
• A formal schema contract between services is desirable
• gRPC is being used for transport

Conclusion

MessagePack occupies the sweet spot between JSON's flexibility and Protocol Buffers' performance. It requires no schema file, no code generation step, and no toolchain setup — you can adopt it by swapping your JSON library calls with MessagePack calls and immediately get 2-5× smaller payloads and 3-5× faster serialization. Its extension type system handles custom types cleanly, and its streaming unpacker handles arbitrarily large streams of records without buffering the entire input. For Redis caching, real-time WebSocket APIs, log forwarding pipelines, and any application that already uses JSON but has hit performance or bandwidth constraints, MessagePack is the natural next step.

Related conversions

Document conversions that follow this topic naturally:

• PDF to DOCX
• DOCX to PDF
• MD to PDF
• HTML to PDF
• ODT to PDF
• PDF to TXT