Protocol Buffers (Protobuf): Google's Binary Serialization Format

Protocol Buffers (Protobuf): Google's Binary Serialization Format

What Are Protocol Buffers?

Protocol Buffers (commonly called protobuf) is a language-neutral, platform-neutral, extensible mechanism for serializing structured data — developed by Google and open-sourced in 2008. Unlike JSON or XML, protobuf is a schema-first binary format: you define your data structure in a .proto file using the Protocol Buffer language, then the protoc compiler generates strongly-typed serialization/deserialization code in your target language (Python, Go, Java, C++, JavaScript, Rust, C#, Kotlin, Swift, etc.).

Protobuf is used for:

• gRPC — the high-performance RPC framework built by Google; protobuf is its default wire format
• Internal APIs at Google, Uber, Netflix, Lyft, Square — most large technology companies use protobuf for service-to-service communication
• TensorFlow's SavedModel and TFRecord formats
• Protocol Buffers for configuration (Bazel BUILD files, Android resource linking)

Protobuf encoded messages are typically 3–10× smaller than equivalent JSON and 10–100× faster to encode/decode.

The .proto File: Schema Definition

// user.proto
syntax = "proto3";

package myapp;

option go_package = "myapp/pb";
option java_package = "com.myapp.proto";

// Enumerations
enum UserStatus {
  USER_STATUS_UNSPECIFIED = 0;  // proto3: default value must be 0
  USER_STATUS_ACTIVE = 1;
  USER_STATUS_SUSPENDED = 2;
  USER_STATUS_DELETED = 3;
}

// Nested message type
message Address {
  string street = 1;
  string city = 2;
  string country_code = 3;
  string postal_code = 4;
}

// Main message
message User {
  uint64 id = 1;
  string name = 2;
  string email = 3;
  UserStatus status = 4;
  Address address = 5;
  repeated string tags = 6;          // list/array
  map<string, string> metadata = 7;  // key-value map
  google.protobuf.Timestamp created_at = 8;
  optional string phone = 9;         // proto3 optional (distinguishes missing vs empty)
}

// Service definition (for gRPC)
service UserService {
  rpc GetUser (GetUserRequest) returns (User);
  rpc ListUsers (ListUsersRequest) returns (stream User);
  rpc CreateUser (CreateUserRequest) returns (User);
  rpc UpdateUser (UpdateUserRequest) returns (User);
}

message GetUserRequest {
  uint64 user_id = 1;
}

Field Numbers and Wire Format

Every field in a protobuf message has a field number (the integer after =). This is what's actually stored in the binary encoding — not the field name. This is the key to backward compatibility.

Wire types (3 bits):

• 0 — Varint (int32, int64, uint32, uint64, sint32, sint64, bool, enum)
• 1 — 64-bit (fixed64, sfixed64, double)
• 2 — Length-delimited (string, bytes, embedded messages, repeated fields)
• 5 — 32-bit (fixed32, sfixed32, float)

Encoding format: (field_number << 3) | wire_type as a varint, followed by the value.

Varint encoding is the key to protobuf's compactness: small integers (0–127) take 1 byte; larger integers take 2+ bytes proportional to their magnitude. A value of 1 takes 1 byte regardless of whether the field type is int32 or int64.

Example Binary Encoding

User { id: 42, name: "Alice", status: ACTIVE } in protobuf:

08 2A        → field 1 (id), varint: 42
12 05        → field 2 (name), length-delimited: 5 bytes
  41 6C 69 63 65   → "Alice"
20 01        → field 4 (status), varint: 1 (ACTIVE)

Total: 11 bytes. JSON equivalent {"id":42,"name":"Alice","status":"ACTIVE"} = 39 bytes.

Working with Protobuf in Python

Installation

pip install protobuf grpcio-tools

# Compile .proto to Python
python -m grpc_tools.protoc \
  -I./protos \
  --python_out=./generated \
  --grpc_python_out=./generated \
  ./protos/user.proto

Encoding and Decoding

# generated/user_pb2.py is auto-generated
from generated import user_pb2
from google.protobuf import timestamp_pb2
from google.protobuf.json_format import MessageToJson, ParseDict
import datetime

# Create a message
user = user_pb2.User()
user.id = 42
user.name = "Alice Smith"
user.email = "alice@example.com"
user.status = user_pb2.USER_STATUS_ACTIVE
user.address.street = "123 Main St"
user.address.city = "Springfield"
user.address.country_code = "US"
user.tags.extend(["admin", "verified"])
user.metadata["plan"] = "pro"
user.metadata["locale"] = "en-US"

# Set timestamp
ts = timestamp_pb2.Timestamp()
ts.FromDatetime(datetime.datetime(2024, 1, 15, 12, 0, 0,
                tzinfo=datetime.timezone.utc))
user.created_at.CopyFrom(ts)

# Serialize to bytes
binary = user.SerializeToString()
print(f"Binary size: {len(binary)} bytes")

# Deserialize from bytes
user2 = user_pb2.User()
user2.ParseFromString(binary)
print(user2.name)          # "Alice Smith"
print(user2.status)        # 1 (USER_STATUS_ACTIVE)
print(list(user2.tags))    # ['admin', 'verified']

# Convert to/from JSON (for debugging)
json_str = MessageToJson(user)
user3 = user_pb2.User()
ParseDict({'id': 99, 'name': 'Bob'}, user3)

# Size comparison
import json
json_size = len(json_str.encode())
print(f"Binary: {len(binary)}B  JSON: {json_size}B  ratio: {len(binary)/json_size:.2f}")

gRPC Service Implementation

import grpc
from concurrent import futures
from generated import user_pb2, user_pb2_grpc

class UserServicer(user_pb2_grpc.UserServiceServicer):
    def GetUser(self, request, context):
        # request.user_id is the field from GetUserRequest
        user = fetch_user_from_db(request.user_id)
        return user_pb2.User(id=user.id, name=user.name, email=user.email)

    def ListUsers(self, request, context):
        # Streaming RPC: yield multiple responses
        for user in get_all_users():
            yield user_pb2.User(id=user.id, name=user.name)

# Start server
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
user_pb2_grpc.add_UserServiceServicer_to_server(UserServicer(), server)
server.add_insecure_port('[::]:50051')
server.start()
server.wait_for_termination()

Schema Evolution: Forward and Backward Compatibility

Protobuf's field-number-based encoding makes schema evolution safe:

reserved keyword prevents field number reuse:

message User {
  reserved 3, 15, 9 to 11;         // these numbers are retired
  reserved "old_field_name";        // this name is retired
  uint64 id = 1;
  string name = 2;
  // field 3 was "phone" — now reserved
}

proto3 vs. proto2

proto3 (current, recommended):

• All fields are optional by default; no required
• Default values: 0 for numbers, empty string for strings, false for bool
• optional keyword (proto3 optional) lets you detect whether a field was set
• No default option on fields
• Unknown fields are preserved (since proto3.5)

proto2 (legacy):

• required / optional / repeated keywords
• Custom default values allowed
• More verbose but more explicit

Comparison: Protobuf vs. JSON vs. Avro vs. Thrift

Practical Tips

• Never change a field number — it breaks backward compatibility with all existing serialized data
• Always reserve retired field numbers and names immediately when removing a field
• Use uint64 not int64 for IDs and non-negative integers — no sign extension overhead in varint encoding
• Use sint32/sint64 (zigzag encoding) for integers that can be negative — much more efficient than int32 for negatives
• Don't use required (proto2) — it makes schema evolution nearly impossible
• Proto3 optional is not the same as proto2 optional — use it when you need to distinguish "field not set" from "field set to default value"
• Well-known types (google.protobuf.Timestamp, Duration, Struct, Any, Empty) cover most common cross-language type needs — prefer them over custom re-implementations
• Use --experimental_allow_proto3_optional flag during protoc compilation for proto3 optional support in older toolchains

Protocol Buffers' schema-first design, compact binary encoding, generated code, and excellent schema evolution story make it the gold standard for service-to-service communication in microservice architectures.

Related conversions

Frequent conversions across the catalogue:

• JPG to PNG
• MP4 to MP3
• PDF to DOCX
• PNG to JPG
• MP3 to WAV
• DOCX to PDF