What Are Protocol Buffers?
Protocol Buffers (protobuf) is Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. Invented at Google in 2001 and open-sourced in 2008, protobuf is used internally at Google for almost every inter-service communication — billions of messages per second across thousands of services. It is the serialization format behind gRPC, Google's high-performance RPC framework.
Unlike JSON or XML, Protocol Buffers are a binary format: the encoded data is not human-readable, but it is dramatically smaller and faster to serialize/deserialize. This makes protobuf the serialization format of choice for high-throughput microservices, mobile apps that care about bandwidth, and any system where performance matters at scale.
The .proto Schema File
Protocol Buffers use a schema-first approach. You define your data structures in a .proto file using the Protocol Buffer Language (proto3) — a purpose-built IDL (Interface Definition Language). The protoc compiler then generates efficient serialization/deserialization code in your target language from this schema.
syntax = "proto3";
package ecommerce;
option go_package = "github.com/example/ecommerce/pb";
// A product in the catalog
message Product {
uint32 id = 1;
string name = 2;
string description = 3;
float price = 4;
ProductCategory category = 5;
repeated string tags = 6;
map<string, string> attributes = 7;
}
enum ProductCategory {
CATEGORY_UNSPECIFIED = 0;
CATEGORY_ELECTRONICS = 1;
CATEGORY_CLOTHING = 2;
CATEGORY_BOOKS = 3;
}
// Request for listing products
message ListProductsRequest {
ProductCategory category = 1;
uint32 page_size = 2;
string page_token = 3;
}
// Response for listing products
message ListProductsResponse {
repeated Product products = 1;
string next_page_token = 2;
}
// Service definition (used with gRPC)
service ProductCatalog {
rpc ListProducts (ListProductsRequest) returns (ListProductsResponse);
rpc GetProduct (GetProductRequest) returns (Product);
}
The numbers (= 1, = 2, etc.) are field numbers — these are the actual identifiers used in the binary encoding. Field names in the .proto file do not appear in the serialized binary at all; the number is everything. This is why field numbers, once assigned and deployed, must never be reused for different fields.
Proto3 vs Proto2
Proto2 (2008) — the first public version. Supports required, optional, and repeated field labels. Allows custom default values. Still used in some legacy systems.
Proto3 (2016, current) — simplified field labels (everything is optional by default). Removed required. Added map fields natively. Better support for unknown fields. JSON mapping is standardized. Recommended for all new projects.
// proto2
message OldMessage {
required string name = 1; // must be present
optional int32 age = 2 [default = 0];
}
// proto3
message NewMessage {
string name = 1; // optional by default, zero value if absent
int32 age = 2;
}
Data Types in Protocol Buffers
| Proto type | Notes | Generated type (Go / Java / Python) |
|---|---|---|
double | 64-bit float | float64 / double / float |
float | 32-bit float | float32 / float / float |
int32 | Variable-length, inefficient for negatives | int32 / int / int |
int64 | Variable-length | int64 / long / int |
uint32 / uint64 | Unsigned | uint32/uint64 / int / int |
sint32 / sint64 | ZigZag-encoded, efficient for negatives | int32/int64 / int / int |
fixed32 / fixed64 | Always 4/8 bytes | uint32/uint64 / int/long / int |
bool | true/false | bool / boolean / bool |
string | UTF-8 text | string / String / str |
bytes | Arbitrary binary data | []byte / ByteString / bytes |
message | Nested message (object) | struct / class / class |
enum | Named integer constants | iota / enum / IntEnum |
repeated | List / array | slice / List / list |
map<K,V> | Hash map | map / Map / dict |
Working with Protocol Buffers
Installing protoc
# macOS (Homebrew)
brew install protobuf
# Ubuntu/Debian
apt install -y protobuf-compiler
# Windows (Chocolatey)
choco install protoc
# Verify
protoc --version # libprotoc 26.x
Generating Code
# Generate Go code
protoc --go_out=. --go-grpc_out=. proto/product.proto
# Generate Python code
protoc --python_out=. proto/product.proto
# Generate Java code
protoc --java_out=. proto/product.proto
# Generate TypeScript (requires ts-proto plugin)
protoc --plugin=protoc-gen-ts_proto=./node_modules/.bin/protoc-gen-ts_proto \
--ts_proto_out=. proto/product.proto
Serializing and Deserializing (Python)
# pip install protobuf
import ecommerce_pb2 # generated file
# Create and serialize a message
product = ecommerce_pb2.Product()
product.id = 42
product.name = "Wireless Headphones"
product.price = 79.99
product.category = ecommerce_pb2.CATEGORY_ELECTRONICS
product.tags.extend(["audio", "bluetooth"])
# Serialize to binary bytes
binary_data = product.SerializeToString()
print(f"Binary size: {len(binary_data)} bytes")
# Deserialize
product2 = ecommerce_pb2.Product()
product2.ParseFromString(binary_data)
print(product2.name) # "Wireless Headphones"
# Convert to JSON for debugging
from google.protobuf import json_format
json_str = json_format.MessageToJson(product)
print(json_str)
Inspecting .proto Files Without Code Generation
# Decode a binary protobuf file without a schema (shows field numbers + raw values)
protoc --decode_raw < data.bin
# Decode with schema
protoc --decode=ecommerce.Product proto/product.proto < data.bin
Performance vs JSON
Protocol Buffers consistently outperform JSON in benchmarks:
| Metric | JSON | Protocol Buffers |
|---|---|---|
| Encoded size | 1× (baseline) | 3–10× smaller |
| Serialization speed | 1× | 3–6× faster |
| Deserialization speed | 1× | 5–10× faster |
| Schema required | No | Yes |
| Human-readable | Yes | No |
| Language support | Universal | 15+ official, more via plugins |
The size advantage is particularly significant for repeated structures. A JSON array of 1,000 objects repeats every field name 1,000 times; protobuf repeats only field numbers (1-3 bytes each).
When to Use Protocol Buffers
Use protobuf when:
- You are building microservices (especially with gRPC)
- Bandwidth or latency is constrained (mobile apps, high-frequency APIs)
- You have large volumes of homogeneous structured data to store or stream
- You need strong typing and schema evolution across multiple services/languages
- You are dealing with data that will be read/written by multiple languages
Do not use protobuf when:
- You need human-readable output (use JSON)
- You need a schema-free format (use JSON)
- You are building a simple internal API with a single language (JSON is fine)
- You need to stream data to a browser without a protobuf library (use JSON)
Schema Evolution: The Killer Feature
One of protobuf's most important advantages is safe schema evolution. Because field numbers (not names) identify fields in the binary, you can:
- Add new fields freely — old decoders simply ignore unknown field numbers
- Remove fields by marking them
reserved— prevents accidental reuse - Rename fields without breaking the binary format
message Product {
reserved 8, 9; // deleted field numbers — never reuse these
reserved "old_field"; // deleted field name — prevents accidental reuse
uint32 id = 1;
string name = 2;
// price field 3 removed — use reserved above
string sku = 4; // new field added in v2 — old clients ignore it
}
This allows long-lived distributed systems to evolve their data model without coordinating simultaneous deployments of all services — a critical operational advantage at scale.
Protocol Buffers in the Ecosystem
- gRPC: Built on protobuf. Defines both messages and service interfaces in
.protofiles. Generates client and server stubs automatically. - Google Cloud: Pub/Sub, BigQuery, and most Google Cloud APIs use protobuf internally.
- Apache Kafka: Many high-throughput event streaming systems use protobuf for message schemas, managed by Schema Registry (Confluent).
- Machine Learning: TensorFlow's model format (SavedModel) uses protobuf. Training datasets in
.tfrecordformat contain protobuf-encoded examples. - Android: Used extensively internally; the Protocol Buffer Lite library is optimized for mobile.
Related conversions
Frequent conversions across the catalogue: