Apache Avro: Schema-Based Data Serialization for Big Data

Apache Avro: Schema-Based Data Serialization for Big Data

Apache Avro is a data serialization framework developed as part of the Apache Hadoop project in 2009. It uses JSON to define schemas and produces compact binary output, combining the schema-first discipline of Protobuf with JSON's human-readable schema authoring. Avro has become the dominant serialization format in Apache Kafka ecosystems, particularly when paired with a Schema Registry — making it the standard for event-driven architectures, data streaming pipelines, and microservice messaging at scale.

What Makes Avro Different

Several properties distinguish Avro from other serialization formats:

Schema-in-file: Unlike Protobuf (which stores schema separately in .proto files and uses field numbers as identifiers), Avro serializes the schema alongside the data in Avro container files (.avro). This means an Avro file is self-contained and can be decoded without external schema files.

Dynamic typing support: Avro schemas are JSON, readable without a compiler, and can be generated or modified programmatically at runtime — useful for schema-driven dynamic data pipelines.

Kafka-native: The Confluent Platform Schema Registry stores Avro schemas centrally, assigning each schema a numeric ID. Kafka messages then carry only a 5-byte schema ID prefix + the Avro binary payload — eliminating per-message schema overhead while enabling decentralized schema evolution.

Row-oriented: Unlike Parquet (columnar), Avro stores data row by row, making it efficient for record-at-a-time streaming but less efficient for column-selective analytical queries.

Avro Schema Language

Avro schemas are written in JSON. The primitive types are:

"null", "boolean", "int", "long", "float", "double", "bytes", "string"

Complex types include records, enums, arrays, maps, unions, and fixed-size byte sequences:

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.users",
  "doc": "A registered user in the system",
  "fields": [
    {"name": "id", "type": "long", "doc": "Unique user identifier"},
    {"name": "username", "type": "string"},
    {"name": "email", "type": "string"},
    {
      "name": "status",
      "type": {
        "type": "enum",
        "name": "UserStatus",
        "symbols": ["ACTIVE", "INACTIVE", "SUSPENDED", "DELETED"]
      },
      "default": "ACTIVE"
    },
    {
      "name": "profile",
      "type": {
        "type": "record",
        "name": "UserProfile",
        "fields": [
          {"name": "full_name", "type": "string"},
          {"name": "country_code", "type": "string"},
          {"name": "age", "type": ["null", "int"], "default": null}
        ]
      }
    },
    {
      "name": "roles",
      "type": {"type": "array", "items": "string"},
      "default": []
    },
    {
      "name": "metadata",
      "type": {"type": "map", "values": "string"},
      "default": {}
    },
    {
      "name": "created_at",
      "type": {
        "type": "long",
        "logicalType": "timestamp-millis"
      }
    },
    {
      "name": "avatar_data",
      "type": ["null", "bytes"],
      "default": null,
      "doc": "Optional binary avatar image"
    }
  ]
}

Unions and Nullable Fields

The ["null", "int"] pattern for nullable fields is idiomatic Avro — a union of null and int. The first type in a union is the default type, so "default": null requires null to be listed first. This is the most common source of confusion for Avro newcomers.

Logical Types

Avro logical types annotate primitive types with semantic meaning:

Binary Encoding

Avro's binary encoding is compact and field-order-dependent. Unlike Protobuf (which uses field numbers as tags), Avro relies entirely on schema agreement between writer and reader — there are no field identifiers in the binary stream. The reader must use the same schema (or a compatible evolved schema) as the writer.

For the User record above, encoding proceeds field by field in schema order:

id (long):              ZigZag varint → 84 (42 ZigZag-encoded)
username (string):      length (varint) + UTF-8 bytes
email (string):         length (varint) + UTF-8 bytes
status (enum):          ZigZag varint index → 0 = ACTIVE
profile (record):       nested field-by-field
  full_name (string):   ...
  country_code (string): ...
  age (union):          type_index (0=null, 1=int) + value
roles (array):          block_count (varint) + items + 0 terminator
metadata (map):         block_count (varint) + key-value pairs + 0 terminator
created_at (long):      ZigZag varint milliseconds
avatar_data (union):    type_index + bytes

Integer values use ZigZag encoding (same as Protobuf sint64) to minimize bytes for small values and negative numbers.

Arrays and maps use block encoding: a block count followed by that many items, then another block count, continuing until a terminating block count of 0. Negative block counts indicate that the total byte size of the block follows, enabling skip-ahead for readers that don't need to decode every item.

Python: fastavro

The fastavro library is the recommended Python interface for Avro (it is significantly faster than the official avro package):

import fastavro
import io
from datetime import datetime, timezone

# Define schema programmatically
schema = fastavro.parse_schema({
    "type": "record",
    "name": "Order",
    "namespace": "com.example.orders",
    "fields": [
        {"name": "order_id", "type": "long"},
        {"name": "customer_id", "type": "long"},
        {"name": "product_sku", "type": "string"},
        {"name": "quantity", "type": "int"},
        {"name": "price_cents", "type": "long"},
        {"name": "currency", "type": "string", "default": "USD"},
        {"name": "placed_at", "type": {"type": "long", "logicalType": "timestamp-millis"}},
        {"name": "notes", "type": ["null", "string"], "default": None},
    ]
})

# Sample records
records = [
    {
        "order_id": 1001,
        "customer_id": 42,
        "product_sku": "WIDGET-A",
        "quantity": 3,
        "price_cents": 2999,
        "currency": "USD",
        "placed_at": int(datetime(2024, 3, 15, 10, 30, tzinfo=timezone.utc).timestamp() * 1000),
        "notes": None,
    },
    {
        "order_id": 1002,
        "customer_id": 77,
        "product_sku": "GADGET-B",
        "quantity": 1,
        "price_cents": 14999,
        "currency": "EUR",
        "placed_at": int(datetime(2024, 3, 15, 11, 45, tzinfo=timezone.utc).timestamp() * 1000),
        "notes": "Gift wrapping requested",
    },
]

# Write to Avro container file
with open('orders.avro', 'wb') as f:
    fastavro.writer(f, schema, records, codec='snappy')

# Read from Avro container file
with open('orders.avro', 'rb') as f:
    reader = fastavro.reader(f)
    print(f"Writer schema: {reader.writer_schema['name']}")
    for record in reader:
        print(f"Order {record['order_id']}: {record['product_sku']} x{record['quantity']}")

# In-memory serialization (for Kafka)
def avro_serialize(schema, record):
    buf = io.BytesIO()
    fastavro.schemaless_writer(buf, schema, record)
    return buf.getvalue()

def avro_deserialize(schema, data):
    buf = io.BytesIO(data)
    return fastavro.schemaless_reader(buf, schema)

raw = avro_serialize(schema, records[0])
print(f"Serialized size: {len(raw)} bytes")
decoded = avro_deserialize(schema, raw)

Avro with Kafka and Schema Registry

Avro's killer feature in production is its integration with the Confluent Schema Registry:

from confluent_kafka import Producer, Consumer
from confluent_kafka.serialization import SerializationContext, MessageField
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer, AvroDeserializer

# Connect to Schema Registry
schema_registry_conf = {'url': 'http://schema-registry:8081'}
schema_registry_client = SchemaRegistryClient(schema_registry_conf)

# Define schema (will be registered automatically)
schema_str = json.dumps({
    "type": "record",
    "name": "OrderEvent",
    "namespace": "com.example",
    "fields": [
        {"name": "order_id", "type": "long"},
        {"name": "event_type", "type": "string"},
        {"name": "occurred_at", "type": {"type": "long", "logicalType": "timestamp-millis"}},
    ]
})

avro_serializer = AvroSerializer(schema_registry_client, schema_str)
avro_deserializer = AvroDeserializer(schema_registry_client)

# Produce with Avro serialization
producer = Producer({'bootstrap.servers': 'kafka:9092'})

event = {"order_id": 1001, "event_type": "PLACED", "occurred_at": 1710000000000}
producer.produce(
    topic='order-events',
    value=avro_serializer(event, SerializationContext('order-events', MessageField.VALUE)),
)
producer.flush()

# Consume with Avro deserialization
consumer = Consumer({
    'bootstrap.servers': 'kafka:9092',
    'group.id': 'order-processor',
    'auto.offset.reset': 'earliest',
})
consumer.subscribe(['order-events'])

msg = consumer.poll(10.0)
if msg:
    decoded = avro_deserializer(msg.value(), SerializationContext('order-events', MessageField.VALUE))
    print(f"Event: {decoded}")

The wire format for a Schema Registry Avro message is:

[0x00][4-byte schema ID][Avro binary payload]

The 0x00 magic byte identifies the message as a Schema Registry message. The 4-byte schema ID allows the consumer to fetch the correct schema from the registry and decode the payload.

Schema Evolution Rules

Avro's schema evolution is more flexible than Protobuf's, governed by compatibility rules:

Backward Compatibility (new reader, old data)

• ✅ Add a field with a default value
• ✅ Remove a field that had a default value
• ❌ Add a field without a default value
• ❌ Remove a required field (no default)
• ❌ Change a field's type incompatibly

Forward Compatibility (old reader, new data)

• ✅ Remove a field
• ✅ Add a field with a default value
• ❌ Rename a field (old reader doesn't recognize new name)

Full Compatibility (both directions)

• ✅ Only add/remove fields that have default values

The Schema Registry enforces these compatibility rules automatically when you try to register a new schema version, preventing breaking changes from reaching production consumers.

Avro vs Parquet vs Protobuf

Conclusion

Apache Avro occupies a specific and important niche: row-oriented streaming serialization with flexible JSON-based schema evolution and native Kafka/Schema Registry integration. While Protobuf is more efficient for microservice RPC and Parquet is superior for analytical storage, Avro's combination of human-readable schemas, robust evolution rules enforced by the Schema Registry, and row-oriented encoding optimized for record-at-a-time streaming makes it the practical standard for Kafka-based event architectures. Understanding Avro's schema language, binary encoding, union patterns, and Schema Registry integration prepares you for building production-grade event-driven systems where schema changes must not break running consumers.

Related conversions

Frequent conversions across the catalogue:

• JPG to PNG
• MP4 to MP3
• PDF to DOCX
• PNG to JPG
• MP3 to WAV
• DOCX to PDF