Avro: IDL & RPC

Avro IDL & RPC

Other serialization formats such as Thrift, Protocol Buffers, and others provide an Interface Definition Language(IDL). IDL allows the user to express a schema similar to you can write code in a programming language, rather than using JSON to specify the schema. Avro offers an equivalent: Avro IDL.

The Avro IDL comes with its own rules and syntax. It appear similar to popular languages like Java, C++, and Python. We’ll present a simple example demonstrating the use of IDL. More details can be found here. Each Avro IDL file defines a single Avro Protocol. When compiled, the output is a JSON format Avro Protocol file with extension .avpr. We can represent a car record in IDL format as follows:

/**
 * Car schema expressed in Avro IDL as a protocol
 */
@namespace("io.datajek")
protocol CarProtocol {

  record Car {
    string make;
    string model;
    int year;
    int horsepower;
  }

string carToString(Car car);
}

Once we define the .avdl file, we can convert it into an .avpr file like this:

java -jar avro-tools-1.9.1.jar idl carProtocol.avdl carProtocol.avpr

The protocol file produced has this content:

{
  "protocol" : "CarProtocol",
  "namespace" : "io.datajek",
  "doc" : "* Car schema expressed as Avro IDL",
  "types" : [ {
    "type" : "record",
    "name" : "Car",
    "fields" : [ {
      "name" : "make",
      "type" : "string"
    }, {
      "name" : "model",
      "type" : "string"
    }, {
      "name" : "year",
      "type" : "int"
    }, {
      "name" : "horsepower",
      "type" : "int"
    } ]
  } ],
  "messages" : {
    "carToString" : {
      "request" : [ {
        "name" : "car",
        "type" : "Car"
      } ],
      "response" : "string"
    }
  }
}

From the protocol file produced, we can generate the Java class Car as follows:

java -jar avro-tools-1.9.1.jar compile protocol carProtocol.avpr .

There’s an interface CarProtocol generated along with the class Car.

This may get confusing, given the many ways you can work with Avro. The following diagram shows the possible work-flows with Avro. Start with an .avdl file and auto-generate code. Or you could start with a .avsc file and auto-generate code. The final option is to skip auto-generated code and choose to work directly with the .avsc file.