Common Data Formats for Web APIs

Clients and servers exchange data using a request-response model, and the data format chosen for this exchange is a defining factor in API effectiveness. The format must be comprehensible to both parties, but selecting one is only the first step. Rapidly changing requirements make updates inevitable, and changes on the server side often demand corresponding client-side updates. Browser-based clients can simply refresh, but installed software (mobile apps, for example) relies on rolling updates where newer versions are gradually deployed. During rollout, at least two data format versions must coexist. This makes both data representation and data versioning critical concerns in API design.

Representation considerations

Several factors govern the choice of data format:

• Low latency: User-perceived latency is a primary concern. API SLAs may mandate response times within strict limits, particularly for time-critical applications like live video conferencing or real-time surveillance, where network delays are unacceptable.

• Fast processing: Human-readable formats aid debugging, especially when third-party applications consume the API. However, they are often less machine-friendly than binary alternatives. Processing time matters significantly when dealing with big dataData warehouses where terabytes of data are stored or retrieved in a single transaction. or resource-constrained IoT devices designed for small payloads.

• Multiple-format support: For public APIs consumed by external applications, key questions arise:

  • How many formats will the API support?

  • What schema will be used to encode and decode these formats?

  • How will data integrity be preserved during format conversion?

Many languages provide built-in encoding and decoding, but these methods are often inefficient in both processing time and encoded data size.

• Restricted data format: Business goals sometimes constrain format choices. A large organization like Google, with interconnected services (Maps, Search, and YouTube), cannot easily switch formats across its ecosystem. Changing the data format can also affect database storage, introducing unnecessary transformation overhead.

• Custom data format: Developers can invent custom encoding formats when strongly needed, but at the cost of interoperability: such data may only be understood by their own applications.

• Flexibility: APIs evolve, and newer versions may alter data formats. During rolling updates, you might need to roll back because of bugs or security issues, and multiple format versions must coexist. Data formats exhibit different types of flexibility:

  • Forward-compatible: Existing code can read data written by a newer codebase (new attributes are added gracefully).

  • Backward-compatible: New code can read data written by an older codebase (existing attributes are changed gracefully).

  • Fully compatible: Both forward- and backward-compatible.

The best format for any design problem is a trade-off among these factors. In general, a good format supports the following: