Streaming Data
Explore how to stream data efficiently in Python REST APIs by leveraging Redis Pub/Sub for real-time event delivery. Understand how to avoid polling overhead, integrate streaming endpoints, and build scalable applications using streaming mechanisms like Server-Sent Events.
We'll cover the following...
A common pattern with any HTTP API is the need to receive events. In a lot of cases, there is no way to achieve this other than regularly polling the API. That can cause a lot of stress on the HTTP endpoint, as it requires setting up a new connection, which means a lot of overload with TCP and SSL.
Streaming basics
A more efficient way is to use streaming. An adequate technology here is ServerSent Events message protocol defined by HTML5. Alternatively, it would be possible to use Transfer-Encoding:chunked defined by HTTP/1.1 or even the WebSocket protocol. However, chunked encoding is more complicated, and the WebSocket is a little bit overkill for the simple use case presented here.
To implement any streaming mechanism in a scalable and efficient manner, you need to be sure that your backend offers that feature. It could be a messaging queue, a database or any other software that provides a stream of events that the application can subscribe to.
If the offered API has to poll its backend regularly to know about new ...