Designing Logging and Aggregation System

In a large system, important sub-services (like web layer, app layer, etc.) continuously produce diagnostic data in the form of local logs. Different monitoring services need to consume this huge data for timely alerts.

The key observation here is that producers and consumers of data might be working. at different speeds, and might be active at different times. We need to decouple producers and consumers of the data so that they could work asynchronously.

In the above diagram:

• tailers are the processors with no global reduce step
• aggregators are processors with global reduce steps (Lobster is an in-memory aggregator)
• writers to scribe can re-try or send data to different scribe servers if the first one does not respond
• Communication between clients and service use RPC (for example Thrift based RPC)

We use a publish-subscribe system (Scribe) where data producers can publish the data. Publishing is like writing except that the system allows consumers to subscribe for different data, and once everyone has consumed a specific data, it could be garbage ...