Measuring traffic #

So far, we measured the latency of our applications, and we created alerts that fire when certain thresholds based on request duration are reached. Those alerts are not based on the number of requests coming in (traffic), but on the percentage of slow requests. The AppTooSlow would fire even if only one single request enters an application, as long as the duration is above the threshold. For completeness, we need to start measuring traffic or, to be more precise, the number of requests sent to each application and the system as a whole. Through that, we can know if our system is under a lot of stress and make a decision on whether to scale our applications, add more workers, or apply some other solution to mitigate the problem. We might even choose to block part of the incoming traffic if the number of requests reaches abnormal numbers providing a clear indication that we are under Denial of Service (DoS) attack.

We’ll start by creating a bit of traffic that we can use to visualize requests.

for i in {1..100}; do
    curl "http://$GD5_ADDR/demo/hello"
done

open "http://$PROM_ADDR/graph"

We sent a hundred requests to the go-demo-5 application and opened the Prometheus's graph screen.

Retrieve the number of requests #

We can retrieve the number of requests coming into the Ingress controller through the nginx_ingress_controller_requests. Since it is a counter, we can continue using the rate function combined with sum. Finally, we probably want to know the rate of requests grouped by the ingress label.

Please type the expression that follows, press the Execute button, and switch to the Graph tab.

sum(rate(
  nginx_ingress_controller_requests[5m]
)) 
by (ingress)

We can see a spike on the right side of the graph. It shows the requests that went to the go-demo-5 applications through the Ingress with the same name. In my case (screenshot below), the peak is close to one request per second (yours will be different).