Taking the Pulse

Never forget the work

When we arrived on Wednesday night, I immediately set up my laptop in my parents’ home office. I can work anywhere I have broadband and a cell phone. Using their 3 MB cable broadband, I used PuTTY to log into our jumphost and start up my sampling scripts.

Looking up the GUI

During the run-up to the launch, I was part of load testing this new site. Most load tests deliver results after the test is done. Since the data come from the load generators rather than inside the systems under test, it is a “black-box” test. To get more information out of the load test, I had started off using the application server’s HTML administration GUI to check vitals like latency, free heap memory, active request-handling threads, and active sessions. If we don’t know in advance what we are looking for, then a GUI is a great way to explore the system. If we know exactly what we want, the GUI gets tedious. On the other hand, if we need to look at thirty or forty servers at a time, the GUI gets downright impractical.

Making the perl modules

To get more out of our load tests, I wrote a collection of Perl modules that would screen-scrape the admin GUI for me, parsing the HTML for values. These modules would let me get and set property values and invoke methods on the components of the application server, built-in as well as custom. Because the entire admin GUI was HTML-based, the application server never knew the difference between a Perl module or a web browser. Armed with these Perl modules, I was able to create a set of scripts that would sample all the application servers for their vital stats, print out detail and summary results, sleep a while, and loop. They were simple indicators, but in the time since site launch, all of us had learned the normal rhythm and pulse of the site by watching these stats. We knew with a single glance, what was normal for noon on Tuesday in July. If session counts went up or down from the usual envelope, if the count of orders placed just looked wrong, we would know. It’s really surprising how quickly we can learn to detect problems.