We've been hard at work putting together web traffic reporting for all sites on Acquia Hosting, and it's now available in the Acquia Network subscription interface. These statistics provide a level of traffic detail across our hosting stack that is not available from other sources such as Google Analytics or log file analyzers like awstats, particularly, how many requests are being served from each layer of the Drupal and LAMP stack. We're also making available the fully expanded data behind these numbers as a CSV file download ready for import into your reporting tool of choice.

Where the numbers come from

We rotate all server logs once per day, but just before rotating each log file we run it through a custom-built log summarizing utility we developed in-house. The summarized data includes: total HTTP requests, total GET requests, total POST requests, total "File not found" responses, server errors, bytes sent, and number of "pages" vs. number of "images" served. We then import that summary data into a central database.

Details, details

The Hosting stats interface offers a few different views of the summaraized data:

• Summary by day, week, month
• Details by individual host names
• Total requests: How many HTTP served through the load balacner
• Uncached requests: How many HTTP went through the load balancer and were served from the backend app servers
• Drupal requests: How many (uncached) requests invoked Drupal

This view of the data gives you sense of how much content is being delivered from cache vs backend servers.

The "All data" CSV file download offers far more detail:

• Server id: Which server delivered these requests
• Service: Which server layer delivered these requests (Varnish, Nginx, or Drupal)
• hits: Total HTTP requests
• GETs: Total GET requests
• POSTs: Total POST requests
• OK: Total HTTP OK responses
• not found: Total HTTP 404 responses
• server errors: Total HTTP responses where response code was greater than 450
• bytes sent: Total bytes sent from the load balancer
• pages: Total number of requests where the request url resembles that of a typical page request
• images: Total number of requests where the request url resembles either an image, css, javascript, Flash, icon, or other type of common static content

These stats are all being gathered on a daily basis.

What the future holds

For starters we'll be importing historic server log data from the previous 4 months, so soon those stats will appear as well.

We also intend to release the source code for our server log summarizing utility. We feel it could useful for anyone with a need to summarize large web server logs quickly.

And finally, I hesitate to dangle this feature tease out here, but I'll drop a hint: realtime.

Stay tuned.