Questioning the High-Performance Holiday Report

Recently the engineers at Strangeloop Networks repeated their annual performance benchmarking of the top-ranked E-Commerce websites during the 2011 holiday online-shopping season (in November). This is a very popular and influential report in the E-Commerce industry. This is also an interesting report for me to read because I worked with a few of these companies last year to improve their website’s performance just to be prepared for the holiday load. The report is filled with commentary on trends in website development and performance characteristics of over 1000 websites, and while it may not appear to be controversial on first reading, there’s an interesting bullet-point in the Strangeloop conclusions:
Top-ranked sites are slower, not faster, than the rest of the pack. The primary reason: bigger pages containing more objects (files such as images, CSS, and JavaScript).”
This is a surprise, especially if you assume that the top websites to have the most priority, attention (and investment) to ensure high performance and scalability.  But then you see newcomer (ranked 78th) entering the report at #1 with page load times under 3 seconds on average. You might suspect, as I did, that this report is just evidence of an underdog effect for these lesser-ranked E-tailers ambitiously striving to compete, using every tactic available. Or it may indeed be proof that applying the web performance optimization (#wpo) rules are a very cost-effective competitive advantage to any company’s online initiatives.

But before you crack out the champagne to toast the winners, let’s take a closer look at what’s not revealed in the report:
  • The report includes no disclosure of the financial investment in the systems delivering these super-fast performance and response times. We all know that it costs money to throw hardware at a performance problem or bottleneck. Perhaps these companies just spent loads of money to make it into the top-ten. Or did they employ #wpo practices to optimize the sites?
  • Although the report doesn’t include directly observed qualitative assessment of the websites, it does reference a 2010 study of a 57% abandonment rate for sites with over 3-second page load time.  Specifically absent is any reference to the conversion rate measured by these top-ten websites during the timeframe of Strangeloop’s testing in November. Is slower more profitable?
  • Aside from pointing out that the page sizes are getting bigger and more complex, it would be helpful to understand the efficacy of the end-user experience. Was the extra size worth it? Is bigger better? Did customers notice the upgraded experience?
  • The “repeat view load time is 20% slower” means that while #wpo techniques work to improve the the initial loading of a web page, there is less attention to the repeated loading of the same page. Does this indicate a misaligned optimization for browser caching? Did any websites require repeated loading as part of their shopping cart processing?
  • “The average page load time is 10.0 seconds. (Median load time is 8.4 seconds.)” According to the 3-second threshold listed above, our E-commerce industry’s average conversion rate is considerably less than 30% – nowhere near our full potential.
Email the guys at Strangeloop for the detailed test results:


Post a Comment