Subscribe to our
weekly update
Sign up to receive our latest news via a mobile-friendly weekly email
In this article you will learn how to benchmark web performance against competitors using three steps.
One of the frequent questions I receive from clients is on “How do I benchmark my performance against the competition?”. There are different approaches to benchmarking, some better than others. The key to a successful benchmark, is to plan it carefully and collect the right data points.
I recommend companies to follow the following 3 steps:
Over the years I have come across several benchmarks that failed for various reasons. Some of the major pitfalls are:
Case Study: E-commerce Benchmark
Recently we assisted an e-commerce customer that had created a benchmark in Catchpoint to compare how the homepages of key competitors ranked. The benchmark included the homepages of BestBuy, Amazon, Apple, and Newegg. The goal was to understand where their homepage ranked relative to their competitors, and to determine the steps to improve their web performance.
Based on the data collected they came to the conclusion that the homepage of Apple.com was the fastest. There are several factors on why Apple’s homepage is fast:
This might seem like a successful benchmark, however, there was one little issue that made the benchmark inaccurate**.**
The goal of the client was to compare the homepages of the competing ecommerce sites. But in the case of Apple they were testing the corporate homepage, which had a different business goal – and therefore different design and implementation. The homepage of the e-commerce site for Apple is www.apple.com/store and not www.apple.com.
When benchmarking to the correct e-commerce site of Apple, the picture changed. Apple was not much faster than the rest of the stores. (I kept the home page of Apple to show the differences).
To get a better look at the impact at user experience, we also looked at other metrics like time to title and render start time.
Visually, this is what it looked like loading those 5 sites from a node located in New York on the Verizon Backbone (using a 400 ms timer, the blink of an eye is 400 ms).
We also implemented the use of Apdex, an excellent way to score and compare numbers from diverse pages. Apdex normalizes the data based on target goals, which vary from webpage to webpage (as we saw with Apple). For demonstration purposes I used an Apdex target response of 5,000 ms (5 seconds) for all the tests above.
To sum it up, a successful benchmark depends on clear end goals, everything else depends on it.
Happy Benchmarking!
Mehdi – Catchpoint
(methodology: all sites were measured using 26 US Nodes, every 10 minutes with Internet Explorer 8.)