Most of the art of load testing is discovering the right question. Once you know the question, the right approach almost always becomes clear.
For example, when we wanted to know how a Rails app scaled, we picked Apache Benchmark. This worked well, producing the solid statistical evidence we were searching for, to be graphed and digested within a report.
Last month, we were asked to load test an application, to discover whether the audience could be safely let loose to play at a demo: Would the user experience be good enough at the expected load?
For subjective criteria, hard numbers are often tricky to interpret and may be misleading. Loading the system, then allowing users to evaluate the perceived performance is often a more revealing route. Apache JMeter simulates complex user interactions well, so running one scripted instance per expected user seemed a promising strategy.
A good plan in principle, but one that needed 40 spare machines and big fat pipes. Time for Amazon EC2.
- Through the web interface we
- Installed (JMeter and the load simulation) over
ssh, scripting in multi-threaded Ruby to avoid IO bottlenecks.
- Started the simulations.
- Played with the app, evaluating subjective performance at load.
- Remembered to destroy all instances in every region
All very smooth :-)