Web testing in general is an oversaturated market. There's also a lot of high-level language support for various web protocols, so bodging together something not notably more interesting than Apache's 'ab' and hanging it out on sourceforge is a popular practice. Our requirements for a standardized testing service are somewhat more constrained, however. Needs include:
- HTTP and HTTPS queries
- built-in support for SOAP-based queries
- support for multithreaded, high-performance, high-load parallel testing
Beyond that, we're looking for a mature product with good vendor or community support.
Within those constraints, the options dwindle surprisingly quickly. Selected for more intensive evaluation were:
- Apache JMeter: a popular Jakarta project, fairly polished load testing tool that includes off-the-shelf support for SSL and queries to SOAP-based web services. (http://jakarta.apache.org/jmeter/index.html)
- Tsung, specificially designed to stress-test HTTP, SOAP and Jabber servers. Started out as a Jabber testing tool, more basic than jmeter, and is more XML-oriented. (http://tsung.erlang-projects.org/)
- TestMaker, also open-source but not GPL. Its primary thrust is SOA service testing, but it provides a nice-looking framework for scripting fairly arbitrary tests; also extends out of the HTTP universe. (http://www.pushtotest.com/Downloads/features.html)
Of these tools, TestMaker seems most technically interesting, jmeter has the most community support, and Tsung is in the middle somewhere. All of them can run tests and generate reports noninteractively. (All can probably also do the job required.) One also gets a choice between java, xml/erlang,
and jython/java. Tsung may also be interesting for its out-of-box jabber testing support.
There are various commercial applications that handle this sort of testing as well, but they don't currently offer us much. The 800-pound gorilla in the commercial field is LoadRunner, and from all reports it is exceptionally cumbersome to get going. Not to mention expensive, without source code, and overfeatured for our needs.
[Adapted from a report written in September 2006.]