There are times when you need to test every page in an Umbraco installation for errors - for example, after performing a major upgrade, or installing new components.

Working on such a site recently, which needed upgrading from Umbraco 6.1.6 to 6.2.5, I needed to perform such a test. Doing this manually would have been especially arduous and time-consuming on a site with several hundred pages. So the question is, how to automate this kind of testing?

If you've ever done substantial load testing on a website before, you'll probably be familiar with JMeter - an Apache Software Foundation project designed to be a useful tool in the arsenal of any web developer. JMeter allows you to simulate multiple users to your website, to allow you to test the throughput & performance of your code and analyse the results.

JMeter has a number of useful features which allow it to do more than just "slam" your website for performance benchmarking. These include control statements (such as if-else and for-each), and response parsing tools.

By manipulating these, you can set up a basic (single-threaded, sadly) crawler that will parse your entire site, looking for a given text expression in the output. Umbraco will, when a macro fails to execute correctly, output the string "Error loading macroengine script" followed by the filename of the script that failed to run. We can pick up this string (plus any ASP.NET error 500's thrown for more traditional MVC errors).

Firstly, we set up JMeter to crawl a sitemap.xml:

Initial JMeter spider setup pointing to my sitemap.xml

We then add an XPath Extractor to select the <loc>tags in the sitemap, as we will then use these to crawl the complete site. The XPath is /urlset/url/loc and each value is outputted into a testUrl variable.

Setting up XPath Extractor for the spider

We then proceed to set up a for-each loop, iterating over every value of testUrl, outputting it to a local variable called url.

Setting up the for-each iterator variables

Finally, we pass the url variable to a HTTP Request action, which fetches the page in question. Note the ${url} parameter.

HTTP Request with the looped variable

Lastly, we set up a listener, to catch any incidences of the required text string "Error loading macroscript". Note the "Not" checkbox - checking this means pages that fail to contain our string will "pass", whereas pages that do contain the string will "fail" and therefore be flagged. Any pages that fail to return a 2xx or 3xx code will also "fail" in this test.

Conditional assertion to catch pages that contain errors

Let's run it!

List result of JMeter test, with a number of errors

Eesh, that's not pretty. I need to fix a few URL rewrites!

The JMeter test plan can be downloaded here. Save the file with a .jmx extension, open it in JMeter, adjust the URLs accordingly, and let it loose on your sites. Let me know if you find it useful!