Sunday, 16 August 2009

Rigour in Testing

This week I went to a presentation by James Bach on the Myths of Rigour at the Wellington Testing Professional Network Presentation.

The biggest thing that I took away/had reafrimed was that rigour/process should not be blindly followed. The person executing the process needs to understand why they are doing what they are doing. If something is just being blindly followed aspects of it may be missed as instructions only cover what the instruction writer knows to write down there is a lot of steps that are implied or just done because not doing them would be stupid or not mean things would work right, so the writer doesn't think to write them down. If the background knowledge is there and the understanding of the process is there these implied steps should be piked up naturally and issues will be detected better. Also part of a tester personal attributes should be inquisitive and personally want to have an understanding an a in depth knowledge of how things work and why they work that way. Else you might as well just believe the developer when they say yes it is tested and work fines.

If you just follow a process how do you know if the output you are producing are correct or are helpful? This can be for testing itself but also for the documentation that surrounds testing. There are templates which say a test plan should look like X and a test case should look like Y. But if you don't know the why behind all the headings is it helpful? Also an organisation's test plan template may be 25 pages sure fine when on a project where have multiple testers and is going to six months plus of testing work, but what about when one person needs to do some testing that is only going to take a couple of days to complete? Is this 25 page template useful? If you don't understand it you will have to just fill it all out and it is going to take longer than the testing is going to take. If you understand this, one would hope that you can only fill out the sections that make sense and quickly explain why you haven't filled in some of the sections seeing they are not applicable to what you need to do.

One example I have seen where someone who understands something forgets to teach all the basics along the way. I was giving some business users some help with some basic SQL queries. They were saying SQL was hard and they just didn't understand what they were doing. They said that the other person had spent a day with them teaching them SQL they knew about selects, update, inserts, deletes, wheres and joins. So I was a little perplexed that they were having issues with a SELECT * FROM foo WHERE bar = foobar. But after some talking with them found out they didn't actually really know what a database really was and what tables were. After taking a step back and saying a table is like a spreadsheet with columns and rows and a where was like a filter. That a database was a collection of tables and a basic of joins using a person spreadsheet and a address spreadsheet, which could be of type street or postal they were away running.

Coming back to the rigour aspect I go back to my time at university. Seeing I did honours there I spent a lot involved in academic rigour. One of the big things with that type of rigour was that someone with a similar background should be able to follow your method and reach the same results. They mightn't draw the same conclusions from the results but the results themselves should be the same. So this aspect of rigour is applicable to what we do in the software testing world. Our testing of systems should be repeatable by people other than ourself and repeatability comes into great importance with regression testing and test automation especially when it is being run every night or on every check in. Even when it is automated the person receiving the result needs to be able understand the process even do it manually, so that they can fully understand the results and draw accurate conclusions from it all.