Sunday, 5 August 2007

Continuing on from TAP towards a database

Well TAP (Test Anything Protocol) does log well and is easy to parse into some type of report. That is all good but what happens when you want to record results across multiple test runs, across threaded harnesses, across a cluster of boxes or multiple test harnesses. Yes you can store all the tap log files in a directory but there are problems with searching and sorting finding out when something was last run. To start managing it you are going to have to do some major grepping and sorting, so you might as well put it in a database so can get at the results with SQL.

TAP is generic and can easily any test result to it as it is generic but covers off all the data you really need. So when designing the database need to make it generic like TAP so you can use it with any testing application, but there is also need to make it extensible so can tailor to what/how you are testing. The extending should be done so you can all the pre-existing reports etc. without any breakage and also have one central database that can be storing all the results for multiple applications and/or harnesses without them stepping on each others toes.

I am thinking the best way to do the extensibility is via setting up child tables as means the queries/reports running on the default tables will continue to work. Having child tables is better then just adding more columns to the database because you won't get lots and lots of null columns depending on what test harness is being run.

Seeing I am currently writing a test harness at work at the moment I am thinking about storing the results and making reports, so this is really going through my mind at the moment. I am thinking I might look to see if there is some type of standard database that can use that already has some reporting stuff written. Just thinking about Mozilla's Litmus will have its database somewhat covers this so might give that a look.