Monday, 30 July 2007

TAP - Test Anything Protocol

TAP is a really cool idea. A log format for the logging of test results coming out of some automated process. It has some great benefits:
  • Really easy for a human to read with tail -f. Hate trying to read XML or some format that is designed for computers to parse easily
  • It is also really easy to to write a parser for it. Just some easy string splitting and you have the data out ready to write to an HTML page or something.
  • Also with the parsing don't need to wait for everything to finish as with something like XML it is only well formed once the whole thing is complete. As some tests runs may take hours or even days. If you are running for that long you really want some type of results along the way so you know that everything is going alright.
I have been playing with a writing a testing framework at work. I am using TAP for the logging the results. TAP is logging the results ok; but I am hitting some limitations in it.
  • Well my testing app is threaded so keeping the TAP log to spec is not easy. So I am currently logging each thread to a separate log file. In this app it isn't actually to bad as each thread is testing a separate input file so having the results is alright as it make some sense. And when making the report it is easy enough to just iterate through all the log files in a directory to make a nice little report.
  • Another thing is it is quite generic and missing things that I would like:
    • Run time. It would be nice if there was a way to mark run time in the file somehow so it is always in the same place or can regex find it in a common way.
    • Also need a way to link it back to a master test plan. How to link a number in TAP to a test in a test script or something. Thinking making a reporting matching the test execution to the test scripts in a requirements in a test matrix style type thing.
    • Also how do you get logging of results across test runs. As some tests might get run in one test execution and not tested in another one. Need some way of bringing all this together.
    • Also searching of the results and generating reports is a bit hard.
So TAP good at logging there and then. But thinking about long term storage, resulting and traceability matrix type stuff need something more substantial and extensible so can add more data than just the generic stuff. All this has to to be done so the extensibility doesn't break the the generic so can use a basic report generator even when have custom data. So that leave me thinking inheritance and databases suspect that is another blog post. Need to do some more thinking first.