Sunday, 26 April 2009

"What is testing?" The question that we all get asked

One question which I get asked and I'm sure a lot of others as well is "What is testing?" or some derivation of that. Well most of the time I will just say something like "I'm paid to break stuff" and "preferably before the users do". This is somewhat of the truth and without spending too long explaining it all (as most people don't care that much) is accurate enough for them. And then I actually enjoying find it fun finding new and interesting ways to break the new piece of code which I have been given to test. Though I prefer it when it is a challenge to break it, it is not all that fun to just have it break when you look at it oddly and have it die terribly. Though testing of course is more than just this with checking meets requirements, fit to deploy to production etc.. Then again if I don't break or find some cool or weird behaviour, I feel a little disappointed and like I am not doing my job properly.

While thinking about this I think back to an old blog post of mine where was looking at the difference between QA and Testing. QA is really what the project auditors do by looking at processes, scope etc. as QA can look at if you are following a set process to ensure quality of the process which results in the delivery of the final product. I think is this more like what would do in say the food industry where they ensure the processes are correct and only sample a small percentage of the output while in Software Testing we try and cover as many combinations as possible to get as close to 100% coverage as can given the project constraints.

Saturday, 25 April 2009

soapUI and a few other tools

I have a bunch of tools which I need to have a good look at but just need to find the time to do so. They normally just sit in open tabs in Firefox and I live in hope that Firefox doesn't lose them for any reason.

Though this week I did get some time to play with soapUI. I have used soapUI before but that was a while ago and someone else had already set up the SOAP queries and was an old version and I just left it at that because had a tight deadline to meet and just had to get the testing done. Which meant back then I didn't have all that much time to learn the tool. Though this week I had to give a demo of soapUI to some people, so I had do some playing with it and learn about it quickly.

Well it was a lot more powerful than I thought it was. Up to now I knew it could make SOAP requests based on a WSDL and then have those requests editable to send to your servers. There is the one gotcha that need both the WSDL and the XSD in the same directory if pulling it from a directory on your computer . Well that feature is all good and comes in handy as is an easy way to send requests in and get a response back from a web service.

So this week I found out it will also do proper testing rather than a tool just to do do ad hoc SOAP requests. You can set up test cases that will do assertions and what not so it will tell if you get the correct response back or not. If the build in assertions are not enough there is Groovy scripting, so you can do what ever validation you want. It can then run the test suite and make you a nice little report at the end of it. It can also go the next step and does load test using your test suites. I haven't really dug that much into load and performance testing as recently the projects I have been on haven't needed it all that much in the areas I have been testing. It also looks like you can script this this all up and rather than having to use a GUI to trigger it the test runs off. This would be quite nice as then you can put it on your automated build and validation server and have run all the time and alert the developers as soon as a regression occurs or use it as test driven development, where all the tests are created ahead of time and they keep developing until all the tests past.

The other neat feature I found out that soapUI has is that it can create you mock web services. This will come in handy where you have you an application which is a web service client and the application which is providing the web service isn't available yet and you need to start testing. So long as the WSDL is ready you can quickly make a mock web service which will respond to your requests with either hard coded or if you want can use Groovy to make your responses a little more dynamic.

If you already have the client and server side built and either want to build a test suite, migrate a suite of tests from a another tool or want to get some response timings there is a proxy tool. What you can set this up to do is sit between the client and the server and it will record everything that is going backwards and forwards and can covert this into a test case or a mock web service.

Best of thing of all is that this it is an open source tool. There is a pro version which you will need pay for. It adds things like coverage, access to data sources and sinks (which you can use other tools to populate or validate) and a better global store for your Groovy scripts amongst some other things.

The other tools which I would like to have a better look and haven't had a good chance to learn about yet are:
  • Mantis - This is a defect tracking tool. I have had a look at this and am really liking it. What I need to do it is get a proper work flow set up in it and some dummy projects which match some of our real projects, so people can get a real feel for it and try and get it in for at least the up to System Testing of projects while they are still internal and haven't gone to the client yet, where on the whole need to use their own corporate defect tracking system.
  • Sonar - This a tool for static analysis for the code. I haven't done much more looking at than what is on the website for it, but from that it looks quite interesting.
  • Testlink - A Test Case management tool. Like sonar haven't done much more looking at than what is included on it web site.
  • Session Tester - A tool for writing down your session notes. This is one to watch to see how it develops over time (it is only 0.2). Though it doesn't 100% work how I like to work as I like my notes to be much more of a time line with everything intertwined on timestamps, so when come across and issue or bug you can easily see where it fits in with the actions that you did and have timestamps to aid looking in the logs.
  • Spawner – This a tool which will generate data to populate a database do you when you need a bunch of data in the tables. I have had a look at it previously one this blog. IT will do all your different data types and what not. It is far better than the way I previously did it by writing a Python to make me a bunch of insert statements.

Sunday, 19 April 2009

Visual Representation of Defect Queues

One of our devs has had the idea of placing our defect queue on the wall using a grid and catalog cards. Y axis is severity and priority and X is status. It is simple and is working well to see where our defects are currently sitting and where any bottle necks ares. The downside to it is that it does require effort to keep the wall and our client's defect tracking system in sync.

Over some drinks on Friday night we were discussing how we like it and how it was helping. Though of course the bit of keeping the wall and the defect system in sync. Well we started thinking about Microsoft Surface. Well Surface does have the tactile of moving defects around when changing the status. So it would be really cool if some would either make a defect tracking tool or an addon to one to linked into Surface. Then you could get a big screen running surface and mount on the wall and manage the defect queue that, changing statuses, filtering. You also don't need defect reports in the same way to your PM as they just walk past and visually see the defects progressing and where bottle necks are.

I am also thinking that for the next project while in the system test stage before getting the client involved to work solely off the the wall for those defects without using a defect tracking tool. Sometimes the low tech solution works really well and because we are in IT we always go for the techy solution rather and the simple one that may work better.