Sunday, 24 May 2009

Testing of Requirements... Is always a good thing™

I was having a read of The Impact of Requirements issues on Testing which is on the SoftEd Resources page. It says that 40% of software errors come from requirements errors. It would tend to agree with this as on the whole as it there is possibility for ambiguity and ambiguity leads to the business thinking it should do X, the developer codes Y and the tester tests it assuming it will do Z. Which leads to the standard inclusion of:

I'm sure everyone has seen this picture before, it is the standard diagram indicating misunderstandings in requirements.

What I see the problem is that testers only get involved when it is time to the output from development. This is always the case and I doubt it will really change until the client can understand spending time and money on testers to be engaged from day zero of the project and test every output along the way before going onto the next stage in the project life cycle. For example the project requirements are normally only signed off by the client. This isn't always the best as the the client doesn't always understand the technology fully and they described it to the BAs so they are can fill in the gaps and ambiguity without thinking about it. Having testers involved at this stage would bring in a viewpoint from people who have been involved in requirements gathering and also bring the testing viewpoint to things and looking for holes and defects.

Why I don't think this will fly without a lot of re-education?:
  • Testing will slow it down
  • This will require people so therefore require more money to be spent

But I personally think these are false realities:
  • It may lengthen the time for requirements gathering, but the overall project will be less. As picking up these errors before development will mean that some pieces will not have to be redesigned. redeveloped and retested. This is the whole the sooner in the cycle you detect a defect the cheaper it is to fix. Though usually it is only looked at with Unit Test vs. System Test vs. User Acceptance Test vs. Production.
  • The second point of needed more people and therefore money is linked into what I just said above. You will just be spreading the testing effort out over the project and will hopefully make the time spent testing the requirements back by spending less time retesting.

What is needed? Re-education in that life cycles should have test team involvement the whole way through the project not just post development. This will need to be taught in project management courses and made part of the methodologies used. Some of the Agiles are better at this as the test team is at the kick off meetings etc. and everyone is co-located so ambiguity can be sorted out quickly without needing to schedule meetings etc. Though seeing Agile doesn't scale to larger projects it would be good if some how some of these aspects could be worked into methodologies for the larger projects to use.

Sunday, 17 May 2009

Can defect tracking systems be more helpful?

I wrote about Visual Representation of Defect Queues a little while ago and the whole idea has been sitting in the back of head churning away.

Most defecting tracking systems are good at showing lists of defects. They also maybe able to show you scoreboards/traffic lights or graphs. Lists are good for a person who has a list of defects in ranked ordered assigned to them to fix or retest. The graphs are good for management as they can see the counts and if they are increasing or decreasing. There is more information than that stored in most defect tracking systems which could come in handy. There is a lot of information store about times things spent in X statuses. The above two examples is either a snapshot in time or a series of snapshots with little information shared between the snapshots. So is there are way to use more of this information stored inside of these defect tracking systems?

Really so far we have got something which is good for the management and the client and something good for the developer defect fixing or tester retesting, nut what about the team lead? Is there a middle ground in between using some of this extra information? How is their team's progress? Can they get a picture of there team in more detail than the managers but maybe a bit less than the developers personal queue?

As mentioned before, in my Visualising Defect Queues post, there is the swim lane type approach. What it can give you is a 2D view of everyone's personal list. Along the X you display the current status of the defect and the Y you can show the severity/priority of the defects. Using index cards stuck to the wall you can some more information as a number on a scoreboard means something, but people are more visual than that, so adding the cards can give you more information. For instance one thing that you can more easily see is bottle necks as it is there in front of you and over the day/week you can just see the cards moving along and you can see bottle necks as large groups of defect start to bunch together. That is extra information which you weren't getting before by just display the information differently. This isn't really using the extra information which is stored in a defect tracking system's database.

If we take the swim lane idea, now we can give each person a colour for the defects assigned to them. Now could get a view for bottle necks as a whole and also the individual people in your team. Is one person sinking? Is it the whole team generally? Could the defects be better distributed amongst the team members? Cool this is now getting more useful and we are using more information in the database to display useful information. What else can we can we display?

Well maybe adjust the transparency of the colour for how long it has been sitting in that status, so now the bottle necks could be more useful again one person may just have the one but they have it for quite a while, so these more opaque cards might need some looking into. There maybe some else has a whole lot of defects sitting with them, but they are the quicker turn around defects so all have the day 1 transparency on them. With just a scoreboard things would look fine as number are moving well and the graph trends look fine but it is masking information about things which could improve your team's performance.

Maybe failed retest could have a border on them to identify them which would mean they might be worth an extra code review as something failing retest more than once never looks to good.

Is this display more information? Yes
Will this lead to information overload? No
Why? Well as if display this visually humans are quite good at picking up patterns from it.
How could this help? Well hopefully team leads can better manage their team and start to see patterns and identify issues with the project sooner than when management ask about a something in particular or the client asks about their pet defect which they raised ages ago isn't fixed.

So what needs doing? Well a defect tracking project needs to take this idea and run with it. Could implement it in flash, so could just drag the little cards around the swim lanes. Then for the people using an old box hooked up to a projector could have it continuously up on the wall, so the defect situation is always there on the wall, just like the story wall in agile. Or maybe MS would like to get in on the act and implement it in MS Surface.

If you were doing one of the Agiles with their swim lanes you could use the same framework. And seeing in Agile things are time boxed, you could have the stories slowly moving across the screen in regards to their complexity and expected time to do. Though maybe this would cause people to rush if they saw things moving faster than they could actually do the work as they found they encountered some type of snag. Which may decrease the quality of the product coming out.

Spawner revisited

Just an update on Spawner well Seth the developer of Spawner has already placed the idea of generic text mask into SVN. So looking forward to the build which includes that feature. It is really cool when you make a suggestion to an open source product and you see you idea get implement.

Sunday, 10 May 2009

Firewall port testing #3

Here is an update to my Firewall port testing script. You can find more about it in my first two epsiodes about it (one and two). I have made some changes to it nothing big just some comments and should now get the host name on Linux systems for placing in the results file name. This code is released under a do what ever you want with it license.


"""PortTester will test to see if the machine that this script is running
on has access to another server on a given port. The servers to test are
outlined in input.txt and the results are placed in the
results_<servername>_<YYYYMMDD>-<HH24MMSS>.txt file. This output file is

The input file should look like:
Router Web Admin,,80
Won't work,,12345

import telnetlib
import thread
import os
import datetime
import platform

class PortTester(object):

thread_count = 0

def __init__(self, input, log):

for row in input:
thread.start_new(self.testPort, (row['info'], row['host'], row['port'], log))
self.thread_count = self.thread_count + 1
while self.thread_count > 0:

def testPort(self, info, host, port, log):
print 'Testing %s. host %s on port %s' % (info, host, port)
connection = telnetlib.Telnet(host, port, 5)
log.write('%s,%s,%s,pass\n' % (info,host,port))
log.write('%s,%s,%s,fail\n' % (info,host,port))
self.thread_count = self.thread_count - 1

def main():
test_list = readCSV('input.txt', ',', 1)
hostname = ''
if == 'posix':
hostname = platform.node()
hostname = os.getenv('COMPUTERNAME')
output = open('results_%s_%s.txt'%(hostname,'%Y%m%d-%H%M%S')), 'w')
PortTester(test_list, output)

def readCSV(path, delimiter, header_row):
text = open(path, 'r').read()
text = text.replace('%s%s' % (delimiter, delimiter), '%s %s' % (delimiter, delimiter))
lines = text.split('\n')
rows = []
if header_row:
headers = []
if lines[0] != '':
headers = lines[0].split(delimiter)
tmp = []
for head in headers:
headers = tmp
del lines[0]
for line in lines:
if line != '':
values = line.split(delimiter)
row = {}
for i in range(0, len(headers)):
row[headers[i]] = values[i].strip()
for line in lines:
if line != '':
values = line.split(delimiter)
return rows

if __name__ == '__main__':

Tablets and Handwriting Recognition in Linux

As I have mentioned before I have an eeePC 901 to write my notes on. One thing that is missing is ability to draw on it and seeing there currently aren't any netbook tablets which run Linux (preferably Ubuntu) yet here in New Zealand, I had to do the next best thing and brought myself a drawing tablet.

It is taking some getting use to as you are drawing on a tablet and it is appearing on the screen. I'm sure this will just take some time to get use to. It is fine for drawing things in Draw. With this means I can draw diagrams as I am going along. Drawing freehand on the computer is quite as can print it, email it etc. a bit like a printable whiteboards (the client site I am currently has printable whiteboards in every meeting and breakout room). This means I can have that one directory which has all my notes and diagrams and don't have to go thumbing through books to find it. It will take some getting us to until my drawing on the screen is as good as a whiteboard or paper but in the mean time it is really cool. I am using Draw because it comes standard in the Ubuntu 9.04 build though there are some Journal/Notebook apps out there that I could try e.g. Xournal, Jarna, Gournal and NoteLab. These all look interesting but having found the need to use them yet as gedit and Draw do what I need currently and also from having a quick look at them none of them have handwriting recognition.

I have also installed CellWriter this provide handwriting recognition and after less than five minutes of training it was fairly accurately picking up charterers. Currently the only characters I am having a lot of issues with are t and +. The only other issue I had in getting CellWriter to work has I had to disable “extended input events”. It is fast and can write at a normalish speed on it. Though it does some odd behaviour when you reach the end of the input line and it sometimes moves the whole word onto the new line and sometime doesn't and also when you want a space at the end of the input line. I need to sit with a bit more and figure out what I am seeing and then a what I expect it to be doing.

Saturday, 9 May 2009

How I track my session notes

Looking around the web most of the session notes look something like what is outlined here. Mine follow pretty much along the same lines. My difference is that I don't separate likes like issues out I just put them in line in my notes section which is more like a time line than notes per so. I mark the issues with '@issue' so I can grep through them later to find the issue bits.

Some of this difference may come down to that I haven't tested a GUI in nearly two years. This may mean that my style has adapted to work more like that as part of digging into an issue my result in looking multiple databases and a handful of log files. Something when you notice an issue it may be the chain leading up to it so it comes in handy to have the issue in line with the notes and having a time stamp on it.

My template current looks like this:






Keeping tracks of testing notes and my eeePC

With post like Exploratory Testing: Recording and Reporting by Michael Bolton in which he links to an article and some slides of his about using notebooks to store your thoughts and notes in, and in particular the Moleskine. I have tried keeping notes and such on paper in a book, my biggest problem is that my hand writing is shocking and a lot of time even I have issues deciphering it later on. I have started to use to my eeePC 901 to start taking notes on.

I am running Ubuntu 9.04 on it. I normally just use gedit  to do my note taking in. gedit is just nice and simple with nothing that gets in the way of just writing what you are doing. There are downsides to using a netbook for taking notes rather than paper:
  • Battery
  • Boot Time
  • Sketches and Diagrams
  • Saving
  • Small keyboard

Well I am getting over five hours of battery on it with the WiFi off which is fairly handy to do what you need to do with going to a plug. Well with Ubuntu 9.04 I am getting about 35 seconds to the login screen and 10 seconds from the login screen to the desktop. Well I do still have a a book next to me so I can still draw while I am going along and just make a note that I have a diagram in my book. Well have autosave turned on to save every minute so the chances of losing something is fairly low. I do admit the small keyboard is a bit of pain especially seeing I am use to using a MS Natural Ergonomic Desktop 7000 but I am just writing notes rather than long documents so it is alright. If am using it at a desk it is easy enough to plug in a second keyboard into it.

Well there are some advantages of using a netbook over paper:
  • Easy to put in timestamps
  • Only bad spelling and not having to worry about my handwriting

What is really nice about using a text editor is that it is really easy to put in timestamps into your testing notes. This comes in really handy when you need to come back and look at logs so can tie things together easyly. Text is nice and easy to read and can do spell checking though that can be a bit of a pain seeing I use a lot of shorthand, acronyms and technical words which aren't in dictionaries.

One thing they both lack is not copy and paste and no insert of screen shots directly into your notes.  I have always had a text file on the computer and also directory for screenshots and logs.

So why don't I just use the computer I am and write notes there? Will I like the idea of the computer sitting to my right is my notebook and is for taking notes while the computer directly in front of me is working on. Other things like remote desktop can capture alt-tab which makes switching between servers and local text files a bit of a pain.

Sunday, 3 May 2009

Populating Databases with random data using Spawner

Spawner is an open source tool for generating data to populate databases. There are Linux and Windows binaries for it.

First thing out of box is the Linux version won't run on my Ubuntu 9.04 install with the following error message “./spawner: error while loading shared libraries: cannot open shared object file: No such file or directory”. Seeing I have Wine installed, I just thought I would get the Windows version and try that. Well under Wine it loaded up fine, so won't spend any more time trying to get it to work under Linux. Though if anyone knows what I need to do to get it to work please leave a comment.

Well Spawner opens up with a fairly simple UI.

The first tab is the main one in which you describe the table that you want to populate. It can randomly make numbers, strings and dates, but it can also do some smarts with these like sequences, ranges, number of word, number of characters, humans like names, emails, addresses and SSNs (shame this isn't tweakable to handle IR number for here in NZ XXX-XXX-XXX or maybe bank account numbers XX-XXXX-XXXXXXXX-XXX). It will also make IPv4 addresses but not IPv6 addresses.

Once you have set up your output options. It will make you a CSV file, a SQL file with inserts or directly place the information into a MySQL database. Just select how many rows you would like and press the go button and voilĂ  there you go a file with all your data in it ready to place into your DB.

Some features that I would like to see in it are what I have mentioned above with more data types and/or maybe something where you could define your own format mask, which would come in handy for country specific things like bank account and phone numbers. The other thing which could come in handy with be the ability to do parent child tables. Though thinking about you do it manually using number ranges on the ID column. It also doesn't seem to handle putting binary data into a column though I guess with this would need to point it at a directory with some images or documents in it. Another output type that would come in handy would be an XML type which could give the XSD to and could tell Spawner how to populate each field be it randomly or with a a static value. Though with XML the child table feature may come in handy so you can do nesting of data.