Sunday, 18 July 2010

Selenium Grid Part 3 - Integrating Selenium Python tests with Hudson and having them run in parallel

In part 3 of the my Selenium Grid series I will be looking at using Python as the scripting language to write the tests. I am borrowing the ideas from Mozilla QA's testing of their sites. Their SVN server has some Python base Selenium scripts (SUMO SVN and AMO SVN). I saw an interesting video of them talk about Mozilla talking about their use of Selenium at a Conference.

Firstly down Selenium RC and from the zip file copy /selenium-python-client-driver-1.0.1/selenium.py and place that in your test directory. This is the library file for allowing Python to talk to the Selenium Grid Hub. Another file that you will need is XML Test Runner.  This XML Test Runner is an extension to Python's built in Unit Testing that will output jUnit style XML reports that Hudson can read and display the results for.

The test suite I am setting out below will run the tests in parallel and has provisions to do more in the future. It can be set up in the future to run different sets of tests, run all the tests in different browsers and has some early examples of external configuration and common functions.

So in my Selenium job in Hudson I changed the execute shell to be: ./Swrunner.py and said that the published jUnit test results are *.xml. I also had to chmod the working directory to give the Hudson user permission to write out the jUnit results out to this directory. One other change I made was to stop using the Selenium Grid Hudson plugin as that seem to have issues with running more tests than there were available nodes, so I fixed this by like in Part 1 of just running the Hub from the command line and that seems to work well.

You can either see the scripts below to download them in a zip file.

selenium.py - from Selenium RC zip file

SWconfig.py:
'''
Created on 17 July 2010
@author: Dave

These are connection details

Idea from: http://viewvc.svn.mozilla.org/vc/projects/sumo/tests/frontend/python_tests/vars.py?view=markup
'''

class connectionParameters:
    server = '172.24.128.200'
    port = 4444
    baseurl = 'http://shihadwiki.com'
    page_load_timeout = 600000
    browser = '*firefox'

SWfunctions.py:
'''
Created on 17 July 2010
@author: Dave

These are some generic function examples for running ShihadWiki.com
'''

     
def openSite(sel, page_load_timeout):
    sel.open("/wiki/Main_Page")
    sel.wait_for_page_to_load(page_load_timeout)
    
def clickMenu(sel, link, page_load_timeout):
    sel.click('//li[@id=\'n-%s\']/a' % (link))
    sel.wait_for_page_to_load(page_load_timeout)

SWrunner.py:
#!/usr/bin/python
'''
Created on 17 July 2010
@author: Dave

This is the test runner to run all the tests

Idea from: http://viewvc.svn.mozilla.org/vc/projects/sumo/tests/frontend/python_tests/suite_sumo.py?view=markup
Idea from: http://saucelabs.com/blog/index.php/2009/09/running-your-selenium-tests-in-parallel-python/
'''

import SWfunctions
from SWconfig import connectionParameters
from SWtestList import testList
import sys
from subprocess import Popen


args = sys.argv

type = None

if len(args) < 2:
    type = 'smoke'
else:
    type = args[1]

tests = []    

if type == 'smoke':
    tests = testList.smoke_list
elif type == 'full':
    tests = testList.full_list

browsers = []
browsers.append('*firefox')

processes = []

for browser in browsers:
    vars = connectionParameters
    for test in tests:
        processes.append(Popen('python %s %s %s %s %s %s' % (test, vars.server, vars.port, browser, vars.baseurl, vars.page_load_timeout), shell = True))
        
for process in processes:
    process.wait()

SWtestList.py:
'''
Created on 17 July 2010
@author: Dave

These are the tests to execute

Idea from: http://viewvc.svn.mozilla.org/vc/projects/sumo/tests/frontend/python_tests/tests.py?view=markup
'''

class testList():

    smoke_list = []
    #smoke_list.append(file-name.py)
    smoke_list.append('test-gigs.py')
    smoke_list.append('test-songs.py')
    smoke_list.append('test-discography.py')   
    
    full_list = []
    full_list.extend(smoke_list)

test-discography.py:
from selenium import selenium
import unittest, time, re
import sys
import SWfunctions
from  xmlrunner import *

host = None
port = None
broswer = None
baseurl = None
page_load_timeout = None

class test_discography(unittest.TestCase):
    
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium(host, port, broswer, baseurl)        
        self.selenium.start()
        self.selenium.set_timeout(page_load_timeout)
        
    
    def test_discography(self):
        sel = self.selenium
        SWfunctions.openSite(sel, page_load_timeout)
        SWfunctions.clickMenu(sel, 'Discography', page_load_timeout)
        self.assertEqual("Discography - Shihad Wiki", sel.get_title())
        try: self.failUnless(sel.is_text_present("Discogarphy"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == '__main__':
    args = sys.argv
    host = args[1]
    port  = args[2]
    broswer = args[3]
    baseurl = args[4]
    page_load_timeout = args[5]
    suite = unittest.TestSuite()
    suite.addTest(test_discography('test_discography'))
    #suite.addTest(test_songs.test_songs)
    #unittest.TextTestRunner().run(suite)
    runner = XMLTestRunner(file('results_test_discography_%s.xml' % (broswer), "w"))
    runner.run(suite)

test-gigs.py
from selenium import selenium
import unittest, time, re
import sys
import SWfunctions
from  xmlrunner import *

host = None
port = None
broswer = None
baseurl = None
page_load_timeout = None

class test_gigs(unittest.TestCase):
    
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium(host, port, broswer, baseurl)        
        self.selenium.start()
        self.selenium.set_timeout(page_load_timeout)
        
    
    def test_gigs(self):
        sel = self.selenium
        SWfunctions.openSite(sel, page_load_timeout)
        SWfunctions.clickMenu(sel, 'Gigs', page_load_timeout)
        self.assertEqual("Gigs - Shihad Wiki", sel.get_title())
        try: self.failUnless(sel.is_text_present("Gigs"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == '__main__':
    args = sys.argv
    host = args[1]
    port  = args[2]
    broswer = args[3]
    baseurl = args[4]
    page_load_timeout = args[5]
    suite = unittest.TestSuite()
    suite.addTest(test_gigs('test_gigs'))
    #unittest.TextTestRunner().run(suite)
    #runner = xmlrunner.XmlTestRunner(sys.stdout)
    runner = XMLTestRunner(file('results_test_gigs_%s.xml' % (broswer), "w"))
    runner.run(suite)

test-songs.py
from selenium import selenium
import unittest, time, re
import sys
import SWfunctions
from  xmlrunner import *

host = None
port = None
broswer = None
baseurl = None
page_load_timeout = None

class test_songs(unittest.TestCase):
    
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium(host, port, broswer, baseurl)        
        self.selenium.start()
        self.selenium.set_timeout(page_load_timeout)
        
    
    def test_songs(self):
        sel = self.selenium
        SWfunctions.openSite(sel, page_load_timeout)
        SWfunctions.clickMenu(sel, 'Songs', page_load_timeout)
        self.assertEqual("Category:Songs - Shihad Wiki", sel.get_title())
        try: self.failUnless(sel.is_text_present("Songs"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == '__main__':
    args = sys.argv
    host = args[1]
    port  = args[2]
    broswer = args[3]
    baseurl = args[4]
    page_load_timeout = args[5]
    suite = unittest.TestSuite()
    suite.addTest(test_songs('test_songs'))
    #suite.addTest(test_songs.test_songs)
    #unittest.TextTestRunner().run(suite)
    runner = XMLTestRunner(file('results_test_songs_%s.xml' % (broswer), "w"))
    runner.run(suite)

xmlrunner.py - from http://www.rittau.org/python/


The Series:

Upcoming ideas for the series:
  • Putting the test suite in SVN and running from there

Sunday, 11 July 2010

Selenium Grid Part 2 - Getting Hudson to run the tests

In Part 1 of this series I just did a basic Selenium Grid setup. In this part I will look at getting Hudson set up and have that running your tests on command or triggered when some one checks in some code. I will just get it set up to run the tests on command from a local directory as at home I don't have a source control system or code that can be checked in and then tested.

For this I assume you already familiar with the concepts and setup in Part 1.
On the Hub server shut down Selenium Grid Hub and following the Ubuntu/Debian instructions on the Hudson site:

You can now got to http://[hub ip]:8080/ and Hudson will be up and running.

In Hudson:
  • Manage Hudson -> Manage Plugins -> Advance
  • Click "Check Now" in bottom right corner
  • From the Updates page install any plugins that need to be updated
  • From the Available Plugins in page select Selenium Plugin which has the description of "This plugin turns your Hudson cluster into a Selenium Grid cluster"
  • Install it

Hudson is now responsible for running the Selenium Grid Hub and you can see it at http://[hub ip]:4444/console you can also see a new item on the Hudson menu for Selenium Grid and this will show you that your Remote Control nodes have hooked up.

So now we will make a job that runs the example build in parallel so we can see it run on the nodes.
  • I set up a Free Style Project
  • I gave it a custom workspace of /home/selenium/selenium-grid-1.0.8 this is where I unzipped things in Part 1
  • I'm just executing shell command of ant run-demo-in-parallel
  • Save that and from the dash board click build and watch the screen on your nodes
  • You will see the browsers pop up on the nodes and do stuff. Like Part 1 the tests will fail


The Series:

Upcoming ideas for the series:
  • Putting the test suite in SVN and running from there

References:

Selenium Grid Part 1 – Getting Started

I have been looking at Selenium Grid which is a tool that allows distributed multi platform and multi browser testing. I have found it is a bit like The Grinder and the How To guides seem to miss something, so I going to write about my initial set up and writing in hope it will help others get up and going with what looks like a really useful tool.

My initial setup will involve three Ubuntu VMs. One Ubuntu Server running the hub and two Ubuntu Desktops acting as the nodes. They were set up using the VMWare Player creation tool that does all the work for and installs VMWare Tools and hides most of the install prompts for to help speed the roll out. Then I changed the APT details to use my internal APT Mirror and updated, I also changed the time zone to be New Zealand and installed an SSH server. Everything else I have done the VMs will be list below as it will count as getting Selenium Grid up and running.

Install some things that are needed on all of the machines (sun-java6-jdk is the Canonical Partner repository):
  • sudo apt-get install unzip sun-java6-jdk ant ant-optional

I rebooted the systems after this so all the updates could apply and the PATHS all get updated.

On all of the machines download Selenium Grid (http://selenium-grid.seleniumhq.org/download.html) and extract it

The ant sanity-check will check that everything is installed correctly

On the hub machine start the hub
  • ant launch-hub

Give it a little bit to start up and then go to the following URL and you can check that it is all up and running by looking at http://[hub ip]:4444/console

On each of the nodes run the following command to start the remote controls, you will have to do this from a terminal on the desktop, so it knows where the display is so can run your browser.
  • ant -Dport=5555 -Dhost=[node ip] -DhubURL=http://[hub ip]:4444/ launch-remote-control

Have a look at http://[hub ip]:4444/console and you should see your two nodes are connected.

On your desktop from the selenium-grid-1.0.8 directory run the following (you will need the JDK and Ant on this machine as well)

  • ant run-demo-in-parallel

On both of the desktops you will see Firefox open up a do stuff. It does fail as I believe Google has changed their site since the example tests were written. But you can see everything opening up and running the tests in parallel over the two desktops.


The Series:

Upcoming ideas for the series:
  • Putting the test suite in SVN and running from there

References

Thursday, 8 July 2010

Selenium Connecting to New Windows

I have been evaluating Selenium as of late. One thing I was having issues with was connecting to new windows opened with target="_blank" link using the selectWindow function. It wasn't working and a looked high and low for information on why I could not connect to the newly opened window. I couldn't find anything helpful. Even got no replies to my question on the email list. I was finally poking around and found out that selectWindow just won't work in Selenium IDE and that you need to use Selenium RC or Selenium Grid. So hopefully this blog post will help people who have the same issue as I had figure out how to get this working. I guess I'm not the only one who has done their first investigation and evaluation of how Selenium works using the Selenium IDE before moving on RC and Grid. Also while I was using RC and the Grid I was using Python files rather than the default HTML that IDE uses. Below you can see the files of my example case it will also close the window and reconnect with the original launching window.

Python Test Script:
from selenium import selenium
import unittest, time, re

class test_link_to_blank(unittest.TestCase):
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium("localhost", 4444, "*chrome", "file:///D:/code/selenium/sandpit/new-window/base-page-blank.html")
        self.selenium.start()
    
    def test_test_link_to_blank(self):
        sel = self.selenium
        sel.open("file:///D:/code/selenium/sandpit/new-window/base-page-blank.html")
        self.assertEqual("Base Page Blank", sel.get_title())
        try: self.failUnless(sel.is_text_present("Base Page Blank"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.click("link=Link")
        sel.select_window("title=Pop Up Window")
        self.assertEqual("Pop Up Window", sel.get_title())
        try: self.failUnless(sel.is_text_present("Pop Up Window"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.close()
        sel.select_window("null")
        self.assertEqual("Base Page Blank", sel.get_title())
        try: self.failUnless(sel.is_text_present("Base Page Blank"))
        except AssertionError, e: self.verificationErrors.append(str(e))
    
    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

base-page-blank.html:
<html>
<head>
<title>Base Page Blank</title>
</head>
<body>
<p>Base Page Blank</p>
<p><a href="popup.html" target="_blank">Link</a></p>
</body>
</html>

popup.html:
<html>
<head>
<title>Pop Up Window</title>
</head>
<body>
<p>Pop Up Window</p>
</body>

Saturday, 3 July 2010

Videoing Testing

A little while back I went on James Bach's Rapid Software Testing Course run via SoftEd here in Wellington. One thing he mentioned and I took away is recording what testing you do. He said he went as far as recording everything with video camera. This is helpful as it records what you are doing and can see the chain of events leading up to a problem. Hard disks are getting cheap these days and can store many hours worth of video. I haven't gone as far as recording everything I do yet. What I have started to do instead of spending time copying and pasting screen shots (though Lightscreen helps a load with taking screen shots) is just record the whole reproduction case. By the time you have taken individual screen shots and written the steps that you have performed and tried to explain what the defective behaviour looks like I find that it is quicker just to record a video and attached that to the defect than taking screen shots and writing it up. I have also found the "I can't reproduce it" comments decrease as well. On Ubuntu I use gtk-recordMyDesktop and on Windows I use CamStudio (the old Open Source on Soureforge). Give it a go I have found it extremely worthwhile seeing the screen recording tools are free.

Sunday, 27 June 2010

CITCON NZ 2010

This weekend I spent time at CITCON NZ 2010 (Its pronounced kitcon rather than citcon). It is an Open Spaces Conference on Continuous Integration. I really enjoyed it and depending on the location will look at going to it again next year. I now have a lot of ideas and links to research. I will look at writing up some more blog posts at what once I have digested and thought about this more.

New Blog

A while back I set up a new blog Dave's Two Cents it where I write all sorts of things that don't fit this blog's main focus which is software testing.