Asko Soukka: Cross-browser test your Plone add-on with Robot Framework, Travis-CI and Sauce Labs

Bootstrap Templer

Create buildout directory for Templer installation:

$ mkdir templer-buildout
$ cd templer-buildout/

Get bootstrap.py:

$ curl -o http://downloads.buildout.org/2/bootstrap.py

Create buildout.cfg:

[buildout]
parts=templer

[templer]
recipe=zc.recipe.egg
eggs=
    templer.core
    templer.plone

Run bootstrap and buildout to install Templer:

$ python bootstrap.py
$ bin/buildout

Create a new product with Templer

Call the buildout-installed to create a new product with Robot Framework test example:

$ templer-buildout/bin/templer plone_basic example.product

Be careful to answer true for the following question about including Robot test templates:

robot tests (should the default robot test be included) [false]: true

Run buildout:

$ cd example.product
$ python bootstrap.py --distribute
$ bin/buildout

Update Robot Framework tests to be Selenium grid ready

Using Sauce Labs with Robot Framework (Selenium library) is similar to using robot with your own selenium grid. It mainly requires making the browser opening keyword configurable with a few selected variables.

Update src/example/product/tests/robot_test.txt with:

*** Settings ***

LibrarySelenium2Librarytimeout=10implicit_wait=0.5

Suite SetupStart browser
Suite TeardownClose All Browsers

*** Variables ***

${ZOPE_HOST}  localhost
${ZOPE_PORT}  55001
${ZOPE_URL}  http://${ZOPE_HOST}:${ZOPE_PORT}

${PLONE_SITE_ID}  plone
${PLONE_URL}  ${ZOPE_URL}/${PLONE_SITE_ID}

${BROWSER}  Firefox
${REMOTE_URL}
${DESIRED_CAPABILITIES}  platform:Linux
${BUILD_NUMBER}  manual

*** Test Cases ***

Plone site
    [Tags]  start
Go to  ${PLONE_URL}
Page should containPlone site

*** Keywords ***

Start browser
    ${BUILD_INFO} =  Set variable
    ...           build:${BUILD_NUMBER},name:${SUITE_NAME} | ${TEST_NAME}
Open browser  ${PLONE_URL}  ${BROWSER}
    ...           remote_url=${REMOTE_URL}
    ...           desired_capabilities=${DESIRED_CAPABILITIES},${BUILD_INFO}

Let me explain what all those variables are about:

• ZOPE_HOST should match the host for which ZServer is started during the test setup (ZServer host is configured with ZSERVER_HOST-environment variable. It defaults to localhost, but must be configured to 0.0.0.0 for testing Internet Explorer (we do that in buildout soon).
• ZOPE_PORT should match the port number which ZServer is started to listen during the test setup (ZServer pot is configured with ZSERVER_PORT-environment variable. It defaults to 55001, which is just OK, until you wan to run multiple tests simultaneously on the same machine.
• ZOPE_URL is a convenience variable for accessing Zope application root.
• PLONE_SITE_ID is the Plone portal object id (and path name) for the test site. It default to plone, but it can be configured with PLONE_SITE_ID-environment variable. The default should be ok for most cases.
• PLONE_URL is a convenience variable for accessing the Plone site front-page.
• BROWSER selects the browser to run the tests with. The supported values depend on Selenium Python-package and can also be read from the documentation of Open browser-keyword in Selenium2Library keywordsdocumentation.
• REMOTE_URL enables testing with Selenium grid by defining the url of the Selenium hub to use.
• DESIRED_CAPABILITIES is used to pass various extra parameters for Selenium hub (e.g. the browser version to use or test metadata).
• BUILD_NUMBER is used to identify the Travis-CI build on Sauce Labs.

When robot tests for Plone are run using bin/test, all the variables above can be overridden by defining corresponding ROBOT_-prefixed environment variable (e.g. ROBOT_REMOTE_URL).

Add Travis-CI configuration with Sauce Labs -support

There are a few steps in adding Travis-CI-support into your product.

At first, create travis.cfg to do the required magic for minimizing buildout-time and setting a few required environment variables. Thanks to the great community, we can just extend a public template:

[buildout]
extends=
    https://raw.github.com/collective/buildout.plonetest/master/travis-4.x.cfg

package-name=example.product
package-extras=[test]

allow-hosts +=
    code.google.com
    robotframework.googlecode.com

[versions]
plone.app.testing=4.2.2

[environment]
ZSERVER_HOST=0.0.0.0
ROBOT_ZOPE_HOST=0.0.0.0

[test]
environment=environment

Create .travis.yml for letting Travis-CI to know how the environment should be set up and the tests run:

---
language:python
python:'2.7'
install:
-mkdir -p buildout-cache/eggs
-mkdir -p buildout-cache/downloads
-python bootstrap.py -c travis.cfg
-bin/buildout -N -t 3 -c travis.cfg
-curl -O http://saucelabs.com/downloads/Sauce-Connect-latest.zip
-unzip Sauce-Connect-latest.zip
-java -jar Sauce-Connect.jar $SAUCE_USERNAME $SAUCE_ACCESS_KEY -i $TRAVIS_JOB_ID
-f CONNECTED &
-JAVA_PID=$!
before_script:
-bash -c "while [ ! -f CONNECTED ]; do sleep 2; done"
script:bin/test
after_script:
-kill $JAVA_PID
env:
global:
-ROBOT_BUILD_NUMBER=travis-$TRAVIS_BUILD_NUMBER
-ROBOT_REMOTE_URL=http://$SAUCE_USERNAME:$SAUCE_ACCESS_KEY@ondemand.saucelabs.com:80/wd/hub
matrix:
-ROBOT_BROWSER=firefox ROBOT_DESIRED_CAPABILITIES=tunnel-identifier:$TRAVIS_JOB_ID
-ROBOT_BROWSER=chrome ROBOT_DESIRED_CAPABILITIES=tunnel-identifier:$TRAVIS_JOB_ID
-ROBOT_BROWSER=internetexplorer ROBOT_DESIRED_CAPABILITIES=tunnel-identifier:$TRAVIS_JOB_ID

Let me describe:

1. Lines 5–8: Run buildout.
2. Lines 9–15: Download and start Sauce Connect.
3. Line 16: Run tests.
4. Lines 17–18: Shutdown Sauce Connect.
5. Lines 19–22: Define required environment variables for letting robot to use Sauce Labs.
6. Lines 23–26: Define build matrix for running the tests with Sauce Labs' default Firefox, Chrome and Internet Explorer. tunnel-identifier-stuff is required for Sauce Labs to allow more than one simultaneous tunnels for the same user account.

Next, define your Sauce Labs username and access key as secret, encrypted, environment variables SAUCE_USERNAME and SAUCE_ACCESS_KEY.

Currently, Sauce Labs offers unlimited free subscription with three simultaneous connections (e.g. running tests for three different browsers at the same time) for Open Source projects. Just make sure to register the account for your project, not yourself. Public repository url is required for the creating the account and it cannot be changed afterwards.

1. Install Travis gem for Ruby (and install Ruby before that when required):

   
   $ gem install travis  # or sudo gem ...
   

2. use travis-command to insert encrypted environment variables into the product's .travis.yml:

   
   $ travis encrypt SAUCE_USERNAME=myusername -r mygithubname/example.product --add env.global
   $ travis encrypt SAUCE_ACCESS_KEY=myaccesskey -r mygithubname/example.product --add env.global
   

Make sure to use your own Sauce Labs username and access key, and your product's Github-repository path (with format username/repo).

Finally, enable Travis-CI-tests for you product either at Travis-CI.org or at GitHub.

Done. If I forgot something, I'll update this post.

Behind the basics: Test level status reporting for Sauce Labs

By default, Sauce Labs doesn't really know did the Selenium tests on it pass or fail. To pass that information from our test runner on Travis-CI to Sauce Labs, we need to add some extra code into our test setup.

At first, append the following into the end of src/example/product/testing.py:

try:
# Inject keyword for getting the selenium session id
importSelenium2Library
Selenium2Library.keywords._browsermanagement.\
_BrowserManagementKeywords.get_session_id=lambdaself:\
self._cache.current.session_id
exceptImportError:
pass


classKeywords(object):

defreport_sauce_status(self,job_id,test_status,test_tags=[]):
importbase64
importhttplib
importos

try:
importjson
json# pyflakes
exceptImportError:
importsimplejsonasjson

username=os.environ.get('SAUCE_USERNAME')
access_key=os.environ.get('SAUCE_ACCESS_KEY')

ifnotjob_id:
returnu"No Sauce job id found. Skipping..."
elifnotusernameornotaccess_key:
returnu"No Sauce environment variables found. Skipping..."

token=base64.encodestring('%s:%s'%(username,access_key))[:-1]
body=json.dumps({'passed':test_status=='PASS',
'tags':test_tags})

connection=httplib.HTTPConnection('saucelabs.com')
connection.request('PUT','/rest/v1/%s/jobs/%s'%(
username,job_id),body,
headers={'Authorization':'Basic %s'%token}
)

returnconnection.getresponse().status

This code does two things:

1. The first part monkey patches Selenium2Library with a custom keyword for providing access to test specific Selenium session id. This is the key (job id) for sending the test status back to Sauce Labs.
2. The second part defines a custom Robot Framework keyword library with a keyword for passing the test status (and other information) back to Sauce Labs.

Next, we update src/example/product/tests/robot_test.txt to store the session id during the setup of every test and send the test result back to Sauce Labs during the teardown of every test:

*** Settings ***

LibrarySelenium2Librarytimeout=10implicit_wait=0.5
Libraryexample.product.testing.Keywords

Test SetupStart browser
Test TeardownRun keywordsReport test statusClose All Browsers

*** Variables ***

${ZOPE_HOST}  localhost
${ZOPE_PORT}  55001
${ZOPE_URL}  http://${ZOPE_HOST}:${ZOPE_PORT}

${PLONE_SITE_ID}  plone
${PLONE_URL}  ${ZOPE_URL}/${PLONE_SITE_ID}

${BROWSER}  Firefox
${REMOTE_URL}
${DESIRED_CAPABILITIES}  platform:Linux
${BUILD_NUMBER}  manual
${SESSION_ID}

*** Test Cases ***

Plone site
    [Tags]  start
Go to  ${PLONE_URL}
Page should containPlone site

*** Keywords ***

Start browser
    ${BUILD_INFO} =  Set variable
    ...           build:${BUILD_NUMBER},name:${SUITE_NAME} | ${TEST_NAME}
Open browser  ${PLONE_URL}  ${BROWSER}
    ...           remote_url=${REMOTE_URL}
    ...           desired_capabilities=${DESIRED_CAPABILITIES},${BUILD_INFO}
Run keyword and ignore errorSet session id

Set session id
Keyword should existGet session id
    ${SESSION_ID} =  Get session id
Set test variable  ${SESSION_ID}  ${SESSION_ID}

Report test status
Run keyword unless'${SESSION_ID}' == ''
    ...    Report sauce status  ${SESSION_ID}  ${TEST_STATUS}  ${TEST_TAGS}

Please, note how we had to replace suite setup and teardown with test setup and teardown). Also, because the injection of Get session id is a bit fragile, it's called and used in a way that won't fail when the keyword doesn't exist at all.

This worked for me. I hope it works for you too.

example.product is available at: https://github.com/datakurre/example.product