We use SauceLabs at work to handle the maintenance of environments specifically for running web tests. I highly recommend using SauceLabs over trying to maintain your own internal testing lab. (I know I helped create our Android web test environments and maintain the iOS Selenium environment. It was a lot of work and not a lot of fun especially with deadlines breathing down my neck.)

As mentioned in a previous post, my started using Splinter for writing web tests. Splinter has excellent support for running remote Selenium tests, and hence support for running tests in SauceLab virtual VMs. However one thing I drove me nuts is that when looking at the SauceLabs dashboard, I had no idea if a test scenario succeeded or not.

Adding Success and Failure Messages

It turns out that a SauceLabs VM has no notion of a successful or failed test… or more correctly a Selenium web driver doesn’t. But you can correlate a SauceLab VM test run (job) with a successful or failed test.

The only issue is that the example given uses nose and selenium rather than lettuce and splinter. So how did I get it working for splinter and lettuce? By adding the following code after a scenario is run:

@lettuce.after.each_scenario
def report_to_saucelabs(scenario):
     #Remember to quit your webdriver after you run your tests!
    scenario.browser.quit()  

    # This was the most sure fire way to check if this was a remote web driver vs a local one.
    if not isinstance(scenario.browser, splinter.driver.webdriver.remote.WebDriver):
        return 

    # Now gather all the failed steps and figure out what status needs to get sent
    failed_steps = [step for step in scenario.steps if step.failed]
    test_successful = bool(failed_steps) == False

    # Finally we have to connect to SauceLabs using the saucelabs Python client (pip install sauceclient)
    sc = sauceclient.SauceClient(SAUCELABS_ACCOUNT, SAUCELABS_ACCESS_KEY)
    sc.jobs.update_job(scenario.browser.driver.session_id, passed=test_successful)