End-to-End JavaScript Testing is Easy (the minimal way to do it)

It’s not actually that hard to automate browser testing! Let’s dive in. We won’t use any obscure or (relatively speaking) bleeding-edge technologies here; just the basics. We’re trying to strip out all the fancy names and get to a nice, minimal, testing setup.

The players

The first question to answer is this: what do we use to test with? You’ve probably heard of Capybara, QUnit, Jasmine, Mocha, Karma, Protractor, WebDriver, Selenium, etc., but which does what? Let’s sort it out.

Regular old JS, do-it-all testing.

Candidates: QUnit, Jasmine, Mocha

What they’re good for: Testing JavaScript

If you want to assert that 2+2 is 4 using your fancy new addition framework, these are the usual suspects. QUnit is very old but tried-and-true (jQuery uses it); Jasmine is newer; and Mocha is the newest. From what I’ve heard Mocha has better asynchronous support than Jasmine, so when in doubt, I just use that.

Make the browser work automatically

Candidates: Selenium

What they’re good for: Automating web browsing

Yes, Virginia, there is only one actual browser automator, and that’s Selenium! I spent years (well, hours) googling this before I realized what was going on.

All the smoke and mirrors about PhantomJS and Protractor and Karma aside, any testing framework that is doing automatic testing of an actual web page in a real browser uses Selenium. (PhantomJS is a pseudo-browser, Protractor uses Selenium, and Karma, I think, is just for testing raw, not-the-same-as-your-real-web-page JavaScript).

Talking to Selenium from JavaScript

Candidates: WebdriverJS (update: now WebdriverIO, thank goodness), WebDriverJS (I wish I was joking), Nightwatch.js

What they’re good for: Talking to Selenium from your JavaScript node app.

You can use Selenium from almost anything – a Java/C# program, C++ I think? – but here we’re going to stick with JavaScript. In order to access the browser and do things like browser.get(“http://google.com”) from 35 different browsers all at once, we need a library, and that’s where WebDriverJS comes in. Just use WebDriverJS with the capitalized D, it’s the official one.

*Update: WebdriverJS is now WebdriverIO (still lower-case D). Its main claim to fame is requiring slightly less typing. Should you use it?

It’s not by the Selenium people as far as I know, but it’s been around for a while so I’d feel safe using it.

Cooking it up

Now that we’ve got that sorted out, our path is pretty simple, and we’ll need all three. What we’ll do is:

  1. Start Selenium in the background; it can automatically spin up Firefox, IE, and Chrome windows on its own like magic.
  2. Install WebDriverJS so we can talk to Selenium (https://code.google.com/p/selenium/wiki/WebDriverJs)
  3. Create tests in Mocha that use the WebDriverJS library to automate the browser.

Here we get the best of every world. Selenium, a well tested browser automator, handles making Firefox do spooky automatic clicking. We use the official WebDriverJS to talk to Selenium, and we put our tests, just like any other test, in a standard Mocha example – no need to have some separate, slightly-different-syntax not-invented-here framework just to do our E2E testing.

While I could write a tutorial that would rapidly go out of date, I think it would be better to just link to the latest up to date tutorials and make some notes.

The recipe

Prerequisites: node, npm, java

  1. Start by installing Mocha and get the basic test running.
  2. Download the Selenium server.
  3. Install WebDriverJs (selenium-webdriver) so you can use Selenium in Node.
  4. Download and include the proper Selenium drivers for Firefox/IE/Chrome
  5. Run Selenium with Java, making sure to specify the drivers you downloaded in the previous step.
  6. Replace the basic test you wrote in Mocha with this one: https://code.google.com/p/selenium/wiki/WebDriverJs#Writing_Tests
  7. That’s it!

Some caveats I ran into:

  • First, while mocha understands being installed locally and globally (npm install vs npm install -g), selenium-webdriver only understands being installed locally.
  • Downloading the drivers for FF/IE/Chrome is a bit of a pain since the web links redirect to different places.

Other than that though, just use the standard instructions for Mocha/Selenium/WebDriverJs. No need to remember anything else from a random blog post (AKA, this one).

Conclusion

The only tricky part about autotesting your web browser is deciding what to use. Using Mocha + Selenium with WebDriverJS is a nice, commonly supported combination that’s minimal.

Adding things like Capybara, Nightwatch, and Protractor is nice, but frankly, you’ve got to install Selenium anyway, and you’re probably already using Mocha or something similar for your browser testing. All those new, fancy-name frameworks are doing is adding a bunch of syntactic sugar and extra dependencies. Alternatively, there are plenty of tutorials for setting this up, that require all sorts of caveats and weird tricks to remember. Who needs that?

7 comments

  1. Hi Chris,

    This is a great post so thank you! We’re just starting to learn about automated testing and trying to select the tools that we need. This post puts everything together in one place for a quick global overview of everything that you need to get started quickly. We had focused in on using selenium and javascript but there are so many different components and frameworks to choose from and this post has helped quite a bit.

    My questions are as follows:
    Has anything changed within the landscape of tools over the last year since you wrote this post that might change which products/tools your recommend?

    We want to use browserstack or saucelabs. Furthermore, we’d like to reduce the amount of items we need to install or maintain locally and leverage cloud service wherever possible. Does that change any of you recommendations?

    Can you elaborate on nightwatch.js? It seems like it adds some good features but I’m not really sure how critical it is and exactly what key features it will add and/or what complexities it adds?

    What about unit testing? What would you recommend to include some simple javascript and css unit tests?

    Finally, can you recommend any reporting frameworks for the test results? I’ve read about the Allure Testing Framework but I’m not sure what the other options are available or recommended?

    I know the purpose of your post was to keep the testing process as minimal as possible but I was hoping you could elaborate on these points since we’re looking to start off our automated testing initiative with a few more bells and whistles. Thanks again for the great post!

    -Paul

  2. Hi Paul,

    Thanks for writing in! Yeah…the tyranny of choice. It got much simpler once I knew which places to start with. In regard to your questions…

    1. I haven’t been following the tooling space heavily at all recently. I do remember there being a new tool to automate API testing and API documentation, but my personal philosophy is “use only as much as you need to make the job simple.”

    2. I’m not that familiar with cloud-based services, but I would be very surprised if they didn’t have a lot of dedicated documentation on the subject.

    3. I haven’t used Nightwatch myself. I did some googling and the best I can get is that it’s yet another wrapper around Selenium, like WebdriverIO or WebDriverJS:

    WebDriverJs, e.g. the version we installed using npm install selenium-webdriver is just one version of a WebDriver Client API written in JavaScript. If you’re keen to programatically control browsers via JavaScript, there are also other options.

    http://code.tutsplus.com/tutorials/an-introduction-to-webdriver-using-the-javascript-bindings–cms-21855

    4. This is a question I can’t answer very well. Our tests generate XML reports when they’re done; a Jenkins plugin just knows what to do and they show up in our CI right away. :)

  3. Hey Damian! There were several ppeole involved with migrating the various projects. From my experience it was relatively painless, as the WebDriver API is much improved over Selenium RC. Having decent page objects in place also meant that a lot of the migrations happened with little or no changes to the tests themselves. I’m aware of a few issues involving native interactions with WebDriver, however these weren’t available at all with Selenium RC so it’s still better than what we had before. All these issues will be worked out sooner or later. If you’re interested in more migration stories, I would suggest popping into #mozwebqa on irc.mozilla.org. Also, many of the team also hang out in #selenium on freenode too.

Leave a Reply

Your email address will not be published. Required fields are marked *