On a recent Clarity call, one of the topics we discussed was JavaScript Testing Frameworks.

It’s a complex topic, more complex than it is in many other platforms, because JavaScript testing frameworks are pretty granular – there are lots of pieces and they can be used together or independently.

There are:

• Test runners
• Unit testing frameworks
• Integration testing frameworks
• Assertion frameworks
• Mocking frameworks

Some tools are very specifically just one of these. Other tools cover more than one category or blur the lines between several of these.

If you’ve used some of these, you might have a preference based on what feels right for you. If you haven't, you might use a different criteria: popularity.

If you don’t have a preference, picking the most popular test framework is useful for a few reasons:

• You are crowdsourcing the research. If the most people are using a particular framework, there’s often a reason for that. It might not be the best framework, or the up-and-coming framework, but it’s probably mature and reasonably solid.
• If you have an issue, you probably won’t be the first person to have that issue, and finding answers to your problems is often easier.
• When hiring or outsourcing, it’s a lot easier to find someone with experience with the popular frameworks than the niche frameworks.

Gauging Popularity

Of course, if you’re trying to assess the popularity of a tool, there’s a bunch of different ways you could do that, including:

• npm statistics
• stack overflow questions
• job trends (e.g. indeed)
• google search trends

Candidates

What are some of the primary candidates for JavaScript and testing?

QUnit

QUnit is a unit testing framework that was originally part of JQuery.

Popularity:

• NPM: ~20k downloads in the last month. Often over 500 downloads per day.
• Job Trends: QUnit was initially a strong contender against Jasmine until early 2013, after which it has stayed steady while Jasmine has quadrupled. In early 2014, Mocha also surpassed it and has now doubled.
• Stack Overflow: a distant third in total questions, and rarely top of the trending tags for Javascript test frameworks.
  Basically, QUnit has been around for a while, it’s mature, but it’s clearly not at the same popularity level as Jasmine and Mocha. It’s probably not a terrible choice, but those people looking for the popular framework should focus their energy on either Jasmine or Mocha instead.

Jasmine

Jasmine is a behaviour-driven test framework, intended to be a fairly complete system on its own – it comes with assertions, spies, mocking.

Popularity:

• NPM: Almost ~300k downloads in the last month. Was hitting ~1k/day in 2014, regularly over 5k/day in Q1 2015 and now often over 10k/day.
• Job Trends: Jasmine is the leading JavaScript testing framework, although Mocha has been making steady progress, and is probably the strongest alternative.
• Stack Overflow: Jasmine is the leader in tag trends and total questions, but the data is close and varies wildly.
• Google: Jasmine got the early lead and has held it. Mocha is making some headway.

Mocha

Test framework designed with an extension model in mind.

Popularity:

• NPM: Mocha seems to dominate NPM downloads, with 2M downloads in the last month, and many days over 50k downloads/day this year.
• Job Trends: Mocha is a relative newcomer, but has already surpassed QUnit and is making headway on Jasmine. Still, Jasmine is featured in twice as many job postings as Mocha, so it’s still the underdog.
• Stack Overflow: Half of Jasmine’s total questions, but pretty competitive on tag trends.
• Google: Mocha got the latest start on the search trends, but has already surpassed QUnit. It’s not clear if it’s making headway on Jasmine on search trends.
• Stackshare: Mocha handily beats Jasmine on Stackshare. I’m not sure if people are more happy to share that they’re using Mocha (e.g. aspirational) or if the kinds of people on Stackshare are more likely to use Mocha (e.g. selection bias).

Mocha is clearly a credible alternative to Jasmine, and seems better suited to people who want to mix and match pieces of the framework rather than relying on a single complete solution. I feel like Mocha is ‘in fashion’ or ‘trending’ right now now. Still, I don’t think you’ll go that far wrong with either Mocha or Jasmine.

Karma

Karma is a test runner, popularized by AngularJS. It is test framework agnostic, and works with QUnit, Jasmine and Mocha.

Popularity:

• NPM: Almost 1M downloads in the last month and regularly over 30k dowloads/day this year.
• Job Trends: Karma is the only JavaScript test runner with meaningful penetration on job postings that I could find, although it’s certainly less common than any of the test frameworks.
• Stack Overflow: Vastly ahead of all the other JavaScript test runners on total questions.
• Google: Karma’s pretty recent on Google search trends, but it’s already competitive with Mocha in terms of search.

Chai

Chai is an assertion framework, but one that has an extension model of its own, allowing yet another layer of library to layered on top of it.

Popularity:

• NPM: ~850k downloads in the last month and regularly over 20k downloads per day. Chai is about twice as popular as Should, but they’re in the same league.
• Job Trends: Chai is the only assertion framework to have meaningful results on job trends. Of course, doesn’t help that its two strongest competitors, expect.js and should.js have more generic names that make searching a little harder.
• Stack Overflow: Doesn’t seem to place on tag trends, but well ahead of should.js on total questions.
• Google: Up alongside QUnit, but below Mocha and Jasmine.

Should.js

Should is a BDD-style assertion framework for node and browsers.

Popularity:

• NPM: Almost 400k downloads int he last month, and regularly over 10k/day downloads this year.
• Job Trends: Although it’s harder to search for “javascript” and “should” and get useful results, neither “shoud.js” nor “shouldjs” seem to show up on significant numbers of job posts.
• Stack Overflow: Well behind Chai on total questions.
• Google: Hard to get clean results, but “should.js” is clearly below Chai.

Others

There are, of course, countless others. Some of them are specific to a particular platform, but many are simply alternatives to those listed above, or another tool related to testing but not fitting within the broad categorization above.

I’ve tried to cover the most popular set, but there are lots of others on the long tail of popularity, such as:

• Expect is another BDD-style assertion framework, built by Automattic, for node.js and browser testing. At about 200 downloads/day and <10k downloads in the last month, it’s clearly far less popular than Should and Chai.
• Wallaby is another test runner.
• NodeUnit is a test framework built on node.js’s assertions.
• PhantomJS is a headless WebKit with scriptable JavaScript API which can be used for testing.
• JSTestDriver is another test runner.
• Cucumber.js is a behaviour-driven development (BDD) framework for JavaScript.
• Dalek is more of an end-to-end framework, with a runner, assertions, browser support and more.
• Atomus is a test framework for client-side tests in a node environment by simulating a browser.
• Testem is another JavaScript test runner. At 200k NPM installs in the last month, it’s probably the most credible alternative to Karma.
• Sinon adds spies, stubs and mocks, and is often used with Chai.
• Protractor is an end-to-end testing tool for AngularJS.
• Although code coverage tools aren’t specific to testing, they are often used in conjunction with testing. Some of the candidates here are:
• Istanbul
• Blanket
• JSCover
• Saga
• Nightwatch is an end-to-end testing framework that works with Selenium and node.js.
• Jest is layered on top of Jasmine with the goals of making Jasmine that much easier to use (“painless”).
• Chutzpah is another JavaScript test runner that seems to be built with the Microsoft stack at least partially in mind.

What Should I Use?

That’s ultimately your decision, but if popularity is your primary criterion, then Karma with either Jasmine or Mocha & Chai are clearly the leaders.

If you want to talk about this or something else, call me on Clarity, or get in touch.