Aug 042012

Before giving a conference talk on Node.js earlier this summer, I ran some limited JavaScript benchmark tests, since I couldn’t find any recent (this year) data posted. Inspired by John Resig’s performance analysis a few years ago, I decided to expand on that study by updating browsers and adding more benchmark suites to get a more comprehensive view of performance. I’ll be presenting results for the 5 top desktop browsers:

  • Google Chrome v20.0.1132.47 (V8 Engine)
  • Microsoft Internet Explorer v9.0.8112.16421 (Chakra Engine)
  • Firefox v14.0.1 (SpiderMonkey Engine)
  • Opera v12.00 (Carakan Engine)
  • Safari v5.1.7 (SquirrelFish Engine)

Browser/engine versions are up to date at time of writing. Values reported are the mean of 10 consecutive runs with each browser. Tests were performed on a Intel Core i7 Macbook Pro with 8GB RAM, running Windows 7 Ultimate SP1.

First, let’s check out the SunSpider 0.9.1 JavaScript benchmark test:

Interestingly, the latest version of Microsoft’s Chakra engine slightly edged out Chrome and its V8 engine in my test. Fear not, the result is not statistically significant. However, I do have to give credit to MS. For the capabilities tested by SunSpider, Chakra currently performs as well as V8. The remaining browsers went Opera, Firefox, Safari, from best to worst. These differences, though they appear small on the graph, were statistically significant (p < 0.01).

Moving on, I went to the V8 JavaScript Benchmark Suite. This is the suite used by Google internally to optimize and test the V8 engine, so expect V8 to perform well on this. Results:

Chrome/V8 takes a solid lead here, with Firefox/SpiderMonkey a distant second, and everyone else trailing.

Finally, Mozilla’s Kraken benchmark suite:

Again, Chrome/V8 and Firefox/SpiderMonkey in a league of their own, with Chrome slightly ahead. Results here were statistically significant (p < 0.01) as well.

So, for now, the V8 engine remains the champ (nothing to worry about here, Node.js users), though its nice to see an unexpected second wind from contenders like IE9 in some benchmarks. If I get time, I’ll try to add a few more benchmark suites to round this out, though I did find some reported results for Chrome vs Firefox on Peacekeeper that maintain V8′s superiority.

 Posted by at 9:23 pm
  • Hoopz

    Why test Safari on Windows rather than on Mac OS X? 5.1.7 isn’t even the latest version.

    • Hoopz

      Not trying to be a Mac fanboy, but the vast majority of Safari users are on Mac, not Windows. So I just re-ran the tests on OS X Mountain Lion with Safari 6. If I compare my results to yours using Chrome V8 as the reference level, the results are different enough for all three tests. And I’m using Chrome 21, which one would assume would slightly outperform 20.

      Sunspider: Chrome 21 249ms, Safari 6 248ms
      Your Safari takes 1.2x longer, mine is equal.

      V8 bench: Chrome 21 10599, Safari 6 5209Y
      Your Safari gets 1/3 of the score, mine gets 1/2.

      Kraken: Chrome 21 2723ms, Safari 6 4007ms
      Your Safari takes 4.3x longer, mine 1.47x.

      Chrome is still the undisputed champion, but Safari does not seem as bad as this makes it out to be.

      • Constantine Aaron Cois

        That’s a good question!

        In a comparative study, you have to limit the uncontrolled variables as much as possible. In this case, changing the OS (or even the computer I ran the tests on) could cause unaccounted for variations in results. So, to make the comparisons as valid as possible, I had to run all of the tests in the exact same environment. Since IE is not available for OSX, the only environment that could support all browsers was Windows. Unfortunately, Safari 6 has not yet been released for Windows.

        I may do some tests of Safari 6 on OSX as an addendum, but it would not be directly comparable to the others, as the OS environment could cause unknown differences in outcomes.

        • Hoopz

          I completely understand, however, as a developer I’m interested in the browsers people have and use every day. Given that you’re testing on a Macbook, it would be interesting for future benchmarks to cover both platforms to get a more representative picture. Also as a side note, I had to measure your graphs to compare results, it would be convenient to get the raw numbers at the end too.

          • Constantine Aaron Cois

            That’s a good call – I’ll try to post the actual data today or tomorrow.

45 queries in 1.651 seconds