Top Five Browsers Tested And Ranked
“The fastest browser on Earth.” That’s what the Opera homepage has said. After five release candidates and two betas, the boys from Oslo just recently launched the final build of Opera 10.50. That’s a pretty heavy claim to make, though, especially with well-establish competition already in the field demonstrating impressive performance at an unbeatably-low price. Does it hold up?
And although you might have missed it due to iPad hysteria, both Mozilla and Google recently released new versions of their browsers as well. Opera isn’t the only one making bold statements on performance, either. In the days leading up to the unveiling of Apple’s impossibly over-hyped slate, Mozilla dropped Firefox 3.6 and claimed a whopping 20% speed gain over version 3.5. Three days later, Google quietly unveiled Chrome 4.0 stable, though the actual version number is 4.0.249.78.
Let us not forget that Cupertino also offers a Web browser. In fact, Apple claims Safari is "the world’s fastest Web browser." Sounds strangely familiar, doesn’t it? Even Internet Explorer, which hasn’t had a release since last year, is in the headlines lately. But it’s the charges being leveled againstIE that are drawing the most attention.
We’ve seen mounting pressure to abandon Microsoft’s browser by everyone from Homeland Security to McAfee and Google--even some foreign governments. If it’s so horrible, than why is anyone still using it? Is there a performance justification for why almost 60% of Internet users still cling to IE?
We decided that somebody needed to look at these browsers. So, we put all five major Web browsers through a gauntlet of benchmarks and time trials. We did this to conclusively answer the question: which is the fastest browser on Earth (at least until one of these companies launches another major update)?
System Specs And Testing Methodology
Test System Specs
Web Browser Specs
Why Windows?
Anyone who has been following my articles knows I usually work on Linux coverage. I chose Windows as the operating system for this article for three major reasons. First, it’s what most of our readers use. Second, it supports native versions of all five major Web browsers. And third, Opera’s claim that version 10.50 is "the fastest browser on Earth" was made when they released 10.50 for Windows only (the Mac and Linux clients have yet to be released at time of writing). I chose Windows 7 simply because it is the latest version. I chose the Ultimate edition because it should cover all sub-licenses. And finally, I chose the 64-bit edition because most of our readers probably have newer hardware (and plenty of RAM), thus requiring a 64-bit operating system. If you want a Linux-based browser battle, or a comparison of browser performance across operating systems, speak up in the comments below.
Setup
Once Windows was installed, it was updated before installing the browsers. After the browsers, we installed SilverLight, Flash, and Java. Most of the benchmarks were run for five iterations per browser per test, unless specifically mentioned otherwise.
Methodology
The OS was allowed to “warm-up” before running a new test. Each time we finished a benchmark, the cache was cleared on all browsers and the system reset before beginning a new benchmark.
Editorial Leaning
In the interest of full disclosure, I should go over the history of my own browser usage. I’ve been running Chrome as my primary browser for several months now. Before that, I was a Firefox user since it’s very first release. I’m reluctant to admit that I switched to Internet Explorer for version 3.0, and I suppose it must have been Netscape Navigator before that. I personally haven’t used Internet Explorer in ages, and truth be told, I have never actually used Safari before this. I check out Opera just about every time they come out with a new release, but two weeks is probably the longest uninterrupted duration that I have ever seriously used that browser. At Bestofmedia, we use Firefox to interact with our content management system.
Benchmark Results: Startup Times
In order to test startup times, we set the test Web pages as the home tabs on each browser and loaded each site into cache before testing. The OS was reset and allowed to ’warm-up’ before each iteration of the test. Times were taken using a stopwatch and measured in seconds. Timing began from the when I pressed enter on a highlighted browser icon, to the time that the browser reported the locally-saved and cached Web pages as fully loaded. I timed five runs, and used the average to report the final time.
I referenced the Quantcast Top 100 to select eight popular Web pages to test. The Google homepage was used as the test page for the Single Tab test. Bing, Hulu, Twitter, and Wikipedia round out the 5 Tab test. For the 8 Tab test, we brought in craigslist, Facebook, and YouTube from our Page Load Times benchmark. We had originally wanted to time ten tabs instead of eight, but Internet Explorer will only allow up to eight home tabs.
It’s no real surprise that Chrome starts up faster than the other contenders. After all, a quick start-up has been one of Google’s main goals with Chrome. The real surprise was how long it took Safari--a full ten seconds compared to Chrome’s one second and Firefox’s four seconds.
Opera rocks the five-tab startup test with Internet Explorer close on its heels. Chrome and Safari take third and fourth, while Firefox lags behind the pack to finish last.
Once again, Opera loads the pages first, followed by Chrome, Firefox, and IE. Safari took double the time to open eight tabs as it did five. Since this is a stopwatch test, the 0.3 seconds longer it takes Opera to open eight tabs than it does five could very well be human error. Essentially, this means that those three extra tabs didn’t even phase it - very impressive.
Benchmark Results: Memory Usage
Windows Task Manager reports that our operating system (Windows Ultimate 64-bit) uses approximately 18% of physical memory at boot (about 700MB). Both Chrome and Internet Explorer have two entries in the Windows Task Manager when only one window and one tab are open. Both browsers add additional entries when new tabs or windows are opened. All of the Web pages we used were saved to the hard drive on January 23rd, 2010.
The Google homepage was used as the test page in the Single Tab test.
Apple’s Safari uses the least memory with only one tab and window open. Google Chrome is also light on memory with a single tab, despite reports to the contrary. Opera uses the most memory, nearly twice as much as third place finisher Internet Explorer.
Bing, Google, Hulu, Twitter, and Wikipedia make up the 5 Tab test pages.
Firefox uses the least amount of memory in this test. Opera places last with Chrome not far behind, both almost doubling the memory usage of Firefox.
For the ten-tab test, we visit craigslist, Facebook, Tom’s Hardware, Yahoo!, and YouTube from our Page Load Times benchmark, and add them to our five-tab test consisting of Bing, Google, Hulu, Twitter, and Wikipedia.
Firefox again uses the least amount of memory. Chrome and Opera give us another pair of upsets, again finishing fourth and fifth (respectively). This time Opera utilizes more than twice as much memory as our winner: Mozilla Firefox.
The big surprise here is Opera’s and Chrome’s poor showing in the multi-tab tests. Overall, Firefox delivers the best memory usage results. It comes in first place for the five- and ten-tab usage tests, but fourth in the single-tab metric.
Benchmark Results: Page Load Times
Page load times were measured using a customized version of WebMonkey’s Browser Load Time StopWatch JavaScript (no, not the LifeHacker one). Along with Tom’s Hardware, I picked four popular Web sites from the Quantcat Top 100 popularity ratings to test page load times: craigslist, Facebook, Yahoo!, and YouTube. All five Web pages were saved to the hard drive on January 23rd, 2010. I launched the JavaScript in all five browsers, for five iterations each. All five runs were averaged to achieve the final score. All times were recorded by the JavaScript in milliseconds.
craigslist
Not surprisingly, craigslist was the first Web page to load on each of the browsers, with the exception of Internet Explorer. This is most likely due to the classified site’s simple design and complete lack of Flash content.
Chrome takes the lead with the simple craigslist home page, followed closely by Firefox and Safari. Internet Explorer takes fourth, while Opera comes in dead last.
The load times for Facebook fluctuate the most out of the Web sites tested during our page load time testing. This was experienced across all benchmarked browsers.
This is one of the only tests where Internet Explorer outperforms all the other browsers. For whatever reason, Microsoft’s browser loads the Facebook homepage with extreme haste. Firefox, Chrome, and Opera take second, third, and fourth (respectively). Safari takes almost twice as long as the second-place finisher Firefox, and more than four times as long as IE.
Tom’s Hardware
Firefox renders Tom’s Hardware first, closely followed by Chrome, then Safari and Opera. Internet Explorer loads our page the slowest.
Yahoo!
Load times for Yahoo! fluctuate in Safari by a full 100% during the tests. Beside the Facebook variation on all browsers, this is the most significant aberration during the page load time tests.
Along with Facebook, the Yahoo! home page is the only other test in which Internet Explorer beats its competition. This time, it’s trailed by Chrome, then Safari and Opera. Firefox is last to cross the finish line.
YouTube
Opera handles the uber-popular user-generated video site most adeptly. Firefox (2nd) and Internet Explorer (3rd) finish at practically the same time. Chrome and Safari round out the bottom two.
I suppose the overall winner here would be Firefox. While it loads the Yahoo! homepage last, Mozilla comes in first once, and second three times. Internet Explorer also gives a good showing in these tests. IE places first in two out of five tests. Both Opera and Safari do poorly compared to the others, though at least Opera had one win, loading YouTube incredibly fast.
Benchmark Results: HTML, CSS, And Tables
The benchmarks on this page all reflect a browser’s ability to load simple Web pages made of just HTML with CSS (with JavaScript timers). We also include the NonTroppo Page Loading Test because it measures the loading of such a Web page, also useful for comparing against our customized Webmonkey JavaScript from the previous page.
NonTroppo Page Loading Test
This benchmark times the load of a very simple Web page consisting of text and pictures in a table. A JavaScript applet calculates the load time.
Safari renders this page the quickest, with Chrome not far behind. Opera trails behind, more than doubling the time of Chrome and nearly tripling that of Safari. Firefox comes in a distant fourth, with Internet Explorer in fifth place, nearly doubling Firefox’s completion time.
GUIMark HTML
This is the HTML version of GUIMark’s UI rendering benchmark.
This benchmark puts Apple’s browser out ahead of the pack by a decent margin. Google’s browser comes in last place, barely achieving nine frames per second.
NonTroppo CSS Rendering
This benchmark tests the speed at which our browsers render a page heavy with CSS elements.
Safari again places first, followed by Chrome. Opera places a late third, with Firefox falling behind in fourth. Internet Explorer drags far behind in fifth.
NonTroppo Table Rendering
The NonTroppo Table Rendering test consists of a simple Web page with an absolutely massive table and a JavaScript timer. We used this benchmark to record the full page load time.
The results of our table rendering test complete the pattern set forth by the other NonTroppo benchmarks: Safari, Chrome, Opera, Firefox, and then Internet Explorer.
JavaScript By The Ton
There are almost as many JavaScript benchmarks as there are browsers to benchmark, and we decided to run all of them. Why? Because each browser points to a different benchmark in their marketing efforts. In several cases, the maker of the browser also makes the benchmark. We also like the idea of running all of the benchmarks to... benchmark them. They should all more or less reflect the same results, right?
Google V8
The Google V8 Benchmark Suite is made up of seven individual JavaScript tests, the geometric mean of these tests becomes the V8 Benchmark score. We’d expect this benchmark to favor Chrome, since they are both Google projects. In fact, the reason this benchmark exists is to test Google’s JavaScript engine of the same name. We ran the suite five times and averaged the scores. Version 5 of the Google V8 Benchmark Suite was used.
As you can see, the outcome of this test is as expected: Chrome is the winner, with Opera putting forth a good effort in second, Chrome’s WebKit cousin Safari came in a distant third, but still far ahead of the other two contenders.
JSBenchmark (CelticKane)
JSBenchmark isn’t new; actually it has been around for some time now under it’s former name: Celtic Kane. This benchmark takes only seconds to run, and even provides the average of all iterations automatically. We ran JSBenchmark five times.
JSBenchmark delivers nearly the same results as Google’s V8 Benchmark, except Safari places higher than Opera to come in second place this time. Chrome still wins by a solid margin and Internet Explorer again takes dead last.
Mozilla Dromaeo Javascript Test
Obviously this is the JavaScript benchmark from Mozilla, the makers of Firefox. Because the Dromaeo JavaScript test returns remarkably close scores, we only needed to do three runs per browser. The average of those three runs becomes the final score. Internet Explorer experienced a lockup during the base 64 encoding and decoding portion of the test. Pausing and immediately re-running the test got it going again. Opera locked up on Regular Expressions in Dromaeo JavaScript, like with IE, a pause and re-run got it going again. Though the test was eventually skipped at about 98% - this occured on all three iterations.
In the Mozilla test we see Opera way out in front. So far, in fact, that I find the result highly dubious, especially considering the errors it experiences. Chrome comes in a distant second with Safari trailing behind in third (but still way ahead of Firefox and Internet Explorer).
SunSpider
SunSpider is the JavaScript test that Apple references for much of its Safari marketing effort. The SunSpider project was originally started by the Apple WebKit team. The results of this benchmark are pretty stable, so we only needed to run this test in three iterations. As usual, the average of the three scores became our final result.
Here, we see Chrome take the win with Opera very close on Google’s heels. Safari loses much of its ground to Firefox in this test. IE does what we’ve come to expect by now, placing a distant fifth.
After reviewing the JavaScript benchmarks, we’ve decided that Tom’s has no choice but to run all of them in the future. While I personally lean toward JSBenchmark, since it isn’t affiliated with any browser, its results don’t reflect the outcome in Dromaeo. Until the reason for Opera’s devastating Mozilla score can be explained, I believe we’ll have to run all of them to get the clearest picture. If you disagree, or have an opinion on a better way to benchmark JavaScript, sound off in the comments section below.
PeaceKeeper, Acid3, And DOM
Futuremark PeaceKeeper
PeaceKeeper is a free online synthetic browser benchmark by Futuremark, the maker of popular Windows benchmarking software 3DMark. It tests browser performance independent of your network connection and creates a synthetic number representing the accumulated results of the tests. PeaceKeeper is cross-platform and can be used to compare scores between various browsers across different operating systems. We ran PeaceKeeper three times and averaged the results to achieve a final score.
Opera dominates the PeaceKeeper tests, followed closely by Chrome. Safari and Firefox place a very close third and fourth while Internet Explorer wheezes to the finish line with a score of nearly one fourth that of Mozilla and Apple.
Acid3
The Acid3 test isn’t a performance benchmark. Acid3 checks a browsers adherence to Web standards. Since the outcome will remain the same on each run, we only needed to run this test once per browser.
The images below show how a successfully-completed Acid3 test is supposed to look, followed by the Firefox and the IE results.
Both Opera and Safari score 100% in our Acid3 tests, but not as quickly or as smoothly as Chrome. Safari experiences a brief hangup at 69% before completing the test with a perfect score. Opera also hangs, but at 26%. Because Acid3 requires not only a final score of 100%, but also an exact and smooth render, we have to declare Chrome the winner of this test, even though Opera and Safari also score 100%.
Mozilla Dromaeo DOM
This is the full DOM portion of the Mozilla Dromaeo Benchmark. This test takes the longest to complete by far, so we ran it three times instead of our usual five. The average of the three runs became our final score.
Both Opera and Internet Explorer twice locked up during the DOM Query portion of Mozilla’s DOM Tests. A simple pause and subsequent re-run got the test going both times. However, IE eventually skipped at only 80% complete before moving on to the next portion of the test. Opera eventually skipped at around 95%. This occurred on all three iterations of this benchmark in both browsers.
Dromaeo DOM puts Chrome and Safari in a near tie for first, while Opera places third, despite it’s errors. Firefox takes fourth, and IE predictably comes in last.
Flash, Java, And SilverLight
GUIMark Flash
This is the Flash version of GUIMark; we used the Flex 3 version of this benchmark. Five runs were recorded, the average of which became our final score.
Opera became very slow and unresponsive when opening GUIMark Flash. Minimizing and then maximizing the window seemed to fix this. This did not occur in all iterations; the third fifth run went smoothly.
Again, despite glitchy behavior, this test shows Opera on top. It’s followed by Firefox, Safari, and Internet Explorer with nearly identical scores, and Chrome trailing behind.
GUIMark Java
GUIMark tests various UI rendering technologies, including Java. We ran the test five times and used the average to settle on our final scores. This was version Java 5 Swing of GUIMark Java.
The scores in this test are all very close, with Opera showing the only substantial lead. Second place finisher Firefox only beats out the third place Safari by one-fifth of a frame per second.
Bubblemark SilverLight
Bubblemark was used to test how each browser handled SilverLight content. We ran the SilverLight 2.0 (CLR) version of Bubblemark’s SilverLight test and set it to use 32 balls. We ran five iterations of the test, the average of which became our final score.
Firefox and Internet Explorer both tie for first place. Opera takes a decent third, while the Webkit duo of Safari and Chrome pull up the rear.
Analysis And Conclusion
This brings our Web Browser Grand Prix to an end. Some of our findings weren’t that shocking, such as Internet Explorer’s failure to adhere to Web standards (Acid3). But there were also a ton of interesting notes along the way, like Opera’s gluttony for RAM and Safari’s strong performance versus much newer versions of the other browsers. I already knew that Firefox was beginning to feel slow, but I didn’t know how bad it had become. Safari didn’t live up to its boast of being "the world’s fastest web browser." Apple’s product was beaten by Opera, and owned by Chrome. While Opera came close to living up to its claim of being "the fastest browser on Earth," close just isn’t good enough. Google Chrome is the real speed king. The table below tallies the placing of each browser throughout testing.
Chrome was counted as the first-place finisher for the Acid3 test, while Opera and Safari were both tallied as second. Firefox was counted as third. Fourth place was skipped, and Internet Explorer was counted as fifth, due to it’s utterly terrible score on that test. The SilverLight test was also a tie. This time, IE and Firefox were both counted as first place finishers. Second was given to Opera and third was skipped. Since Safari received a score almost half that of Opera, it was counted as a fourth-place finish. Chrome was given fifth. We also counted Opera’s sketchy first-place finish in the Mozilla Dromaeo JavaScript Test, despite having experienced errors causing it to not complete a portion of the test. If we did not, Chrome would have had an even greater victory.
As you can see, Google Chrome comes out on top. Although it tied with Opera for the most wins by racking up the highest number of second-place finishes, Chrome manages to take the win.
Considering that Safari has gone so long without a major new version, yet still placed so well, we cannot wait to see what’s next from One Infinite Loop.
Mozilla, on the other hand, is a different story. Though I do believe that version 3.6 did bring a significant improvement over 3.5.x, it simply wasn’t enough to compete. Since Mozilla’s latest offering is only a little over one month old, its placing in our Grand Prix is disappointing, to say the least.
This brings us to Internet Explorer, Destroyer of Netscape Navigator. The browser from Redmond finished last no less than fourteen times (more than half of the tests). Internet Explorer’s performance here is nothing less than sad.
Now, before someone cries foul about the over-saturation of JavaScript tests, I have also tallied the wins by category. The three startup time tests are counted as one. We’ve also counted the three memory usage tests as one category. Our five page load time tests, as well as the NonTroppo benchmark are counted just once, too. The table below shows the winners by category.
This table only displays the winners, rather than the previous table’s full placing results. While it may appear that Opera had a better showing than Chrome, it does not reflect how many times that browser was edged out by Chrome when neither placed first. It also does not show that Safari remained smack in the middle throughout or that Firefox had a stranglehold on fourth.
Any way you want to analyze the data, Google’s Chrome comes out on top. That’s why we’re not only calling Chrome the winner of our Web Browser Grand Prix, but we’re also awarding it the Best of Tom’s Hardware Award--the first time we’ve given such an honor to a software product. If you haven’t yet downloaded Google Chrome, you just don’t know what you’re missing.
These benchmarks give us a pretty good picture of which browser is the fastest. What these benchmarks do not reveal is the usability (or overall end-user experience) of these browsers. The staggering number of customization options available for Firefox, or the almost constant (and insufferably bothersome) prompting in Internet Explorer are just two examples of what cannot be benchmarked. Security is also a major concern, and something that was not tested for this article. We focused purely on speed and performance, and in those fields, Google Chrome takes the gold...at least in this round of the raging browser wars.
Author: Adam Overa