Introducing the 2012 Mobile Benchmark Matrix
by Jarred Walton on January 6, 2012 8:02 PM EST2012: Meet Our New Mobile Benchmark Suite
Testing computer hardware can be a difficult process. On the one hand there’s a desire for more information and benchmarks, and on the other hand there’s a desire for timely reviews. Our goal at AnandTech has always been to deliver the most comprehensive reviews possible, and while we strive to timeliness there are occasions where additional testing or questions may delay a review. Ultimately, there’s a balancing act that needs to be maintained, and over time we periodically refresh our review suite and testing methodologies.
With 2012 now here, we’re launching a new suite of benchmarks for our laptop reviews. We'll also have the results from our first laptop using the new tests, courtesy of ASUS' G74SX. Some of the tests have already been in use for a while and others are brand new. In order to provide a single location with a list of our benchmarks and testing procedures, we have put together this short overview. We plan on using the following test suite throughout 2012, and while it’s possible we will add some benchmarks, we don’t have any plans to stop using any of the following at least for the next year.
General Performance Tests
Starting with our general tests, all of these have been in use for several months at least, with many tests dating back to 2010 and earlier. We’ll continue to use the full PCMark 7 suite, PCMark Vantage (x64), Graysky’s x264 HD encoding test, Cinebench 11.5, 3DMark06, 3DMark Vantage (Entry-Level and Performance defaults), and 3DMark 11. We’ll also continue with our battery life tests (now with Internet Explorer 9 in place of IE8) and LCD tests. So for most areas, our test suite remains largely unchanged—we’re finally dropping Cinebench 10, but that’s about it.
As we’ll mention in the conclusion, we’re willing to add some additional general performance benchmarks if there are any specific requests. One of the difficult things to quantify with modern PCs is how fast they are in the things most people do on a regular basis. Part of the problem is that most PCs from the past three or four years are all “fast enough” for generic tasks like surfing the web—if you’re actually reading the content of web pages rather than just repeatedly loading a complex page, I’m not sure most users would notice the difference between a 2GHz Core 2 Duo or Athlon X2 laptop and a quad-core i7-2760QM. This is why battery life is such an important element, as where many wouldn’t notice the difference between a web page loading in two seconds and a web page loading in one second, they’re far more likely to notice two hours of battery life versus four or eight hours. Anyway, let us know if you have other mobile benchmarks you’d like us to consider.
With that out of the way, we’ll save the next page for the major changes: our updated gaming suite.
Facebook
Twitter
RSS
48 Comments
View All Comments
codedivine - Friday, January 6, 2012 - link
Can you include the Chromium compilation test that you to do for CPU tests?And I would very much like to see Blender too if possible.
JarredWalton - Friday, January 6, 2012 - link
I need to see if I can get Anand (or someone else) to walk me through the entire Chromium compilation process. I tried to get that working at one point and after wasting several days with nothing to show for it I gave up. LOL. (Yes, I followed a guide, but even then it didn't work for me.) If someone has a step-by-step guide that they: A) know works, and B) is free (open source) for all needed tools, and C) works with Windows 7 64-bit... post a link. Remember, not only do I need to figure out how to run the compile test, but so does Dustin and anyone else we have doing laptop reviews. :-)Conficio - Friday, January 6, 2012 - link
I'd second that. A compilation test would be great.I'd even love to see some sort of Java test, may be an Eclipse Import a large project from zip and wait until it has finished updating the workspace.
coolhardware - Friday, January 6, 2012 - link
I realize it is not as demanding as many of the other games, but so many people play it (and will play it for years to come) that is sure seems like it deserves a spot.:-(
Still, thanks for keeping your benchmark suites fresh and relevant!
PS if anybody has tips/suggestions on the best <a href="http://www.jdhodges.com/2011/06/best-starcraft-2-g... for SC2</a> please let me know, thanks! [not talking about bells and whistles...but rather comfort, performance and reliability]
coolhardware - Friday, January 6, 2012 - link
Sorry about that malformed URL. Here's the correct one: http://www.jdhodges.com/2011/06/best-starcraft-2-g...PS any other games people are missing or games that you are excited to see WERE added?
JarredWalton - Friday, January 6, 2012 - link
Ultimately, Dustin and I decided to drop SC2 from the list because it:1) doesn't scale at all with more than 2 CPU cores
2) is rather a pain to benchmark
3) is DX9 only
4) runs more than fast enough at all but the most demanding settings
Basically, it's more of a CPU benchmark in most cases, as even with lots of stuff happening on screen the two-core nature of the engine doesn't allow it to scale that well. Sure, at Ultra quality with 4xAA forced on in your drivers (another pain to deal with), it drops FPS down to around 23 on the G74SX, but that still doesn't address the other items I'm concerned with. I figure if Civ5 and TWS2 run on a laptop, SC2 will run fine as well.
coolhardware - Friday, January 6, 2012 - link
Cool, thanks for listing those factors.It makes a lot more sense to me now to drop it, since it sounds like a real PITA to benchmark :-) Also, you raise a good point that if higher end games play okay on a particular laptop then SC2 should be good to go...
Appreciate all the hard work you guys do!
zebrax2 - Friday, January 6, 2012 - link
Maybe you could include at least rage?JarredWalton - Friday, January 6, 2012 - link
We had intended to include Rage, right up to the point where it was released and we discovered that it is largely useless as a benchmark:http://www.anandtech.com/show/4970/rage-against-th...
ltcommanderdata - Saturday, January 7, 2012 - link
Could you consider including Brink? Admittedly it's id Tech 4, but heavily modified to make use of OpenGL 3.1. It also uses virtual texturing like RAGE. The upcoming Prey 2 also uses id Tech 4 so performance in Brink does have forward looking relevance.