Valve Hardware Day 2006 - Multithreaded Edition
by Jarred Walton on November 7, 2006 6:00 AM EST- Posted in
- Trade Shows
Building a Threaded Game Engine
Valve Software researched and experimented with all three multithreading models, and some of the results were promising. The first thing they did was to look at coarse threading, being that it is usually the easiest approach. Like Quake 4 and Call of Duty 2, Valve also has a client-server architecture in the Source engine, so that's where they started. The results can be summarized with the following slide:
There were benefits to the work that was done, but long-term such an approach is destined to fall short. If you only have a client and a server thread, you are only using two processor cores at most, and even then it is unlikely that you're making full use of both cores. The decision was ultimately reached that the only approach truly worth pursuing as a long-term solution is hybrid threading. It will require the most effort but it will also give the most benefit. As long as you're already rewriting large portions of your engine, the thinking goes, you might as well make sure you do it right so as to avoid doing the same thing again in the near future. The next step is to come up with a threading framework that will allow the developers to accomplish their goals.
Operating systems and compilers already provide some support for multithreaded programs. In some cases, this support is sufficient, but when it comes to building a real-time gaming engine anything short of a custom threading tool may not provided the desired flexibility. Valve's problems with OS and compiler level threading can be summarized in the following two slides.
We've already stated several times that making multithreaded programs can be difficult, and the problem is exacerbated when you have many programmers -- some of them junior programmers that have never seen threaded code in their life -- all trying to work together. Valve decided that ultimately what they wanted to create was a threading framework and a set of tools that would allow the programmers and content creators to focus on the important thing, actually creating the games, rather than trying to constantly deal with threading issues. Tom Leonard and a couple other helpers were tasked with creating this toolset, and as you can guess by the existence of this article, their efforts are about to reach fruition.
We're not going to try and snowball you with any more information on the low level details at present. Basically, Valve is working hard to provide a toolset that will enable their developers -- as well as mod creators and engine licensees -- to take full advantage of multi-core platforms. They aren't just looking at dual core or even quad core; their goal is to build an engine that is scalable to whatever type of platform users may be running in the future. Of course that doesn't mean they won't be rewriting portions of their engine at some point anyway, but all signs indicate that they should be at the forefront of creating multi-core games and gaming engines.
Valve Software researched and experimented with all three multithreading models, and some of the results were promising. The first thing they did was to look at coarse threading, being that it is usually the easiest approach. Like Quake 4 and Call of Duty 2, Valve also has a client-server architecture in the Source engine, so that's where they started. The results can be summarized with the following slide:
There were benefits to the work that was done, but long-term such an approach is destined to fall short. If you only have a client and a server thread, you are only using two processor cores at most, and even then it is unlikely that you're making full use of both cores. The decision was ultimately reached that the only approach truly worth pursuing as a long-term solution is hybrid threading. It will require the most effort but it will also give the most benefit. As long as you're already rewriting large portions of your engine, the thinking goes, you might as well make sure you do it right so as to avoid doing the same thing again in the near future. The next step is to come up with a threading framework that will allow the developers to accomplish their goals.
Operating systems and compilers already provide some support for multithreaded programs. In some cases, this support is sufficient, but when it comes to building a real-time gaming engine anything short of a custom threading tool may not provided the desired flexibility. Valve's problems with OS and compiler level threading can be summarized in the following two slides.
We've already stated several times that making multithreaded programs can be difficult, and the problem is exacerbated when you have many programmers -- some of them junior programmers that have never seen threaded code in their life -- all trying to work together. Valve decided that ultimately what they wanted to create was a threading framework and a set of tools that would allow the programmers and content creators to focus on the important thing, actually creating the games, rather than trying to constantly deal with threading issues. Tom Leonard and a couple other helpers were tasked with creating this toolset, and as you can guess by the existence of this article, their efforts are about to reach fruition.
We're not going to try and snowball you with any more information on the low level details at present. Basically, Valve is working hard to provide a toolset that will enable their developers -- as well as mod creators and engine licensees -- to take full advantage of multi-core platforms. They aren't just looking at dual core or even quad core; their goal is to build an engine that is scalable to whatever type of platform users may be running in the future. Of course that doesn't mean they won't be rewriting portions of their engine at some point anyway, but all signs indicate that they should be at the forefront of creating multi-core games and gaming engines.
Facebook
Twitter
RSS
55 Comments
View All Comments
edfcmc - Thursday, November 23, 2006 - link
I always thought the dude with the valve in his eye was Gabe Newell. Now I know better.msva124 - Thursday, November 9, 2006 - link
Um.....you're kidding, right?
JarredWalton - Thursday, November 9, 2006 - link
Nope. That's what Valve said. Graphics and animations can be improved, but there are lots of other gameplay issues that have been pushed to the side in pursuit of better graphics. With cards like the GeForce 8800, they should be able to do just about anything they want on the graphics side of things, so now they just need to do more in other areas.msva124 - Thursday, November 9, 2006 - link
And 640K of RAM should be enough for anybody? Yeah right. There are certainly other things besides graphics that need to be tended to, but when even rendered cutscenes don't look convincing, it's extremely premature to say.
JarredWalton - Thursday, November 9, 2006 - link
So you would take further increases in graphics over anything else? Personally, I'm quite happy with what I see in many titles of the past 3 years. Doom 3, Far Cry, Quake 4, Battlefield 2, Half-Life 2... I can list many more. All of those look more than good enough to me. Could they be better? SURE! Do they need to be? Not really. I'd much rather have some additional improvements besides just prettier graphics, and that's what Valve was getting at.What happens if you manage to create a photo-realistic game, but the AI sucks, the physics sucks, and the way things actually move and interact with each other isn't at all convincing? Is photo-realism (which is basically the next step -- just look at Crysis screenshots and tell me that 8800 GTX isn't powerful enough) so important that we should ignore everything else? Heck, some games are even better because they *don't* try for realism. Psychonauts anyone? Or even Darwinia? Team Fortress 2 is going for a more cartoony and stylistic presentation, and it looks pretty damn entertaining.
The point is, ignoring most other areas and focusing on graphics is becoming a dead end for a lot of people. What games is the biggest money maker right now? World of WarCraft! A game that will play exceptionally well on anything the level of X800 Pro/GeForce 6800 GT or faster. There are 7 million people paying $15 per month that have basically said that compelling multiplayer environments are more important to them than graphics.
msva124 - Thursday, November 9, 2006 - link
Fine. You win.JarredWalton - Thursday, November 9, 2006 - link
Sorry if I was a bit too argumentative. Basically, the initial statement is still *Valve's* analysis. You can choose to agree or disagree, but I think it's pretty easy to agree that in general there are certainly other things that can be done besides just improving graphics. I don't think Valve intends to *not* improve graphics, though; just that it's not the only thing they need to worry about. Until we get next-gen games that make use of quad cores, though, the jury is out on whether or not "new gameplay" is going to be as compelling as better visuals.Cheers!
Jarred
cryptonomicon - Wednesday, November 8, 2006 - link
http://images.anandtech.com/reviews/tradeshows/200...">http://images.anandtech.com/reviews/tra...2006/val...So, what are those steel or aluminum models at the end? Are they the real world references for the Team fortress source weapon models? :D
yyrkoon - Wednesday, November 8, 2006 - link
I'm pretty sure 'it' in this sentence should be "it's", or "it is" (sorry, but it was bad enough to stop me when reading, thinking I mis-read the sentance somehow).
Good article, and it will be interresting to see who follows suite, and when. Hopefully this will become the latest fad in programming, and has me wanting to code my own services here at home for encoding video, or anything that takes more than a few minutes ;)
yyrkoon - Wednesday, November 8, 2006 - link
meh, sorry, that 'typo' is on the second to the last page, I guess about half way down :/