Some Assembly Required

The diagnostic file emitted by a crashing process in a modern operating system can contain a variety of useful information, including exception type, current instruction, CPU state, call stack, and sometimes the entire contents of the current thread’s stack or even the entire process heap. So why is it called a “core dump”?

For years I thought this was an amusing Star Trek reference by the original implementors of UNIX, after all the episodes in which the Enterprise‘s reactor threatens to explode and Geordi has to save them by “dumping the warp core,” but it turns out the actual explanation is much more prosaic.
Continue reading ‘Why is it called a “core dump” anyway?’ »

The annotated slides for my GDC talk on Forensic Debugging and Crash Analysis, containing my speaker’s notes and some narration, are now available for download in PDF format here. The PowerPoint should appear on the GDC Vault and Valve’s Publications webpage soon, too.

This week I’m looking into the Steam side of Valve’s automated customer crash collecting technology, and what we can do to accumulate and usefully expose customer stability data to all our partners who ship with Steamworks. If you think this would be a useful feature for your studio to use in its games, please let me know! You can either contact me through the comment form here, or by mailing me directly at my Valve address.

While debugging a smashed stack may seem like a heroic feat, the most heroic thing about my talk is the amount of time, effort, and care my friends spent to help me put it together. I would never have made it to the GDC, let alone made any sense whatsoever onstage, without the support of all my friends inside and outside Valve.

A special badge of courage is due those those who bravely offered to sit through my rehearsals, gave me details for slides, or in some other way helped distill ninety minutes of inane gibbering into one hour of assembly and win:

• Jeep Barnett
• Dan Berger
• Iestyn Bleasdale-Shepherd
• Bank Charnchaichujit
• John Cook
• Kerry Davis
• Bruce Dawson
• Michelle Garrison
• Bronwen Grimes
• Dave Kircher
• Tejeev Kohli
• Joe Ludwig
• Jason Mitchell
• Kyle Monroe
• Marc Nagel
• Olivier Nallet
• Alfred Reynolds
• Dave Riller
• Mike Sartain
• Dave Saunders

Thanks, guys and gals — I wouldn’t have done it without you.

Thanks to everyone who came to my “Forensic Debugging” talk at the 2011 Game Developers’ Conference. I hope it was valuable to all who attended.

The lecture covered a great deal of ground in a short time, and so the slides necessarily had to go by rather quickly. Eventually a video of my presentation will be at the GDC vault. In the meantime, I’ve exported most of the deck as a series of annotated PDF images here, to help fill in the notes of anyone who might have attended but missed a point or two.

The intent of my talk was to give a general overview of the forensic mindset, the tools available, and demonstrate that rather than being a dark art, the science of crash analysis is something that everyone can learn.

In the realm of “things I really wish I had paid more attention to while we were crunching on Left 4 Dead“, I’ve recently been going through the talks from last June’s Gamefest. There is a lot of really great stuff in there, and I’ve only skimmed through a few of them so far, but among those few the standouts are:

• Xbox 360 Compiler and PgoLite Update: where Bruce Dawson describes the pure awesome that has been poured into the XDK’s CPU performance-analysis tools, such as the Continuous Capture mode for PIX that lets you determine what caused a perf spike after it happens. (Yes, a perf tool can be said to contain pure awesome, for definitions of awesome encompassing branch prediction.) He also presents a lock-free pipe implementation added to the XDK libraries for interthread communication.
• New CPU Features in the XDK: where Dawson continues his reign of awesome in a discussion of improvements to the 360 compiler. He covers faster asserts in array bounds-checking, an improvement to the __fsel intrinsic (the ?: operator isn’t so useless after all), caveats on certain VMX branching operations, and a better way to do SIMD constants. There’s also a bunch of improvements to the 360 compiler itself including better load-hit-store avoidance, better inline assembly, and a more useful /QXSTALLS pipeline simulator (which makes the compiler emit a nerd-readable assembly file for each .CPP with all your performance issues highlighted).
• At Least We Aren’t Doing That: Real Life Performance Pitfalls: Allan Murphy lays out the fifteen most common performance screwups that the Microsoft support team sees. Each of them has been something I’ve gnashed my teeth over in the past ten years myself, and more than one is something game programmers been dealing with since the days of the PS2 (so really we should know better by now).

I’m still working my way through these docs and planning to put together a précis on them for the devs at Valve, so hopefully I’ll be able to post that here along with some of my own thoughts on the topics as well. Half the things in “At Least We Aren’t Doing That” are certainly issues I’ve been known to get into long lunchtime rants over!

Some other interesting documents on my vast and teetering to-read pile are Gustavo Duarte’s articles on PC internals like x86 memory translation, Christer Ericson’s old GDC2003 talk on memory optimization, and Ulrich Drepper’s definitive but dense What Every Programmer Should Know About Memory. The last one in particular I’m reviewing as a background citation to some notes I’m writing myself on console data caches (because I’d rather have something online and free to link to rather than send everyone to the bookstore to for a copy of Computer Organization And Design): it has a lot of good stuff in it, though much of it is specific to Linux running on x86 CPUs and very different from the way game consoles work. In particular, passages such as this one may bring a chuckle to PS3/Cell developers:

A computer can have a small amount of high-speed SRAM in addition to the large amount of DRAM. One possible implementation would be to dedicate a certain area of the address space of the processor as containing the SRAM and the rest the DRAM. The task of the operating system would then be to optimally distribute data to make use of the SRAM…. While this is a possible implementation it is not viable … this approach would require each process to administer in software the allocation of this memory region…. So instead of putting the SRAM under the control of the OS or user, it becomes a resource which is transparently used and administered by the processors.

The transparent, all-knowing processor indeed.

The slides for my talk at the 2008 Game Developers’ Conference, “How To Go From PC To Console Development Without Killing Your Studio,” are now available online at the Valve website. 

The talk describes some of the many pitfalls that face developers who are used to developing their games exclusively for the home personal computer, but are now about to embark upon their first crossplatform title, shipping on both PC and console at the same time. These pitfalls have brought some studios to their doom, but fear not: I also provide techniques for how to deal with each of these issues, without having to rewrite most of your game from scratch.  Some of the techniques are production issues, like dealing with certification. Others address programming problems like improving framerate on the PowerPC — some assembly is required.

My speaker’s notes are embedded in the slides: hover over or click the small dialog bubble that Adobe Acrobat superimposes at the top left.

I’ve also started a Q&A page to address questions people have asked me about this talk, either in person at the conference, or through email since then. If you have questions of your own, please feel free to leave a comment.