GDC 2011: Crash Analysis and Forensic Debugging

This is the support page for my talk at the 2011 Game Developer’s Conference, Forensic Debugging: How To Autopsy, Repair, and Reanimate a Release-built Game.

This talk took place on 1:30pm Wednesday, March 2nd. Eventually the full slides for this program, with annotation, will be available at the GDC Vault and on Valve’s website. In the meantime, I’ve exported most of the deck with my speaker’s notes as a PDF document here, to help fill in the notes of anyone who might have attended but missed a point or two.

Here’s the contents of the bibliography slides from the end of the lecture:

 

 

•Pietrek, Matt. Just Enough Assembly Language To Get By, I & II. Microsoft Systems Journal, Feb-June 1998.  bit.ly/9HKCOk bit.ly/bJf1R0

•(Microsoft) Kirk Glerum, Kinshuman Kinshumann, Steve Greenberg, Gabriel Aul, Vince Orgovan, Greg Nichols, David Grant, Gretchen Loihle, and Galen Hunt. Debugging in the (Very) Large: Ten Years of Implementation and Experience. bit.ly/4lBgkH

•(Jet Propulsion Laboratory) Reeves, Glenn; Neilson, Tracy. The Mars Rover Spirit FLASH Anomaly.
http://hdl.handle.net/2014/39361
•IBM. PowerPC Microprocessor Family: Programming Environments Manual for 64 and 32-Bit Microprocessors.
http://bit.ly/gHemWI
•Intel. Intel 64 and IA-32 Architectures Software Developer’s Manual.
http://www.intel.com/products/processor/manuals/
•SunSoft; IBM. SYSTEM V APPLICATION  BINARY INTERFACE PowerPC Processor Supplement.
http://bit.ly/fRvDe9
•Apple. Introduction to Mac OS X ABI Function Call Guide.
http://bit.ly/foRCnE
•Motorola, Inc. PowerPC Embedded Application Binary Interface.
http://bit.ly/i9GOyz
•IBM. Developing PowerPC Embedded Application Binary Interface (EABI) Compliant Programs.
http://bit.ly/hbhlyk
•Taylor, Ian Lance (Zembu Labs). 64-bit PowerPC ELF Application Binary Interface Supplement.
http://bit.ly/hoWPfF
•ELF-64 Object File Format.
http://bit.ly/fUel3t
•Wikipedia. Executable and Linkable Format.
http://en.wikipedia.org/wiki/Executable_and_Linkable_Format
•Eager, Michael. Introduction to the DWARF Debugging Format.
http://bit.ly/hLBoF2
•Apple. Debugging and Symbolizing Crash Dumps in Xcode.
http://bit.ly/C2RqC
•Microsoft. Symbol Server and Symbol Stores.
http://bit.ly/ecSyzj
•Microsoft. Debugging Tools for Windows.
http://bit.ly/eXwcf2
•WinDbg command cheat sheet:
http://bit.ly/3Jc0p7
•Sen, Saikat. A WinDbg Tutorial.
http://bit.ly/4h4PZx
•Google Breakpad project home page:
http://bit.ly/7FfWzQ

You can leave a response, or trackback from your own site. Follow any responses to this page through the RSS 2.0 feed.

5 Comments

  1. Dustin Russell says:

    Awesome talk!

    March 2, 2011, 3:52 pm

  2. GDC 2011 | .mischief.mayhem.soap. says:

    [...] Crash Analysis and Forensic Debugging, Ellan Ruskin (Valve). Well, OK, more like a teaser, nothing there yet, but observe this space, Bookmark It [...]

    March 2, 2011, 8:06 pm

  3. Daniel says:

    Excellent talk. I spent a good chunk of last summer working with the breakpad library, and I ended up coding a 360 dump processor/stack walker and using XBWatson to watch for crashes, grab the minidump, and pop up a crash dialog on the user’s PC. It was really great having the same interface for handling crashes on PC and 360.

    I would have liked to do the same thing with PS3, but it was a summer internship and I had to get back to finishing my undergrad.

    March 5, 2011, 9:29 pm

  4. Chad Austin says:

    Great presentation! We solve a lot of the same problems at IMVU. I’ve written a series of blog posts on our techniques, including stitching stack traces out of C++ exceptions: http://chadaustin.me/tag/crashes/

    I wanted to make it to your presentation but had a conflict… thanks for spreading the word! There’s definitely an attitude in some studios that “if we code it right, it won’t crash in the field”, but you have no control over what else is on the system…

    March 6, 2011, 3:18 am

  5. Robert says:

    Awesome presentation. We run into the exact same problems at my workplace, and while I didn’t actually glean too much new info, this was just about the best presentation I’ve ever seen on the topic. Mandatory viewing for our new hires :)

    March 7, 2011, 2:29 pm

Leave a Reply