For years I thought this was an amusing Star Trek reference by the original implementors of UNIX, after all the episodes in which the Enterprise‘s reactor threatens to explode and Geordi has to save them by “dumping the warp core,” but it turns out the actual explanation is much more prosaic.

In the days before computers used capacitor-based dRAM, the dominant technology for main memory was to store bits as magnetic polarization in a grid of tiny ferrite cores (iron rings). Thus a machine’s main memory was literally called core memory, or simply core. When a computer of this era crashed, it would simply output the entire contents of main memory to the punchcard printer, literally dumping core to output. Later, these core dumps became large files on the machine’s drum or disk drive, and eventually core memory became obsolete in favor of static and dynamic RAM, but the name remained.

If that sounds painful, consider the Whirlwind computer developed at MIT around 1951 (pictured below). When this 2kB, 0.04mHz behemoth crashed, it would simply display the entire contents of memory as a string of octal numbers on a dedicated CRT screen. Then, an automated camera would take a picture of this CRT on microfilm¹. You, the programmer, would get the developed microfilm the next morning and display it on a projector, which would be your crash debugger. Operand highlighting was done with a brightly colored marker on the film transparency, and the disassembler was a guy you called on the phone to ask what instruction 0125715 meant. At least the dump files themselves were small — about 35 millimeters, more or less.

Control room for MIT's Whirlwind computer, circa 1951

¹Everett, R.R. The Whirlwind I Computer. Proceedings of the 1951 Joint AIEE-IRE Computer Conference, pp. 70-74, Philadelphia, PA, 1951.

This week I’m looking into the Steam side of Valve’s automated customer crash collecting technology, and what we can do to accumulate and usefully expose customer stability data to all our partners who ship with Steamworks. If you think this would be a useful feature for your studio to use in its games, please let me know! You can either contact me through the comment form here, or by mailing me directly at my Valve address.

A special badge of courage is due those those who bravely offered to sit through my rehearsals, gave me details for slides, or in some other way helped distill ninety minutes of inane gibbering into one hour of assembly and win:

• Jeep Barnett
• Dan Berger
• Iestyn Bleasdale-Shepherd
• Bank Charnchaichujit
• John Cook
• Kerry Davis
• Bruce Dawson
• Michelle Garrison
• Bronwen Grimes
• Dave Kircher
• Tejeev Kohli
• Joe Ludwig
• Jason Mitchell
• Kyle Monroe
• Marc Nagel
• Olivier Nallet
• Alfred Reynolds
• Dave Riller
• Mike Sartain
• Dave Saunders

Thanks, guys and gals — I wouldn’t have done it without you.

The lecture covered a great deal of ground in a short time, and so the slides necessarily had to go by rather quickly. Eventually a video of my presentation will be at the GDC vault. In the meantime, I’ve exported most of the deck as a series of annotated PDF images here, to help fill in the notes of anyone who might have attended but missed a point or two.

The intent of my talk was to give a general overview of the forensic mindset, the tools available, and demonstrate that rather than being a dark art, the science of crash analysis is something that everyone can learn.

• Xbox 360 Compiler and PgoLite Update: where Bruce Dawson describes the pure awesome that has been poured into the XDK’s CPU performance-analysis tools, such as the Continuous Capture mode for PIX that lets you determine what caused a perf spike after it happens. (Yes, a perf tool can be said to contain pure awesome, for definitions of awesome encompassing branch prediction.) He also presents a lock-free pipe implementation added to the XDK libraries for interthread communication.
• New CPU Features in the XDK: where Dawson continues his reign of awesome in a discussion of improvements to the 360 compiler. He covers faster asserts in array bounds-checking, an improvement to the __fsel intrinsic (the ?: operator isn’t so useless after all), caveats on certain VMX branching operations, and a better way to do SIMD constants. There’s also a bunch of improvements to the 360 compiler itself including better load-hit-store avoidance, better inline assembly, and a more useful /QXSTALLS pipeline simulator (which makes the compiler emit a nerd-readable assembly file for each .CPP with all your performance issues highlighted).
• At Least We Aren’t Doing That: Real Life Performance Pitfalls: Allan Murphy lays out the fifteen most common performance screwups that the Microsoft support team sees. Each of them has been something I’ve gnashed my teeth over in the past ten years myself, and more than one is something game programmers been dealing with since the days of the PS2 (so really we should know better by now).

I’m still working my way through these docs and planning to put together a précis on them for the devs at Valve, so hopefully I’ll be able to post that here along with some of my own thoughts on the topics as well. Half the things in “At Least We Aren’t Doing That” are certainly issues I’ve been known to get into long lunchtime rants over!

Some other interesting documents on my vast and teetering to-read pile are Gustavo Duarte’s articles on PC internals like x86 memory translation, Christer Ericson’s old GDC2003 talk on memory optimization, and Ulrich Drepper’s definitive but dense What Every Programmer Should Know About Memory. The last one in particular I’m reviewing as a background citation to some notes I’m writing myself on console data caches (because I’d rather have something online and free to link to rather than send everyone to the bookstore to for a copy of Computer Organization And Design): it has a lot of good stuff in it, though much of it is specific to Linux running on x86 CPUs and very different from the way game consoles work. In particular, passages such as this one may bring a chuckle to PS3/Cell developers:

A computer can have a small amount of high-speed SRAM in addition to the large amount of DRAM. One possible implementation would be to dedicate a certain area of the address space of the processor as containing the SRAM and the rest the DRAM. The task of the operating system would then be to optimally distribute data to make use of the SRAM…. While this is a possible implementation it is not viable … this approach would require each process to administer in software the allocation of this memory region…. So instead of putting the SRAM under the control of the OS or user, it becomes a resource which is transparently used and administered by the processors.

The transparent, all-knowing processor indeed.

The slides for my talk at the 2008 Game Developers’ Conference, “How To Go From PC To Console Development Without Killing Your Studio,” are now available online at the Valve website. 

The talk describes some of the many pitfalls that face developers who are used to developing their games exclusively for the home personal computer, but are now about to embark upon their first crossplatform title, shipping on both PC and console at the same time. These pitfalls have brought some studios to their doom, but fear not: I also provide techniques for how to deal with each of these issues, without having to rewrite most of your game from scratch.  Some of the techniques are production issues, like dealing with certification. Others address programming problems like improving framerate on the PowerPC — some assembly is required.

My speaker’s notes are embedded in the slides: hover over or click the small dialog bubble that Adobe Acrobat superimposes at the top left.

I’ve also started a Q&A page to address questions people have asked me about this talk, either in person at the conference, or through email since then. If you have questions of your own, please feel free to leave a comment.