Best Practices for Managing Drag in Game Development

Jamie Fristrom recently wrote about the concepts of velocity and drag on his blog. (Part 1, Part 2, Part 2.5, Part 3)

Velocity, in terms of software development, is the rate that a team can pump out features. Take all of the estimates for all of the features that were implemented in an iteration and divide it by the length of the iteration. Tada, you have the velocity of your project for that particular iteration.

After a few iterations you have historical data. What Jamie found is that, over time, the velocity of a project decreases. He coined the term drag to describe this phenomena.

There are a number of reasons for this drag. Feature creep, newly discovered work, and technical debt are some of the major reasons.

I don’t have enough experience on the production side to comment on those issues. I have now worked on PC, XBox 360, Sony PSP, and Sony PS3 projects and have noticed a number of common day-to-day development issues that contribute to this drag.

Build Times Should Be Fast

Most games are written in C/C++. On bleeding-edge platforms the compiler will also be bleeding-edge. This means that the compiler in general, and optimizer in particular, will be slower than what is actually desirable. Compiling template-heavy code is also burdensome for compilers.

Dynamically linked libraries are not practical on most consoles. Linking all object code into one executable is a long process. In a lot of cases, incremental linking is not available. So, each time a programmer would like to iterate on a piece of code a full relink of the entire executable is required.

Physical Architecture

The code should be physically architected such that their are very distinct layers. Generally, games consist of three distinct layers: Core, Game Engine, Game Specific.

The Core layer will contain code that deals with math, memory management, file loading, parallel programming, the core graphics hardware code, extension languages like Lua, core network code, etc.

The Game Engine layer will contain code such as your simulation system, the lobby engine, network messaging system, sound effects system, particle systems, visibility management, camera system, etc.

The Game Specific layer will contain the finite-state-machines for your bots, the HUD, the front-end UI, gameplay mechanics, etc.

These three layers should be built independent of higher layers. Each layer should have its own Makefile or Solution file. A programmer should be able to add Core functionality without having to build (or even see) the Game Engine or Game Specific layers.

Lower layers should be stable from the perspective of higher level layers. A programmer working on the Game Specific layer should not have to build the Game Engine or the Core layers. The build system should build these two layers and the compiled .LIB or .ELF files should be sitting in whatever revision control system that you use.

A gameplay programmer should be able to arrive at her desk first thing in the morning, sync up to the latest build, and just build the top layer of code to be able to start making edits.

The core programmer should be able to come in to work and not have to sync all of the data for the game. She should be able to develop her code on super simple console applications at the command line.

Distributed Builds

A distributed build system should be used. Depending on the platform, the choices are SN-DBS, IncrediBuild, or distcc. These build tools can not improve link times, but they can ship off files to be compiled on other machines. Having a big 8-way machine (or better) in the server room is a must. This machine should be at minimum building the one gold-copy of the game that QA will be testing and will finally be shipped.

Bulk Builds

A bulk build system can also be used. The bottleneck for a lot of compilers is startup time. Startup time can be reduced by concatenating a number of .CPP files into one file before shipping it off to the compiler. This requires an external build script that can be written in Scons or Jam.

Low Level Code

The lower layers of code are easier to unit test. The Core layer, at minimum, should have a large number of unit tests. This is especially important when targetting multiple platforms. The best unit test suite for C++ I’ve encountered is UnitTest++.

Platform specific code should almost all sit in Core. If there is platform specific code in Game Engine then there needs to be a very good reason why. Parallelism is one tough case for this best practice. Having a performant Game Engine on a SMP system like the XBox 360 and an asymetric system such as the PS3 can be difficult.

Scripting Languages

Code that is written in a scripting language will build much faster than C/C++. The extension language of choice for game programming is Lua. Other languages such as Python, Unreal Script, and QuakeC are also in use. Lisp has been used as an extension language in the past too. Lua can be compiled down to byte code before execution. However, during development it is best to have it interpreted. Only having to change Lua script to add or change functionality in a game is a huge advantage.

A scriptable interface should be designed up front for most systems. It should be fairly clear what functionality is executed in base code or in script. Err on the side of implementing too much in script. Functionality can pulled into core code later if it is required.

Data

There is also gigabytes of data that is editor-friendly but will need to be compiled into a format that is friendly for the target platform to load.

The entire game should build in one line on the command line and one GUI button for artists, designers, and QA.

Load Times Should Be Fast

On consoles, data is usually loaded from optical discs. DVD drives have the added detriment that data loads slower as it is located closer to the center. Data should be laid out contigously to avoid seek time.

Contiguous data loading is also very important during development. Quick iteration time on level data is very important for artists, designers, and gameplay programmers.

The technique usually requires block-loading followed by pointer fixup. Think of the process as analogous to what a linker does. Each dynamic object piece of data in the game should have a pointer to a static C-style struct. The C-style struct should be naturally aligned for the target platform. The loading system should be able to suck data off of a disc or a drive and slam it into a big chunk of RAM. A post-process pass can go through all of the dynamic structures and fixup the pointers to the static data. This works similar to how a Symbol Table or String Table is resolved.

A Proxy System should allow levels to load nearly instantaneously. Every piece of data should have a placeholder. Actual models should pop into place of placeholder cubes. Textures should pop into place as low resolution and then high resolution textures are loaded. White noise should be replace by actual sound effects.

The Proxy System should also allow the hotloading of new data without reloading the entire level. If a piece of data changes on disk, it should automatically load.

Most Importantly: All of this should be done in pre-production