Agile Game Development

Guys:

From a brief study of the game industry, I might list a few impediments to
"agile" software development. Whereas this essay will define "agile" as
"good", we should not make the mistake to think that any agile "best
practice" must slavishly mirror into game development. Achieving the results
shown by "agile" development in other industries - reduced odds of failure,
accurate schedule estimates, and absurdly low defect rates - will require
adjusting the practices to fit game programming topologies.

Planning Game, Early Delivery
With a few exceptions, the game industry could not sustain Early Delivery.
Games are entertainment, and changing levels every few months would destroy
the tenuous addictions that designers so carefully foster. The game industry
has more in common with the motion picture industry than the other sectors
of programming. A game starts as a pitch, very similar to a screenplay
pitch, and ends with a rollout. If the game succeeds it funds its studio's
failures. Like a screenplay, a big game design, up front, must pitch a known
genre to a very clearly defined demographic. That helps a studio's successes
exceed its failures. This practice excuses each game's development from much
internal risk management. A successful game generates enough cheddar to fund
rewriting the next game's engine. (A studio simultaneously shipping two
games generates trade literature headlines.)

Frequent, regular internal releases have many more benefits than merely
preparing for frequent delivery. Frequent releases depend on a steady stream
of feature requests, where each feature can cover any subset of database,
middleware, engine, logic, art (assets), and front-end. Sorting these
requests in order of business value has a powerful effect on design, because
the high-priority items get tested, reviewed, integrated, and reinforced the
most during the remaining development. However...

Continuous Integration, Daily Deployment, Frequent Releases
Game development works with very large teams, split by function into groups
with similar abilities. The business world, working on projects requiring a
narrower range of abilities, has learned to avoid this tactic, and to
colocate very small teams of diverse abilities. Splitting teams by function
requires goal donors (designers and technical leaders) to split feature
requests by function, not by business value. So these forces, coupled with
the incredibly high volume of data born in each game title, powerfully
dis-integrates production, making continuous integration much more difficult
than it could be. A game's source code and data file can take several
computer-hours to rebuild.

So these forces, in turn, inspire game developers to re-invent a practice
that business programmers are learning to avoid - rampant code branching. A
game team might have a mini-branch for the sound guys, a mini-branch for the
AI guys, etc. The cost of re-integrating each teams' new outputs, to finally
create the integrated feature the designer requested, delays the most
important feedback. Designers should review the effect of each request, on
all game aspects, as soon as possible to permit that review to accurately
and usefully drive the new requests.

Pair Programming, Shared Code Ownership
The game industry is widely reputed to program game engines using only the
worst practices of both Waterfall and Code-and-Fix. Designers request engine
abilities far in advance of reviewing the levels that exploit them.
Engineers perform Big Design Up Front to the best of their abilities. When
any Waterfall system runs off the end of its planned design, developers are
forced to switch to Code-and-Fix. Those of us experienced with the long-term
results of Waterfall have all seen projects with many clever designs, many
over-designed parts, and many parts riddled with spaghetti, cross-patches,
and horrid cruft.

I lack the tact that other agilistas exercise daily when I admit that game
development attracts and retains programmers with ... special skills. Pride
of code ownership must preceed the all-night coding sessions that keep such
code afloat. Game developers learn their systems very well, and the industry
relies on their experience shipping games to request them to ship again,
with a slightly better engine each time. Within these mega-iterations,
source code loses many opportunities for collaboration to force in new
flexibilities. "We can't do X until Q completes Y" is a common impediment.

Test-Driven Development, Customer Tests
Over time, game projects grow a huge "decompression debt" of easily-fixed
bugs. Shipping a game requires an early feature-freeze and then a very long
party to stomp insects. However, within the many layers that make up a game
are many latent tests, and an extraordinarily huge number of side-effects
that could easily become "characterization tests". But only pure TDD from
scratch can beat legacy test retrofits. Pure TDD will generate full-feature
test fixtures capable of testing arbitrarily complex aspects of play that
entry-level testers could only dream of.

And all games have a scripting layer of some kind. The Lua language is the
leading candidate for this role. And the great irony of game development is
this: Most business projects don't have a scripting layer, and have Onsite
Customers who should not write source code. Agile development relies on
Customer Tests, written by customers, to constrain each feature they
request. Game projects, by contrast, require designers to author Lua scripts
to add the hooks and triggers to each level. So game projects have Onsite
Customers who can write scripting code, and game projects also have the
scripting layer for them to write with. To close this gap, game projects
need only require designers to author the tests for each high-level feature
they request. And game projects must tune their scripting layers to make
these techniques easy and robust.

A Prescription
Start a game small, not super-big. Maybe start with a designer, two engine
guys, and two asset guys. They finish a level, incrementally adding
challenges, and maintaining a zero-tolerance for bugs. They measure their
velocity, and their gold-owner adds resources until the velocity levels off.
The game remains playable and deliverable at all times. It might not have
all its levels and its resolution cinematics, but it will have zero bugs and
all playability features for the levels it grows.

This recommendation turns traditional development side-ways. The storyboard
becomes the schedule, not the bottleneck, and levels not yet completed are
the ones most easy to change. Shortening the feedback between feature
requests and results, by any means necessary, will lead to agility for the
game industry.

--
Phlip
http://industrialxp.org/community/bi...UserInterfaces

Phlip wrote:
> From a brief study of the game industry, I might list a few
> impediments to "agile" software development. Whereas this essay
> will define "agile" as "good", we should not make the mistake to
> think that any agile "best practice" must slavishly mirror into
> game development.

I'd like to point out a couple of areas where your brief study is
inconsistent with my own experience, but first...

> Achieving the results shown by "agile" development in other
> industries - reduced odds of failure, accurate schedule estimates,
> and absurdly low defect rates - will require adjusting the
> practices to fit game programming topologies.

You're assuming that these are big problems in game development. In
my experience, other problems dwarf these factors.

Failure:

Few games that pass the green light stage get canceled before
completion. In my experience I have only worked on 1 game in 10 that
got a green light and was later canceled. Getting a green light is
usually quite tough so this weeds out most poor game designs before
they get too far. The most common cause for canceled projects (in my
admittedly limited experience of such things) are for marketing and
budget reasons unconnected with the game's development progress. I
suspect this is outside the scope of things agile methods are likely
to cover. That's not to say that failures due to technical problems
don't occur, but I've not heard of any. As far as I'm aware, all of
the recent studios that closed in the UK did so in spite of
delivering their games, not because they couldn't.

Accurate Schedule Estimates:

Contrary to popular belief, most game development teams are fairly
good at estimating project schedules. The problem is usually that
these accurate estimates aren't used. Instead most publishers
substitute ridiculous estimates based around delivery dates dictated
for (usually very questionable) marketing reasons. For example, in a
project that I worked on, our technical director came up with a spot
on estimate for development time based on years of development
experience. This was then cut in half by the publisher and later
extended month by month until we delivered pretty much bang on our
estimate date - in other words late! This is how most experienced
teams deliver accurate estimates and yet paradoxically miss their
deadlines. Again, it's difficult to imagine where agility would help
here.

Low Defect Rates:

I've played a lot of console games over the years, and in that time
I've only ever found one crash bug in a published game. That was
after playing continuously for many hours and then starting over. On
the other hand, I've lost count of the amount of times I've lost
work and had to reboot my PC while using perfectly standard "mature"
development tools. Now it's true that PC games also suffer from this
instability, but these games are mostly written and tested by the
exactly the same people who write the console games. That makes me
think this is a PC / OS problem rather than defects inherent in PC
game code or the game development process.

A defect in game development is a vague concept. A bug can mean
anything from a wrong shade of green on a texture to an inability to
complete the game. The priority of bugs is usually somewhat
arbitrary: a single incorrect spelling of some on screen text can
render the game unshippable; while extremely annoying consequences
of bizarre design choices are often waived as "unimportant" as the
deadline closes. In fact, many of the "bugs" that ship with a game
are emergent in nature. They don't arise from a mistake in the
coding, but from an unintended consequence of the specification of
rules of the game.

Okay, back to your essay:

> (A studio simultaneously shipping two games generates trade
> literature headlines.)

I'm not sure where you got this impression. It does happen. (And
probably more than you realise precisely because it does not
generate trade literature headlines when it does?)

> Game development works with very large teams, split by function
> into groups with similar abilities.

True. And so many game development teams are anything but agile by
definition. However, there are still many smaller game developments
that could benefit.

> So these forces, in turn, inspire game developers to re-invent a
> practice that business programmers are learning to avoid - rampant
> code branching.

I've never seen that myself. Almost every project I've worked on, or
even heard about used simple source control with everybody working
on the same branch. Although I'm not sure what branching has to do
with agility. Is there some pet peeve you have with this practice?

> Pride of code ownership must preceed the all-night coding sessions
> that keep such code afloat. Game developers learn their systems
> very well, and the industry relies on their experience shipping
> games to request them to ship again, with a slightly better engine
> each time.

In my experience, code ownership is imposed on programmers. In that
sense it's more like code responsibility than ownership. Also, in my
experience, programmers frequently collaborate (in secret if
necessary) to get the job done. It's not pride on the part of
programmers, it's a manager's need for a name to blame.

> Shipping a game requires an early feature-freeze and then a very
> long party to stomp insects.

This is usually the policy, but in practice most games are adding
features right up to the burn of the final master. It's particularly
easy to introduce emergent bugs at this point.

> Shortening the feedback between feature requests and results, by
> any means necessary, will lead to agility for the game industry.

But does the game industry need agility? And what for?

In my experience, speeding up the feedback loop hasn't improved much.
If anything, it led to experimentation without purpose. At
best it has had no effect. I once made every predefined gameplay
constant editable from a single text file. The designer took that
version away to "tweak" the design. When he brought it back the
following week he had changed one parameter by 0.01 of a second.
Nobody noticed.

I think game development could benefit from many of the techniques
of agile methods, but I don't think agility in itself offers much.

Phlip,

> With a few exceptions, the game industry could not sustain Early Delivery.

It worked for Id Software's Quake 3. Perhaps that's one of the
exceptions...

You could always release (early and often) to a small, select, trusted
population of early players.

Laurent

Laurent Bossavit wrote:

> > With a few exceptions, the game industry could not sustain Early
Delivery.
>
> It worked for Id Software's Quake 3. Perhaps that's one of the
> exceptions...
>
> You could always release (early and often) to a small, select, trusted
> population of early players.

One myth of XP is it requires early and frequent delivery.

You get all the benefits if you release regularily to a small crowd of
users.

One goal for my essay was skipping over some of these myths, before they
caused the usual diversionist threads.

--
Phlip
http://industrialxp.org/community/bi...UserInterfaces

You need to understand Phlip that Game development is by far the most
complicated form of programming there is! Period. Most of the games out
there today are at the cutting-edge and do well compared to the lame crap
business developers write.

Maybe we should be learning something from them? Also, are you coming from
the philosophy that XP and agile development is the "right" way to program?
If so, I think game developers show it is at least not the only way.

Stede

"Phlip" <phlip_cpp@yahoo.com> wrote in message
news:mu3md.21811$C74.2854@newssvr31.news.prodigy.c om...
> Laurent Bossavit wrote:
>
> > > With a few exceptions, the game industry could not sustain Early
> Delivery.
> >
> > It worked for Id Software's Quake 3. Perhaps that's one of the
> > exceptions...
> >
> > You could always release (early and often) to a small, select, trusted
> > population of early players.
>
> One myth of XP is it requires early and frequent delivery.
>
> You get all the benefits if you release regularily to a small crowd of
> users.
>
> One goal for my essay was skipping over some of these myths, before they
> caused the usual diversionist threads.
>
> --
> Phlip
> http://industrialxp.org/community/bi...UserInterfaces
>
>

Stede Troisi <stede@verizon.net> loquated like no one had ever loquated
before with:

> You need to understand Phlip that Game development is by far the most
> complicated form of programming there is! Period. Most of the games
> out there today are at the cutting-edge and do well compared to the
> lame crap business developers write.
>
> Maybe we should be learning something from them? Also, are you coming
> from the philosophy that XP and agile development is the "right" way
> to program? If so, I think game developers show it is at least not
> the only way.
>
> Stede

Sorry, but there's plenty of non-game development software that is just as
complex. Maybe you meant to say 'pressure driven and competitive' instead
of 'by far the most complicated form of programming there is.' It is
certainly equal to other development tasks, but it is the
time/money/resource constraints that make development harder than most other
forms of development, not the complexity of the code itself.

WTH

Paul Sinnett wrote:

> Phlip wrote:

> > From a brief study of the game industry, I might list a few
> > impediments to "agile" software development. Whereas this essay
> > will define "agile" as "good", we should not make the mistake to
> > think that any agile "best practice" must slavishly mirror into
> > game development.
>
> I'd like to point out a couple of areas where your brief study is
> inconsistent with my own experience, but first...

I listed the three major reasons why "agile" development has been reputed to
be "good". Having never seen a game project fail, up close (for various
definitions of "fail"), I will personally rely on "good". Maybe "avoiding
all-nighters" should be the primary goal.

If I had seen such a project fail, I suppose I could find a way to recast
each failure point back onto my list of agile results just as easily as I
could recast them away from them. It's all "good".

> > Achieving the results shown by "agile" development in other
> > industries - reduced odds of failure, accurate schedule estimates,
> > and absurdly low defect rates - will require adjusting the
> > practices to fit game programming topologies.
>
> You're assuming that these are big problems in game development. In
> my experience, other problems dwarf these factors.

What top three problems would you list? Be brutal - many shops grow
accustomed to some situations, just as ones eyes grow accustomed to the
dark.

> > (A studio simultaneously shipping two games generates trade
> > literature headlines.)
>
> I'm not sure where you got this impression. It does happen. (And
> probably more than you realise precisely because it does not
> generate trade literature headlines when it does?)

My untenable premise is a shop builds their engine, builds a game, and
starts a re-write on the engine. I'm not saying that's rampant (or
especially bad). I suspect that truly flexible and bug-free code would allow
such an engine to obey the "Software Product Lines" style.

> > Game development works with very large teams, split by function
> > into groups with similar abilities.
>
> True. And so many game development teams are anything but agile by
> definition. However, there are still many smaller game developments
> that could benefit.

I don't understand. Are you saying the teams _must_ split, and that therefor
process improvement must be ... "less" agile?

> > So these forces, in turn, inspire game developers to re-invent a
> > practice that business programmers are learning to avoid - rampant
> > code branching.
>
> I've never seen that myself. Almost every project I've worked on, or
> even heard about used simple source control with everybody working
> on the same branch. Although I'm not sure what branching has to do
> with agility. Is there some pet peeve you have with this practice?

Why yes. I have seen code forking churn non-game projects into complete mud.
Proper respects for observing unbranched game projects, and I should not
behave as if I had determined it were the industry average.

> > Pride of code ownership must preceed the all-night coding sessions
> > that keep such code afloat. Game developers learn their systems
> > very well, and the industry relies on their experience shipping
> > games to request them to ship again, with a slightly better engine
> > each time.
>
> In my experience, code ownership is imposed on programmers. In that
> sense it's more like code responsibility than ownership. Also, in my
> experience, programmers frequently collaborate (in secret if
> necessary) to get the job done. It's not pride on the part of
> programmers, it's a manager's need for a name to blame.

Ah, I forgot to blame the managers, too. Point.

> > Shipping a game requires an early feature-freeze and then a very
> > long party to stomp insects.
>
> This is usually the policy, but in practice most games are adding
> features right up to the burn of the final master. It's particularly
> easy to introduce emergent bugs at this point.

Per the "crunch mode" thread, it seems I tried to describe "crunch mode"
here. Feature freezes are not bad (it's just a game, not air traffic
control).

But if scheduling were working, were are the "crunch modes" coming from? My
non-game crunch modes came from not maintaining a zero-tolerance policy
against even minor bugs, early in development.

> > Shortening the feedback between feature requests and results, by
> > any means necessary, will lead to agility for the game industry.
>
> But does the game industry need agility? And what for?

Let's assume they currently have "bad", and need "good". Neither of us have
disagreed, so far, on those fundamental issues. However,

> In my experience, speeding up the feedback loop hasn't improved much.
> If anything, it led to experimentation without purpose. At
> best it has had no effect. I once made every predefined gameplay
> constant editable from a single text file. The designer took that
> version away to "tweak" the design. When he brought it back the
> following week he had changed one parameter by 0.01 of a second.
> Nobody noticed.

That's not feedback. You didn't get early feedback on whether your tool was
needed.

The recommendations, again:

- start small, colocate a team, and pair
- mix the team; design, art, and math
- seamless integration, no code forking
- leverage the script layer for design-tests
- one level at a time, one trigger at a time
- TDD, refactoring, shared code, daily deploment, etc.
- absolute zero tolerances for anything remotely resembling a bug

Within that cycle, I suspect that a design could tweak a parameter via
several different scenarios, including user stories, story tests, pairing
with a developer, tweaking the scripts, etc.

If you complain about any one item in the list, then you play CTips's game
of describing some practice going out of balance, while ignoring the
opposing forces from the other practices.

> I think game development could benefit from many of the techniques
> of agile methods, but I don't think agility in itself offers much.

Maybe I didn't say, "we should not make the mistake to think that any agile
'best
practice' must slavishly mirror into game development," in the first
paragraph.

--
Phlip
http://industrialxp.org/community/bi...UserInterfaces

You have a good point but I still feel game development is in a field
of its
own. First, almost anyone can write a payroll system (think C3) but
inventing cutting-edge game graphics and sounds that have never been
done
before is tough. Add all the other factors that go into game design and
I
think it would be almost impossible to do XP at companies like EA,
Activision and Blizzard.

Then again, your math is probably better than mine and it probably is
easier
for you. It would be interesting to see a AAA title done with XP
though.
Working 90 hour weeks minimum and enough teenagers to make 1 game a day
for
the next 500 years I don't think it will happen any time soon.

Stede Troisi wrote:

> You need to understand Phlip that Game development is by far the most
> complicated form of programming there is! Period. Most of the games out
> there today are at the cutting-edge and do well compared to the lame crap
> business developers write.

You really *are* the offensive twit you appeared to be on the xp mailing
list. At least *consistency* is some kind of virtue.

--
Chris "electric hedgehog" Dollin

"Phlip" <phlip_cpp@yahoo.com> wrote in message
news:Rvgmd.20851$Rf1.20444@newssvr19.news.prodigy. com...
> Paul Sinnett wrote:

[Paul's message doesn't appear to have made it to my server...]

>
>> Phlip wrote:
>
>
> My untenable premise is a shop builds their engine, builds a game, and
> starts a re-write on the engine. I'm not saying that's rampant (or
> especially bad). I suspect that truly flexible and bug-free code would
> allow
> such an engine to obey the "Software Product Lines" style.

I'm not sure where you're going with this one, but several examples
of engines that underlie multiple games come to mind. LucasArts'
SCUMM engine, for example. The Z-machine that not only ran
all the early Zork games, but was reimplemented, ported to just
about every platform in existance and is ubiquitous in the
interactive fiction camp. There are numerous other examples.
I suspect that the norm is to use an existing engine to crank out
mid-level games quickly. It's only the real high end games where
the intention is to push the state of the art where you really want
to do wholesale revisions to existing engines.

>> > Game development works with very large teams, split by function
>> > into groups with similar abilities.
>>
>> True. And so many game development teams are anything but agile by
>> definition. However, there are still many smaller game developments
>> that could benefit.
>
> I don't understand. Are you saying the teams _must_ split, and that
> therefor
> process improvement must be ... "less" agile?

I don't necessarilly think of this as bad. While I've heard of
XP teams that are larger than about a dozen people, the
"conventional wisdom" seems to put an upper limit on
an effective team that is somewhere around that point.
Beyond that, you need a strategy to coordinate multiple
teams, or operate in some other fashion.

>> > Shipping a game requires an early feature-freeze and then a very
>> > long party to stomp insects.
>>
>> This is usually the policy, but in practice most games are adding
>> features right up to the burn of the final master. It's particularly
>> easy to introduce emergent bugs at this point.
>
> Per the "crunch mode" thread, it seems I tried to describe "crunch mode"
> here. Feature freezes are not bad (it's just a game, not air traffic
> control).
>
> But if scheduling were working, were are the "crunch modes" coming from?
> My
> non-game crunch modes came from not maintaining a zero-tolerance policy
> against even minor bugs, early in development.

IMO, crunch mode usually comes from not having a production
quality deployable at frequent intervals throughout the development
process.

For me, the question simply comes down to:

"Would you rather be able to ship a bug-free product that's missing
a couple of levels that you had on the storyboards in time for Christmas,
or would you rather ship the complete game your designers had in mind
and miss the Christmas shopping window?"

This is simply one of the basic XP questions, with the blanks
filled in from the game development lexicon.

However, the question only makes sense if the process is
capable of producing that production quality deployable
every couple of weeks, with a new level or two, fixes
for previous levels from the play testers, game engine
enhancements and so forth.

>> In my experience, speeding up the feedback loop hasn't improved much.
>> If anything, it led to experimentation without purpose. At
>> best it has had no effect. I once made every predefined gameplay
>> constant editable from a single text file. The designer took that
>> version away to "tweak" the design. When he brought it back the
>> following week he had changed one parameter by 0.01 of a second.
>> Nobody noticed.

The basic issue is: which feedback loop needs tweaking? The
only way you tell that is to find the bottlenecks and work on
them. In this example, it sounds like gameplay engine change
requests weren't a bottleneck.

On the other hand, they may well have been. I can very
easily envision a scenario where the engine people were
being burried under a load of change requests for constants
that didn't actually do anything. Externalizing that would
have removed the change request queue.

John Roth

>
> --
> Phlip
> http://industrialxp.org/community/bi...UserInterfaces
>
>