ORIGIN WRITES... BUG STOMPING IN THE PACIFIC

by Bill Armintrout

As I write this, we've just signed off on PACIFIC STRIKE -- it is on its way to the disk duplicators, and we're ready to totter home for a few weeks of sleep.

The last few weeks (and, sometimes, months) of working on a software project are not quite like working on anything else I know of. A game passes through several milestones on its way to being "done" -- ALPHA, which usually means that you've got the basics of the game working; then the BETA stage, when the design team thinks that everything in the game should be working.

Up till now, the game has had a skeleton crew of playtesters. But in BETA, the testers are piled on -- a dozen, maybe two dozen full-time Quality Assurance personnel. Under the direction of their QA leader, the testers play the game in shifts; at any hour, day or night, there is someone at Origin playing the game.

You might think that it would be fun to play games for a living. Well, no. It's NOT fun because you keep finding the bugs!

"Game locked up in elevator flic."

"Was on tail of Jap plane, pulling a tight turn. Suddenly, my plane pitched sharply, the sky turned black, and I lost control of my plane. Was I in orbit?!"

"Shot down Zero, game hung."

"Tried to take off from airfield, but my plane sank through runway and then exploded."

"When Japs sank the battleship, it floated up into the air. Then the game hung."

"Tried to use radio, game crashed to DOS, not enough memory."

The bug reports are entered into a database, the redundant ones are sorted out, and then the Project Leader starts to assign the bugs to his team members.

Some bugs are "killed," meaning that the Project Leader doesn't assign anyone to fix them. At the start of the BETA phase, few bugs are killed; later, when deadline pressure begins to build, more bugs may have to be killed.

What kind of bugs get killed? Here's a good candidate:

"Spoke to the Armorer in the hangar deck. He said 'shall.' I don't think a mechanic would talk like that."

That's a suggestion-type bug. It is easy to fix, but the person who has to fix it may have other (more important) bugs to fix first. Or the writer disagree with the bug, and convince the Project Leader that the wording is right in the first place.

Here's another: "This is supposed to be a morning mission, but the sun is low on the wrong horizon."

Good bug, the kind we fix if we have time...but if we don't fix it, will anybody really care? The crucial thing with bugs is that there are so MANY of them. You have the fix the major ones first. If there's time, the team fixes the minor bugs too.

Some "bugs" turn out to be suggestions for features that are really good ideas, but just aren't in the plans (or budget) for the current game. Other "bugs" may not be fixable within the time allowed for the project, or might even be due to some basic structure of the software (which it is too late to rewrite).


This is what bug phase is like for a team member: Every morning, I sit down at my machine and run the bug-control software to find out which bugs have been assigned to me. Most of the bugs are new; some of the bugs have been transferred to me from other team members; and some are leftovers from yesterday, either because they are low-priority or I'm just not ready to face them yet.

I like to start my day with an easy victory, so I decide to fix one of the new bugs. Bug report reads: "In the briefing in Leyte Gulf mission 4, when the commander points to the map, his hand turns into a giant claw."

I drop into the game, "jumping through the porthole" (our cheat system) to get to the problem mission. The first step is to run the briefing, hoping that the bug might have gone away since it was last reported. Nope! Sure enough, the commander's hand mutates when the arm stretches out. Hmmm...

I compare the data file for that briefing with the original art. There's the problem! The artist assumed the commander would never need to stretch that far, and didn't finish this frame of the animation. A simple art bug. The artist and I confer, and he's off to fix the art. (I keep the bug. In a few hours, he'll have the fix back to me, and I'll drop the new animation frame into the game.)

Time to look into another bug. Rob Irving, our CDA (creative design assistant), has forwarded a bug to me. Seems there's a problem in Guadalcanal mission 3 -- the tester sank the YAMATO, but the debriefing said that he failed. Should be a simple flag bug, right? But Rob says that he checked the mission, and the YAMATO_sunk flag is being set properly. Must be a conversation bug.

So I dive into the debriefing's data file. Uh-oh. No problem here. The script clearly says:

IF(YAMATO_sunk)

SAY(COMMANDER, NEUTRAL, "Good work sinking that ship, son.")

ELSE

SAY(COMMANDER, GRIM, "So the Japs got away...")

END_IF

Well, if the conversation system is right, then the mission must be wrong! (Even Rob is wrong once in a while.) But I play the mission myself, sink the YAMATO -- using the cheats to make it an easy victory (and save myself some time) -- but I get the losing debriefing too.

Well, I'm stumped. So I move on to another bug, hoping I'll be more inspired later...

"I don't think the AI is working," says the next bug report. "When I get on the tail of an enemy plane, he just sits there and let's me shoot him up. Shouldn't he be trying to fight me?"

First of all, I KNOW the bug report isn't correct. I've been playing the game for weeks now, and lots of enemy planes fight back. But the tester is right -- sometimes they don't.

What makes this complicated is that in some cases, the planes aren't supposed to fight back. A plane with a tail gunner, for instance, might figure it doesn't need to dodge. Or a bomber on an attack run might concentrate on its target, rather than the enemy chasing it.

Sounds like I better talk to the tester, and find out exactly what situation he's talking about.

So I run up the stairs to the QA department, hoping that the tester is working on the same schedule I am. Turns out he is. Not only can he make the bug reproduce repeatedly, but he has a savegame I can work with. Terrific!

The exact situation turns out to be this: in Coral Sea mission 2, at the first way point, you get jumped by two float planes (but only if you failed the last mission). The tester has noticed something very important -- the bug only happens if your wing men are out of action!

How can this be? What's setting this up? I know by now that the AI system in our game decides what each plane should do, based upon GOALS in the individual pilot's data files, plus ORDERS given in the mission file for this plane in this mission.

I fly the mission, turning on the full debugging information so that I can "see" what the planes are thinking. Turns out my enemy's GOAL is to "take combat action" (that is, he wants to fight) and his MISSION tells him to attack enemy fighters. So far, so good.

Now I get to the point where the bugs occurs. My wingman, Squints, decides he's taken too much damage and bails out. What happens to my enemy?

The Japanese plane continues to fly as before. In fact, it's as if he's locked up -- flying in a straight line, in a gentle dive that will eventually put him into the sea. What's the debug info say? Goal and mission are unchanged, but his maneuver is listed as NULL. Whoops! There's the bug. But what is causing it?

A few hours and a number of flights later, I finally have my answer. There's a simple error in the "take combat action" routine. Basically, it works like this --

                Are there any enemies around?
                If so --

                        Select a target.
                        Attack the target.

Now here's the problem. This enemy pilot turns out to have been a rookie pilot. The targeting code says that rookie pilots don't easily spot enemies which are directly behind them. So what's happening is that the AI says, "attack the target," but the pilot says, "I don't see anything, so I'm just going to fly in a straight line!"

The correction is trivial. (Most bugs are trivial, once you find them.) This bug was almost certainly a typo in the original code. I rework the code, so that it charts like this --

                Are there any enemies around?
                If so --

                        Select a target,
                        If you have a target --

                                Attack the target.

                        Otherwise --

                            Tell AI that you fail to take a combat action.

The AI is designed to handle this problem, IF the offending code tells it that something is wrong. In this specific case, the enemy rookie pilot will not see me, so it cannot attack, so it looks at its data file for some other goal. The next goal is to "search area for enemies." So the plane now banks around, and if the pilot is lucky, he gets me off his tail and finally sees me.

This bug is an example of the kind of bug that can be hardest to find. First of all, somebody has to notice it. An observant tester who can sniff out this kind of problem is worth his weight in gold. Crashes and hangs are easy to find; misbehaving AI is much more subtle.

Then you've got to fix the bug, and AI is a pretty convoluted realm. The bug might be in any of dozens of places, or in the interaction between those places. Is it the danger code? The maneuver code? The mission priorities code? Each of these areas of code may have a different "owner," a different programmer who last worked on them. (In my case, I'm not a programmer at all, just a designer who knows enough code to help out when things get tight.)

And then you have to be patient. If my luck holds, I've fixed the bug in this specific situation, but tomorrow the testers will find it happening in two other missions -- not really the same bug, but something bafflingly similar.

PROJECT LEADER: "They found your bug in Marianas 2 last night. Thought you fixed it?"

ME: "Look, I fixed it. He had it happen in Guadalcanal 1, and now it never happens."

PROJECT LEADER: "Well, I dunno. Better look into it."

With another victory under my belt, I circle back to that bug I couldn't figure out earlier. A flag bug. Rob says he sets the flag, but the conversation system says it's NOT set.

Maybe it's time to run a manual check. So I go back to my savegame, turn on the flag cheats, and check the individual flag number. Aha! The flag IS set. Hmmm... So why does the conversation system not see the flag?

Loading up the translation table used by the conversation system, I find my first clue. In the game, all flags have numbers rather than names. Looking in the data file, I find --

                YAMATO_sunk     3412

But a few lines below that, something else catches my eye --

                // YAMATO_sunk          3418

The double-slashes mean that the second line has been "commented out" -- the data-file reader should ignore that line. Apparently, the YAMATO_sunk flag used to be flag #3418, but it's been changed to flag #3412.

Suddenly, I have a hunch. I know that the code which allows the conversations to access these flags was "quick and dirty" -- we did the work at the last minute (because we didn't realize we'd need the code!). A few minutes later, my guess proves right: the conversation code doesn't understand about "commented out" lines in the data file! It reads the data file, sees that the YAMATO_sunk flag is #3412, but then sees that it's really #3418 and forgets the first number.

The solution, again, is trivial. I delete the "commented out" line from the flag file. (Surprised? Did you think that I would go in and "fix" the conversation code to recognize commented-out lines? Not with six weeks to go until we ship!!!) AND I add a comment to the flag file, explaining not to comment out any more flag definitions. Then to make sure everyone knows, I send email to the team about the problem.

Three bugs down. Eleven more on my plate, unless the Project Leader has handed out a fresh batch. It's five o'clock. Is it time to order team dinner yet? I'm on twelve-hour shifts, until I can get my bug count down...


I've told you about the bugs I've fixed, because those are the ones which I know the best. But others are fighting different kinds of bugs, some easier to find than others.

WILL McBURNETT, who is doubling as Project Leader and as Lead Programmer, gets all the bugs dealing with mysterious crashes and hangs. Because the bugs themselves provide few clues as to what is happening, he turns his machine into an elaborate testing station -- one computer running the game, and another computer watching him run the game.

He finds a memory trasher, and we think the Big Bug is gone. But the next day, the game still crashes. He fixes another bug, this time in the clipping routines, and we think the game is going to be better. But it gets worse. QA can't even fly a mission today. The game hangs about two seconds into any mission. It's back to the testing station. Turns out that the old bug "masked" another bug, but now it's exposed and killing us.

Finally, Will gets the big crashers. One of the bugs had to do with the sound effects system. The other bug was a bizarre problem in the assembly- language drawing routines, and only occurred when bullets were too close to the camera. Sheesh! But Will got them all. Now we can ship.

ROB IRVING, on the other hand, is deep in the data for each of the missions. He has "created" the virtual landscapes, the timing which dictates when each wave of fighters or bombers comes in, and what instructions their AI is given.

He gets a lot of bugs that really belong to somebody else, but we don't know that until he eliminates data problems as the cause. The bug report might read: "Second wave of enemy bombers turned and ran from me." Is the bug in the mission? No, Rob can't find a problem -- must be an AI bug.

CHARLES CAFFRELLI is our gameflow programmer. He came to our project late, arriving fresh from working on PRIVATEER. For two weeks, the mission editor has been his curse. The testers keep finding new ways to crash the system. He fixes the bugs, but more come at him. Hordes of little bugs. He wishes he'd never heard of the mission editor.

CHRIS PESCI gets no rest, because he is programming the weapon systems, and that's what everybody gets upset easiest about. He fixes things like torpedoes running onto land, or wing men who strafe the battleship instead of dropping their bombs on it, or how to tell the AI to stop making the kamikazes pull up before they impact on their targets. (The AI, you see, thinks that planes should avoid colliding into things. That's mostly true. But not always.)

DAVE BRANDT is a numbers tweaker. Before BETA, he was Data Man -- importing the ships, planes, and ground objects into the game; setting damage points for bombs and rockets; making the damage system work so that the bullet holes appear in your windscreen when you take damage. But now, his job is to make the plane dynamics feel right.

It's not a science, it's an art. Sworn to secrecy, a few very privileged blind testers who also happen to be experienced pilots are putting our game to the test. One might report, "the Devastator is too sluggish at such-an -such altitude." So Dave massages his data. I don't even know what kind of data he has, but some of his data points can be based on absolute facts (we know what the horsepower of that plane was) but others are not so precisely quantifiable.

So Dave works until everyone agrees that the planes feel right. Then he tests the game some more. He's one of the first team members to "get a play-through" -- that is, to play the game all the way through, all the missions, without cheating.


The bug phase goes on and on. At first, you can fix as many bugs as you get each day. Then the team starts to fall behind. Every day, we watch to see who has the most bugs. (Poor Will -- he's got 218 bugs, and they are all mysterious crashes and hangs!)

Then, finally, the crisis seems to sort itself out. We're down to 30 or so bugs, but half of these are tough ones -- multi-day bugs, maybe even week- long bug fixes. Most of the bugs ride the same critical shoulders (Will and Chris); Rob has a lot of bugs too, but his kind of bugs are quicker to fix.

Then we're down to a dozen bugs, and we think we'll "sign off" by Friday. But we don't -- we can't find that crash bug. Maybe by next Friday.

But we don't make it that Friday. QA finds a whole crop of new bugs. Why weren't these found earlier? We can't ship with these bugs! The team is tense, frustrated, ready to finish this project.

And then, suddenly, on a Thursday, the game is done. Key project members sign the documents that say we certify the game is ready to go. The master disks are sent to the factories.

And that's what bug-stomping is like.


Bill Armintrout is a game designer for Origin Systems, Inc. He was the lead designer for PACIFIC STRIKE and, before that, Ultima VII, Part II: THE SERPENT ISLE.