Playtesters are biased

As a game designer you playtest your game while it’s in development and after it is released. You get people to play your product and figure out specifics; is the balance correct, are there exploits, what do they like least and how can we improve that? Playtest feedback is invaluable, but it is definitely not without its pitfalls…

The problem was obvious, the solution was not

We were working on Killzone 2 Online Multiplayer, developing patches after launch and we were made aware of a specific balance issue with one of the weapons – The rocket launcher. Players on forums were loudly letting us know the rocket launcher was too powerful. Too much ‘splash’ damage, making the weapon easy to use for any ‘newbie’ and get kills.

We were internally exploring several options:

• Lower the availability of the rocket launcher ammo in levels
• Lower the overall damage of a rocket launcher grenade
• Lower the range of splash damage of a rocket launcher impact
• Tweak the drop-off of the splash damage range

Option one, lowering the availability, was costly. Removing ammo pickups from levels would mean patching the level content, which (due to how our patching worked back then) meant the patch was going to be big to download. Not preferred.

Lowering the overall damage worked, but it made the weapon feel less like an actual rocket launcher. In Killzone, the feel of the weapons was really important and I wasn’t about to let our core business lose out to balance.

Lowering the range of damage is very tricky to explain to a player. They see an explosion of a specific size and expect a specific range. It feels weird if the damage range is shorter than the explosion – The expectation is a large range. We could change the explosion effects but it made the weapon just feel awkward.

So the only option we were left with was playing with the drop-off of the damage – Essentially making the weapon have a big impact where it hit but make the damage drop quite quickly and then having it do less damage over the remainder of the range. This meant tweaking a specific curve of the weapon, which was a separate file in a separate folder…

Let’s ship a patch!

All game studios use some version of revision history – A system where every file that is edited is ‘checked in’ and tracked, so if it breaks something, we can revert to the previous version and find out who was the culprit. The rocket launcher change was made and checked into the project. Everyone was using the new version internally now. 

For release versions of the game, studios usually make a separate ‘branch’ of the project. This enables developers to keep working on stuff not part of a specific version, while not breaking the next release version. Basically it allows us to pick and choose what goes into each version.

I flagged my rocket launcher curve file to be part of the next patch. I really did. A patch was made and tested by Sony testers before release. They all said the rocket launcher change was awesome. They really did…

The release was a success!

The forums loved the changes! Our biggest rocket launcher critics were all saying it was now “so much more balanced” and people praised us for not ‘nerfing’ it entirely in the process – It was still a useful weapon. I was really happy my solution worked!

And then… I noticed I had flagged the wrong curve file to be included in the patch. I flagged a local test curve that wasn’t connected to the actual rocket launcher at all. I hadn’t changed a thing. The new live version of the game in terms of the rocket launcher was unchanged…

I was shocked! But the audience was so happy, I must have changed something right? I researched it for hours but I was baffled – Nothing was different.

What can we learn from this?

We never changed the rocket launcher again – People were happy, why should we change that? We told them we fixed something and they believed it. But how is that even possible?

People influence each other. When people read posts from loud people saying the rocket launcher is ‘overpowered’ they get influenced by that. Heck, even I got influenced by it. I looked at the data and saw that it was overpowered, right? Truth is, we’re all being influenced by the opinions of others.

And when we told them we fixed it, they believed it. They took it as a truth and then saw it for themselves. We influenced them back, by accident. Total placebo effect!

After that day I had people double-check if all the correct files made it into patches, but more importantly, I made more of a pro-active effort to find facts to back up opinions before I changed anything…

Want to learn more about balancing games after release? Book a 1on1 session!