Trenches comic: Thursday January 12, 2012 - Phantasm

Bogey · January 2012

It’s Not (Always) Our Fault

01/12/2012 - Anonymous

I worked QA on a AAA title for a year, my contract expiring shortly after the game shipped. Upon the game’s release, the official forums were alive with people complaining about bugs and technical issues. A common phrase was, “Didn’t they QA this game?”

What the average gamer doesn’t understand is that a bug isn’t necessarily in a game because a tester couldn’t find it. Chances are, if you were able to find a bug, QA found it first. That’s their job. But once QA files a bug, it’s not magically fixed. Someone—a designer, or artist, or programmer, or whoever—has to look at that bug and find a way to fix it without breaking anything else in the process.

We, as QA, file every bug we find, period. If a bug can’t be fixed due to time restraints, technical limitations, design choices, laziness or stupidity, it’s out of our hands.

So cut us some slack. We have enough problems to deal with as it is.

RMS Oceanic · January 2012

The third panel feels really superfluous. Or maybe the second panel, I dunno.

Arakon · January 2012

Now I just wish more people would read and believe the story. I lost count of how many times our team filed bugs and got a "debug related", "no bug" or "intended" back, and within 30 seconds of the game release, someone writes in a forum about that very bug. Or sending a "please check if this behavior is intended" bug and never getting a response, only for someone to realize in the last 2 hours of the testing phase that the behavior is not intentional and they have to do another version before the release, possibly even including an angry email asking why this wasn't reported.

oldtaku · January 2012

Well honestly, when we say 'Didn't they do any QA on this?' that includes actually paying attention to the bugs filed.

I'm well aware that QA can find and file bugs all day long, especially when you get bored and start doing really weird stuff.

The 'no QA' complaint is always a question of whether the dev or publisher bothered to do QA at all or whether they did QA but didn't bother to fix any bugs. I have never seen anyone speculate that the QA people themselves are the weak link, so you shouldn't take it as an insult.

Juc · January 2012

I think something you need to develop in QA, especially if you are going to listen to people comment about the game post-release, is a very thick skin.

Customers are fairly naive about game development, and you probably shouldn't expect them to know the ins and outs.
Also some developers are equally naive, for some reason, and need to be educated on when was a bug found, the contents of the filed issue and why it wasn't fixed. Provided it wasn't a mistake by QA the developers are generally less dickish when they know the circumstances of why people are complaining post release about an issue.
But also some times it really is QAs fault, and in those instances it's more useful to see how the problem slipped through the cracks, and fixing the procedures that allowed it to happen, than it is to get upset over it.

QA is a very bad profession for people who are emotionally fragile.

AnteCantelope · January 2012

Juc wrote:

I think something you need to develop in QA, especially if you are going to listen to people comment about the game post-release, is a very thick skin.

Customers are fairly naive about game development, and you probably shouldn't expect them to know the ins and outs.
Also some developers are equally naive, for some reason, and need to be educated on when was a bug found, the contents of the filed issue and why it wasn't fixed. Provided it wasn't a mistake by QA the developers are generally less dickish when they know the circumstances of why people are complaining post release about an issue.
But also some times it really is QAs fault, and in those instances it's more useful to see how the problem slipped through the cracks, and fixing the procedures that allowed it to happen, than it is to get upset over it.

QA is a very bad profession for people who are emotionally fragile.

I think I agree with this. When I complain about bad QA, I'm not complaining about bad individual testers who couldn't catch the bugs, I'm complaining about a bad QA process that either didn't fix bugs that it caught, or didn't spend the time and money to catch and fix everything it could. I also don't expect the average person to know the details of development, and I really doubt that anyone who doesn't care to know about the details would be reading Trenches anyway.

Basically, when people complain about bad QA they're complaining about bad management of QA, or they're saying the game is buggy and don't know who specifically to blame. Either way, it shouldn't be too hard for testers to deal with that and move on, nobody is pointing at them individually, and while there are a bunch of crappy aspects to QA I really doubt that this qualifies as one of them.

Also, I could swear I read this exact same tale a month or two ago.

Dark Raven X · January 2012

RMS Oceanic wrote:

The third panel feels really superfluous. Or maybe the second panel, I dunno.

Naw, I disagree fella! As "character is going nuts and weird things are happening" strips go, this is perfect.

Unless that rabbit can talk...

ninja.iceberg · January 2012

Working at a huge gaming company, I've run into my share of waived bugs. After the first few that come back as As Designed, Known Shippable, you kinda just stop caring. You remind yourself that your job is to find and report bugs. That's it. You can fight for it, talk to your lead, or at my company, talk to a producer/developer. But once it's in the database, it's pretty much out of your hands. And you have to let it go. That's how you keep sane. Or you can repro the bug in front of the press when they have a press day on campus.

Arakon · January 2012

Of course it can. Or do you want to say you don't know any talking rabbits?

Jormungandr · January 2012

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

Jacques L'Homme · January 2012

One thing, Isaac could have turned his head just a smidgen to acknowledge Mr. Toots arrival.

Nit picking aside, this is my favorite thus far!

kingworks · January 2012

Quality Assurance is everyone's job.

I think I read that somewhere.

If not, ©kingworks 2012

Tube · January 2012

Jormungandr wrote:

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

I annoyed a guy who worked on Fable 3 by blaming QA for that game. I don't know what I was expecting them to report to be honest. "Game Is Shit, Please Try Harder".

El Skid · January 2012

Lewie P's Stepdad wrote:

Jormungandr wrote:

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

I annoyed a guy who worked on Fable 3 by blaming QA for that game. I don't know what I was expecting them to report to be honest. "Game Is Shit, Please Try Harder".

Yeah, being in QA and working on a well-executed project is amazing. You spend way less time looking at "This shit is broken", slightly more time with "This shit works but it's not intuitive- should probably work a different way", and alot more time on "It'd be nice if I could do this as well". Plus you tend to not need to work 72 hours straight so much, because you're making deadlines and are able to actually...you know...execute your scripts.

Working on a project that isn't going well is more...crappy. You spend more time looking at situations where "I can't do X Y and Z scripts because P is still broken" and "Basic functionality still not working, test failed AGAIN", along with my favorite "Someone has screwed up the code again and it's not loading at all, so I'll just sit here and twiddle my thumbs while my deadlines march ever closer."

I'm guessing fable 3 trended towards the latter, so blaming QA is likely not entirely valid. Plus if you suffered through something resembling the latter scenario and then got blamed for the crappiness of the game, I could see getting a bit upset.

emskadittle · January 2012

The pressure to release is pretty intense at the end and lot of bugs are just ignored or marked to be fixed later.
but I have come to the understanding that if it can be done then our customers will do it, no matter how stupid the flow seems

Warlock82 · January 2012

AnteCantelope wrote:

Juc wrote:

I think something you need to develop in QA, especially if you are going to listen to people comment about the game post-release, is a very thick skin.

Customers are fairly naive about game development, and you probably shouldn't expect them to know the ins and outs.
Also some developers are equally naive, for some reason, and need to be educated on when was a bug found, the contents of the filed issue and why it wasn't fixed. Provided it wasn't a mistake by QA the developers are generally less dickish when they know the circumstances of why people are complaining post release about an issue.
But also some times it really is QAs fault, and in those instances it's more useful to see how the problem slipped through the cracks, and fixing the procedures that allowed it to happen, than it is to get upset over it.

QA is a very bad profession for people who are emotionally fragile.

I think I agree with this. When I complain about bad QA, I'm not complaining about bad individual testers who couldn't catch the bugs, I'm complaining about a bad QA process that either didn't fix bugs that it caught, or didn't spend the time and money to catch and fix everything it could. I also don't expect the average person to know the details of development, and I really doubt that anyone who doesn't care to know about the details would be reading Trenches anyway.

Basically, when people complain about bad QA they're complaining about bad management of QA, or they're saying the game is buggy and don't know who specifically to blame. Either way, it shouldn't be too hard for testers to deal with that and move on, nobody is pointing at them individually, and while there are a bunch of crappy aspects to QA I really doubt that this qualifies as one of them.

Also, I could swear I read this exact same tale a month or two ago.

This exactly. For me it's always either "there is no QA" or "developers are ignoring QA bug reports." I don't think people generally believe QA is incompetant, especially when the games are buggy enough that it's hard to believe some of the bugs were not at least caught/discovered.

Also I frequently say this about Ubisoft games :P I swear some of the buggiest games (at least compared to their otherwise notoriety). Maybe Bethesda too but honestly I find it hard to complain too much because the scope of their games is always insane.

Fatty McBeardo · January 2012

Toots is a snappy dresser!

Delzhand · January 2012

Lewie P's Stepdad wrote:

Jormungandr wrote:

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

I annoyed a guy who worked on Fable 3 by blaming QA for that game. I don't know what I was expecting them to report to be honest. "Game Is Shit, Please Try Harder".

"Could not find files fun.inc, enjoyment.lua, awesome.gcc"

PS - your sig link is borked

vardis · January 2012

Whenever I complain about "QA", I pretty much always am blaming the programmer/developer, though I suppose it may not sound like it. I guess I figured it sounded less harsh to complain about the "lack of QA" than to say, "are these programmers morons to have missed this?"

admanb · January 2012

Delzhand wrote:

Lewie P's Stepdad wrote:

Jormungandr wrote:

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

I annoyed a guy who worked on Fable 3 by blaming QA for that game. I don't know what I was expecting them to report to be honest. "Game Is Shit, Please Try Harder".

"Could not find files fun.inc, enjoyment.lua, awesome.gcc"

PS - your sig link is borked

"I wrote up a report about getting trapped in a bad ending and Peter Molyneux came into my cubicle and peed on me."

Henroid · January 2012

admanb wrote:

Delzhand wrote:

Lewie P's Stepdad wrote:

Jormungandr wrote:

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

I annoyed a guy who worked on Fable 3 by blaming QA for that game. I don't know what I was expecting them to report to be honest. "Game Is Shit, Please Try Harder".

"Could not find files fun.inc, enjoyment.lua, awesome.gcc"

PS - your sig link is borked

"I wrote up a report about getting trapped in a bad ending and Peter Molyneux came into my cubicle and peed on me."

It worked for Will Wright.

Aldo · January 2012

Henroid wrote:

admanb wrote:

Delzhand wrote:

Lewie P's Stepdad wrote:

Jormungandr wrote:

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

I annoyed a guy who worked on Fable 3 by blaming QA for that game. I don't know what I was expecting them to report to be honest. "Game Is Shit, Please Try Harder".

"Could not find files fun.inc, enjoyment.lua, awesome.gcc"

PS - your sig link is borked

"I wrote up a report about getting trapped in a bad ending and Peter Molyneux came into my cubicle and peed on me."

It worked for Will Wright.

What worked? Getting peed on, working for Peter Molyneux or coming in admanb's cubicle?

Aioua · January 2012

Aldo wrote:

Henroid wrote:

admanb wrote:

Delzhand wrote:

Lewie P's Stepdad wrote:

Jormungandr wrote:

Most complaints I've seen on forums, etc, for buggy games do blame "QA" pretty broadly. Or they'll say "I can't believe nobody was able to find this bug before it shipped." and the like. Whether their intention is to blame QA as a general process rather than individual QA testers, I can't say, but I can certainly see where it would irritate the fuck out of someone who had, in fact, reported all of the bugs months ago that people are finding on release.

I annoyed a guy who worked on Fable 3 by blaming QA for that game. I don't know what I was expecting them to report to be honest. "Game Is Shit, Please Try Harder".

"Could not find files fun.inc, enjoyment.lua, awesome.gcc"

PS - your sig link is borked

"I wrote up a report about getting trapped in a bad ending and Peter Molyneux came into my cubicle and peed on me."

It worked for Will Wright.

What worked? Getting peed on, working for Peter Molyneux or coming in admanb's cubicle?

pssst...

Casual · January 2012

oldtaku wrote:

Well honestly, when we say 'Didn't they do any QA on this?' that includes actually paying attention to the bugs filed.

I'm well aware that QA can find and file bugs all day long, especially when you get bored and start doing really weird stuff.

The 'no QA' complaint is always a question of whether the dev or publisher bothered to do QA at all or whether they did QA but didn't bother to fix any bugs. I have never seen anyone speculate that the QA people themselves are the weak link, so you shouldn't take it as an insult.

Having frequently moaned bitterly about "terribad QA" in games I have to say this is my stance too. I don't think I've ever come across a stupidly obvious bug in a game and thought "wow those testers are idiots for not finding this". I assume the truth, that the company knew about the bug and decided to say "fuck it, ship it anyway", and it's exactly this attitude that enrages gamers so much, far more than the idea that a bug was simply never found. So while I feel for testers trying to do good by their companies customers, we're not wrong for blaming these failings on QA, since finding the bugs is only half the QA process, you have to fix them too. :P

Aurich · January 2012

It could also be a matter of poor bug prioritizing. For example: I'd be perfectly fine with TOR crashing to desktop now and then, if they would just fix whatever's making my Jedi's robe pop out like a Victorian ball gown.

Juc · January 2012

Aurich wrote:

It could also be a matter of poor bug prioritizing. For example: I'd be perfectly fine with TOR crashing to desktop now and then, if they would just fix whatever's making my Jedi's robe pop out like a Victorian ball gown.

I suspect your Jedi just really enjoys the force ... if you know what I mean.

Ivar · January 2012

Force fart

admanb · January 2012

Juc wrote:

Aurich wrote:

It could also be a matter of poor bug prioritizing. For example: I'd be perfectly fine with TOR crashing to desktop now and then, if they would just fix whatever's making my Jedi's robe pop out like a Victorian ball gown.

I suspect your Jedi just really enjoys the force ... if you know what I mean.

Consult a medical professional if your Jedi robe pops out like a Victorian ball gown for more than 24 hours.

Penny Arcade

Quick Links

Trenches comic: Thursday January 12, 2012 - Phantasm

Posts