[TRENCHES] Tuesday, July 3, 2012 - Reversion
LegionPerseus VeilRegistered User, Moderator, Penny Arcade Staff, Vanilla Staff vanilla
Reversion
http://trenchescomic.com/comic/post/reversion1
Performance Metrics
AnonymousA number of years ago, I was part of a testing department at a large dev studio. Management at this time had become somewhat obsessed with performance metrics to the point that they no longer seemed to care if any actual work got done. A particular game, which will go unnamed, had been struggling to make the important milestone of zero bugs in the system. To solve this they simply threw more testers at the game, namely, yours truly.
There must have been some disconnect between the higher ups and my manager because I was thrown on the game and promptly told not to enter any bugs. One should take note that my job description entailed entering bugs, eating lunch, and then a further round of entering bugs. That is to say, they had given me the job of not doing my job.
The ends, to this means, was to artificially achieve zero bug status so that someone somewhere in metrics land would smile and pat themselves on the back.
Any bugs I actually found were slipped directly into a word doc and saved for a rainy day. Among many other smaller bugs, I had discovered about 10 crash bugs and a number of desync’s (I was on the online team) during those two days which, for that stage in development, was a prestigious number. The next day when the milestone had passed (Hazzah!) my writing bugs ban was lifted, I dutifully copy and pasted all the bugs I had found the days before, as well as a number I had found that day, into the system in about a ten minute period.
Apparently, this broke someone’s metrics.
My manager appeared about 5 minutes after that. He tells me that I was supposed to enter those bugs slowly over the next few weeks. I suppose I was supposed to intuit this from him, as I was never given such instructions. Never the less, the blame for doing my job fell squarely on me and it was apparently a bad thing.
The game shipped filled with bugs. I quit shortly thereafter.
Posts
No why are Isaac and Cora refusing to admit that Q spoke to them? I know it makes sure the trap is still there, but it feels a little contrived.
As a veteran of the MMO industry, they are both hiding the fact the a dev came over to discuss the issue with them. It's an issue about not bragging that you are "in" with the dev team. Secondly, it's about waiting for official word. you don;t want to say how things will be, and find out they changed later on.
Trust me, as a person who has worked in the environment it is not contrived at all.
Okay then. Understanding++!
kingworkscreative.com
kingworkscreative.blogspot.com
Then again, Im not in game development or Q A so I could be wrong
Maybe. But that would mean Isaac and Cora have been having a total communication breakdown and talking about different things all this time he thought she was the mole. Which seems possible, but unlikely.
As for metrics... Hoo boy. Yeah. That's one of the reasons experienced test managers are a good thing. They see this shit and have the clout to shut it down for being the pointless exercise it is.
your = belonging to you
their = belonging to them
there = not here
they're = they are
Fixed that for you. :P
Although unrealistic goals are also bad.
It didn't really break the metrics themselves, but it invalidated the data. A sudden spike like that doesn't require much in the way of analysis skills to make someone conclude that the previous milestone was deceptively reached. Hell, if he did that (withheld the reporting) of his own volition, he could have been fired. It's very unethical, but it happens outside of the games industry too.
Also, LOL at 0 bug expectancy in anything of reasonable complexity that wasn't developed under that explicit mathematical model of programming whose name I don't remember that also takes 20 times as long to develop under.
I work in anti-fraud. My department is a back office team, responsible for bulk analysis of data and making decisions on cases that aren't straightfoward and obvious. So far, we've been safe from metrics over the last few years, despite a few ominous rumblings. There's a department below us that handles things more mechanically - they decide if something is fraud or not based on a strict set of guidelines, and independent thought is strongly discouraged. They have metrics.
A fraud department with agents making decisions about whether things are fraud or not based in no small part on individual metrics, rather than what is actually on the case. I love the corporate mindset.
I probably have one of the best programming jobs. I have code I've spent years on and no one's ever looked at it but me. I don't have a real manager because our company is small and structurally flat, and our boss doesn't know what we're up to past whether or not we're delivering on time. Which we do.
I don't feel comfortable checking anything in until I know that someone else who knows what they're talking about has taken a look at it and made me justify the weird shit I've done.
your = belonging to you
their = belonging to them
there = not here
they're = they are