As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

Trenches comic: Thursday Dec. 1, 2011

BogeyBogey I'm back, baby!Santa Monica, CAModerator mod
edited December 2011 in The Penny Arcade Hub
i-XK7DNgX.jpg
Untitled.

12/01/2011 - Anonymous

“Has anyone tried maxing out all 4 players’ coins and killing them all simultaneously?”

The team that I’d just joined shook their heads.

“Not yet. Dude, do you have any idea how long that would take?”

We were testing a newly-added multi-player game mode where players collected coins, and—should they die—dumped those coins into the environment for other players to steal. I’d just been added to the team, but the rest had been pulling 12+ hour shifts for months now. They were tired, ground down by the monotony, out of ideas.

“Do we have something better to do?”

It was 10 AM. We started right away.

We spent the next 12 hours collecting coins on 3 characters, getting two players up to 9999 coins and a 3rd up to roughly 7000 before the tedium overwhelmed us. We placed those three characters together and, using the 4th character as a “camera,” we set off a bomb that killed all three on the screen.

A deluge of coins dumped into the environment non-stop for 20 or 30 seconds. At first, everything seemed okay. But then, the frame-rates began to dip, then the screen began to visibly chug, then the game froze momentarily, and then… blue screen. A hard lockup.

We roared a cheer and began to congratulate each other..

“What did you do?!?”

The assistant coordinator approached us from his desk and we explained what we had done and how long it had taken to set up.

“Alright,” he said, turning, already heading to sit back down at his desk, “Reproduce it.”

Fitocracy: Join us in the SE++ group!
XBox LIVE: Bogestrom | Destiny
PSN: Bogestrom
Bogey on

Posts

  • Options
    AthenorAthenor Battle Hardened Optimist The Skies of HiigaraRegistered User regular
    edited December 2011
    ... Ouch.

    Isn't this another case where you can automate or debug set up characters to have that many coins? Save yourself some time?

    Athenor on
    He/Him | "A boat is always safest in the harbor, but that’s not why we build boats." | "If you run, you gain one. If you move forward, you gain two." - Suletta Mercury, G-Witch
  • Options
    Man in the MistsMan in the Mists Registered User regular
    So, did they ever find the assistant coordinator's body?

  • Options
    IvarIvar Oslo, NorwayRegistered User regular
    So, did they ever find the assistant coordinator's body?

    Yes, buried under 26998 coins.

  • Options
    PenumbraPenumbra Registered User regular
    The comic was good.

    The story was a thing of blue-screened beauty.

    Switch Friend Code: 6359-7575-9391
  • Options
    emskadittleemskadittle QA Lead Registered User regular
    This is why they have developer codes, or they would have automated this

    I am not criticising your work, I am testing it!
  • Options
    DelzhandDelzhand Hard to miss. Registered User regular
    Ivar wrote:
    So, did they ever find the assistant coordinator's body?

    Yes, buried under 26998 coins.

    http://www.burymewithmymoney.com

  • Options
    El GuacoEl Guaco Registered User regular
    I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.

    I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

  • Options
    rockmonkeyrockmonkey Little RockRegistered User regular
    El Guaco wrote:
    I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.
    I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

    Not 10k, closer to 1k. And actually if it took 12 hours both times for all 5 people it would be 1,200.00

    NEWrockzomb80.jpg
  • Options
    IncenjucarIncenjucar VChatter Seattle, WARegistered User regular
    rockmonkey wrote:
    El Guaco wrote:
    I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.
    I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

    Not 10k, closer to 1k. And actually if it took 12 hours both times for all 5 people it would be 1,200.00

    Probably more like 2K, once you factor in taxes and so forth, but still, a terrible waste of money for a pretty standard hook. That said, some testers only make minimum wage.

  • Options
    MortalToasterMortalToaster Registered User regular
    El Guaco wrote:
    I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.

    I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

    Adding in cheat codes that work properly and don't otherwise mess up the normal play of the game would require more QA, so you have to take the cost benefit of do I spend $1200 to code in a cheat, QA it, and then another $50 to do the testing, or do I just let them test without the cheat code for only $1200. Sure, the cost-benefit may break out for some bigger/harder to reproduce bugs, but I imagine for a lot of bugs it just isn't worth it.

    Not to mention that you have to take those codes out/turn them off before you ship, which requires more QA (and thus more money).

  • Options
    admanbadmanb unionize your workplace Seattle, WARegistered User regular
    Not cheat codes, but a developer console. They've been a natural part of development and QA since the dawn of time, and except for the weird cases like Id or Bethesda, they are never intended to make it into the final product.

  • Options
    KochikensKochikens Registered User regular
    “Do we have something better to do?”

    Sounds like it wasn't a waste of time because they apparently had nothing better to do anyways.

  • Options
    Warlock82Warlock82 Never pet a burning dog Registered User regular
    El Guaco wrote:
    I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.

    I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

    Adding in cheat codes that work properly and don't otherwise mess up the normal play of the game would require more QA, so you have to take the cost benefit of do I spend $1200 to code in a cheat, QA it, and then another $50 to do the testing, or do I just let them test without the cheat code for only $1200. Sure, the cost-benefit may break out for some bigger/harder to reproduce bugs, but I imagine for a lot of bugs it just isn't worth it.

    Not to mention that you have to take those codes out/turn them off before you ship, which requires more QA (and thus more money).

    I really don't see that breaking things. I mean, I suppose if things are coded poorly, maybe. But ultimately it's probably just an easily-modified counter. Unless the game does a whole ton of other shit when you collect a coin besides just incrementing the counter, it shouldn't be hard to do that programmatically.

    Switch: 2143-7130-1359 | 3DS: 4983-4927-6699 | Steam: warlock82 | PSN: Warlock2282
  • Options
    PorkfryPorkfry Registered User regular
    That's such an edge case I wouldn't even worry about testing something like that. Sure its a test, but not a valuable test considering the setup required in addition to the chance that would actually happen in the real world. I'd think maybe .001% of the population would actually spend 12 hours grinding through to max out the coins just to see what would happen. Past that, the end user isn't normally going to enter that state so maybe running some boundary cases where you could get a large ammount of coins (in a reasonalbe amount of time based off of normal game play) and see if that situation (all dying at the same time) resulted in the same performance hit.

  • Options
    Zazu YenZazu Yen Senior Developer San FranciscoRegistered User regular

    What the assistant coordinator said was nothing compared to what the developers would have said, they would have said something along the lines of, "Reproduce it until you can tell us just how many coins it takes to cause the crash."

    ExistentialExistenceException: Your thread encountered a NULL pointer and entered a state of non-being.
  • Options
    Zazu YenZazu Yen Senior Developer San FranciscoRegistered User regular
    edited December 2011
    Yes a game like that NEEDS a dev console. Although dev consoles can introduce bugs, or hide bugs by allowing testers shortcut past things. I knew of a bug in a game where if you left a door open in one of the starting levels, then went back to the level near the end of the game, when all the levels were opened, the door would start closed (it's default position) but could not be opened because the saved data said it was already open. No one found it because they tested the end game by skipping to it with the dev codes so the door was always closed. That's just a small sample. But just adding coins to a player shouldn't be a big issue.

    Zazu Yen on
    ExistentialExistenceException: Your thread encountered a NULL pointer and entered a state of non-being.
  • Options
    fortyforty Registered User regular
    edited December 2011
    I want to know what game that even is. Why didn't it end within 12 hours? What's the victory condition in this coin mode if getting 9999 doesn't win it for you?

    forty on
  • Options
    AurichAurich ArizonaRegistered User regular
    Porkfry wrote:
    That's such an edge case I wouldn't even worry about testing something like that. Sure its a test, but not a valuable test considering the setup required in addition to the chance that would actually happen in the real world. I'd think maybe .001% of the population would actually spend 12 hours grinding through to max out the coins just to see what would happen. Past that, the end user isn't normally going to enter that state so maybe running some boundary cases where you could get a large ammount of coins (in a reasonalbe amount of time based off of normal game play) and see if that situation (all dying at the same time) resulted in the same performance hit.

    I think that you're severely underestimating how much people like doing shit like that in games. That said, I think a player actually would use a mod or a cheat to pull it off, a la http://www.youtube.com/watch?v=llCJg5LiKo8

  • Options
    jothkijothki Registered User regular
    You really don't want a hard lockup on a consumer product if you can avoid it. Some sort of sanity check if you're going to be doing too much at once can't hurt.

  • Options
    Warlock82Warlock82 Never pet a burning dog Registered User regular
    I would think putting in some kind of limit on the number of coins that a player can drop would solve the issue. Or really, is it necessary to even allow them to pick up 9999 to begin with? That seems really excessive. Even Mario rolled over at 100 ;)

    Switch: 2143-7130-1359 | 3DS: 4983-4927-6699 | Steam: warlock82 | PSN: Warlock2282
  • Options
    Kane Red RobeKane Red Robe Master of Magic ArcanusRegistered User regular
    Did this game somehow never offer coins in larger denominations than 1? If not just implement a x1000 coin for these sort of ridiculous scenarios and move on.

  • Options
    GalacticGalactic Registered User regular
    Are they still going on about video game doors in this strip?

    ^_-;

    Too Galactic for you.
  • Options
    fortyforty Registered User regular
    I assume this was discussed/answered in a previous thread, but what the fuck is the QA lead guy wearing?

  • Options
    Jam WarriorJam Warrior Registered User regular
  • Options
    SwashbucklerXXSwashbucklerXX Swashbucklin' Canuck Registered User regular
    edited December 2011
    A kilt. Edit: Yeah, that.

    SwashbucklerXX on
    Want to find me on a gaming service? I'm SwashbucklerXX everywhere.
  • Options
    KreutzKreutz Blackwater Park, IARegistered User regular
    Those are some hellishly tight socks on the kilt guy.

  • Options
    fortyforty Registered User regular
    See, I figured it was supposed to be a kilt, but I didn't understand why it looked like it was cut into big strips. But I guess you can actually see some tan color between them, so it just must be the folds.

  • Options
    CroakerBCCroakerBC TorontoRegistered User regular
    Porkfry wrote:
    That's such an edge case I wouldn't even worry about testing something like that. Sure its a test, but not a valuable test considering the setup required in addition to the chance that would actually happen in the real world. I'd think maybe .001% of the population would actually spend 12 hours grinding through to max out the coins just to see what would happen. Past that, the end user isn't normally going to enter that state so maybe running some boundary cases where you could get a large ammount of coins (in a reasonalbe amount of time based off of normal game play) and see if that situation (all dying at the same time) resulted in the same performance hit.

    I tend to agree. Here, that would come under the rare category of a high severity, low priority defect; it's really serious, as it causes a hard lock up and kills the user experience. On the other hand, the reproduction path is so ridiculous that only people *looking* to crash the game are going to find it and get their game crashed. SO it would go on the backlog.

    That said, not working game QA, I'm finding some of these stories a bit bizarre - do people really have the spare time to spend on setting up an edge case like this? I suspect if I set my team doing something similar for a day, I would receive several angry emails from a righteously peeved PM.

    Is this amount of dead time common over in the games industry? Seems like we've had a few stories about it...

  • Options
    OptyOpty Registered User regular
    It's not so much dead time as they're probably on a near final build and aren't finding any bugs and are desperate to do something they haven't tried yet to not only break up the tedium but potentially find a bug. They potentially could have found many bugs on the road to creating this one so it wasn't like all the time was wasted. Sometimes some really important bugs are discovered due to that yearning to try something different.

  • Options
    JPandarelliJPandarelli Registered User new member
    edited December 2011
    RE: Untitled

    I can’t say what this task was hoping to accomplish, I can only assume they were attempting to test the graphical limitations.

    They were asked to test dropping 9999x4 coins on the screen.

    I can tell you right now that you don’t need to get 9999x4 to test performance issues. I’d bet dollars to coins that you could have accomplished that crash with just a few hundred coins.

    If the test case was to SPECIFICALLY test WHAT would happen if 9999x4 coins are dropped at once, this team failed miserably. In fact, they didn’t even kill all 4 players, they only killed 3 of them and the 4 was standing off to the side. I can only assume it was the guy with 7000 coins and was probably napping most of the time. If I were to go to my producer and tell him, well, we got 9999x3, you’re damn sure he’d tell me to go back and do it again the right f-ing way. I don’t about you but here we don’t get paid to half-ass our tasks.

    So incompetent testers aside, there are a myriad of methods you unimaginative peons could have used to save yourself some time.

    Savant Way
    Debug the coins to yourself. I don’t know what kind of sadistic programmer wouldn’t have a debug command to give coins.

    Smart Way
    Create a save file. This seems like the most basic step. Make a save file you can reload. You know, so you don’t have to spend another 12 hours reproducing nothing. Load the file with you maxed out, have everyone die, did you crash? Yes.

    Ballsy Way
    Ask the programmers to add a test creature that drops 9999 coins. Trust me, they can do it in about 5 minutes.

    Ghetto Way
    Grab a rubber band. Wrap up that controller and go to lunch. Come back and save yourself some time.

    Spiteful Way
    Make one person do the tedium and copy the characters.

    It’s no wonder that this team has been pulling 12 hour shifts for “months” if they’re off performing meaningless tasks like this. Look, I know QA is all about performing monotonous tasks but what separates the kings from the peasants is the ability to perform tasks quickly and efficiently.

    Achievement Unlocked: Waste combined total of 96 hours to perform a task incorrectly
    Reward: Pink Slip

    JPandarelli on
  • Options
    OptyOpty Registered User regular
    They weren't tasked to do it, they decided to do it. Generally games testing isn't limited by following test cases and a test plan due to how many working components they have so they're given a generic "test multiplayer" sort of task and they decide what that means and how to do it.

  • Options
    AurichAurich ArizonaRegistered User regular
    RE: Untitled

    I can’t say what this task was hoping to accomplish, I can only assume they were attempting to test the graphical limitations.

    They were asked to test dropping 9999x4 coins on the screen.

    I can tell you right now that you don’t need to get 9999x4 to test performance issues. I’d bet dollars to coins that you could have accomplished that crash with just a few hundred coins.

    If the test case was to SPECIFICALLY test WHAT would happen if 9999x4 coins are dropped at once, this team failed miserably. In fact, they didn’t even kill all 4 players, they only killed 3 of them and the 4 was standing off to the side. I can only assume it was the guy with 7000 coins and was probably napping most of the time. If I were to go to my producer and tell him, well, we got 9999x3, you’re damn sure he’d tell me to go back and do it again the right f-ing way. I don’t about you but here we don’t get paid to half-ass our tasks.

    So incompetent testers aside, there are a myriad of methods you unimaginative peons could have used to save yourself some time.

    Savant Way
    Debug the coins to yourself. I don’t know what kind of sadistic programmer wouldn’t have a debug command to give coins.

    Smart Way
    Create a save file. This seems like the most basic step. Make a save file you can reload. You know, so you don’t have to spend another 12 hours reproducing nothing. Load the file with you maxed out, have everyone die, did you crash? Yes.

    Ballsy Way
    Ask the programmers to add a test creature that drops 9999 coins. Trust me, they can do it in about 5 minutes.

    Ghetto Way
    Grab a rubber band. Wrap up that controller and go to lunch. Come back and save yourself some time.

    Spiteful Way
    Make one person do the tedium and copy the characters.

    It’s no wonder that this team has been pulling 12 hour shifts for “months” if they’re off performing meaningless tasks like this. Look, I know QA is all about performing monotonous tasks but what separates the kings from the peasants is the ability to perform tasks quickly and efficiently.

    Achievement Unlocked: Waste combined total of 96 hours to perform a task incorrectly
    Reward: Pink Slip

    I hadn't seen any really silly gooses on this forum 'til now. I guess it's nice to know there's at least one.

    Simply, addressing the writer's story in any of the ways you described would have resulted in no story. He may have actually done one of the things you describe, and made up the story anyway, 'cause people do that. Probably, there are specifics of the story he left out, because who cares?

  • Options
    fortyforty Registered User regular
    edited December 2011
    Like someone said in a prior comic thread, the extreme anonymity of these stories (90% of the time we don't even know the game -- or type of game -- they're talking about) turns most of them into Cool Stories Bro.

    forty on
  • Options
    El GuacoEl Guaco Registered User regular
    forty wrote:
    Like someone said in a prior comic thread, the extreme anonymity of these stories (90% of the time we don't even know the game -- or type of game -- they're talking about) turns most of them into Cool Stories Bro.

    Do people fear reprisals? Or is it just not important to the story? Trash talking past employers is always bad form. Exceptions might be made for specific heinous deeds that can be backed up with facts and you still managed to take the high road in the situation.




  • Options
    Zazu YenZazu Yen Senior Developer San FranciscoRegistered User regular
    Aurich wrote:
    RE: Untitled

    I can’t say what this task was hoping to accomplish, I can only assume they were attempting to test the graphical limitations.

    They were asked to test dropping 9999x4 coins on the screen.

    I can tell you right now that you don’t need to get 9999x4 to test performance issues. I’d bet dollars to coins that you could have accomplished that crash with just a few hundred coins.

    If the test case was to SPECIFICALLY test WHAT would happen if 9999x4 coins are dropped at once, this team failed miserably. In fact, they didn’t even kill all 4 players, they only killed 3 of them and the 4 was standing off to the side. I can only assume it was the guy with 7000 coins and was probably napping most of the time. If I were to go to my producer and tell him, well, we got 9999x3, you’re damn sure he’d tell me to go back and do it again the right f-ing way. I don’t about you but here we don’t get paid to half-ass our tasks.

    So incompetent testers aside, there are a myriad of methods you unimaginative peons could have used to save yourself some time.

    Savant Way
    Debug the coins to yourself. I don’t know what kind of sadistic programmer wouldn’t have a debug command to give coins.

    Smart Way
    Create a save file. This seems like the most basic step. Make a save file you can reload. You know, so you don’t have to spend another 12 hours reproducing nothing. Load the file with you maxed out, have everyone die, did you crash? Yes.

    Ballsy Way
    Ask the programmers to add a test creature that drops 9999 coins. Trust me, they can do it in about 5 minutes.

    Ghetto Way
    Grab a rubber band. Wrap up that controller and go to lunch. Come back and save yourself some time.

    Spiteful Way
    Make one person do the tedium and copy the characters.

    It’s no wonder that this team has been pulling 12 hour shifts for “months” if they’re off performing meaningless tasks like this. Look, I know QA is all about performing monotonous tasks but what separates the kings from the peasants is the ability to perform tasks quickly and efficiently.

    Achievement Unlocked: Waste combined total of 96 hours to perform a task incorrectly
    Reward: Pink Slip

    I hadn't seen any really silly gooses on this forum 'til now. I guess it's nice to know there's at least one.

    Simply, addressing the writer's story in any of the ways you described would have resulted in no story. He may have actually done one of the things you describe, and made up the story anyway, 'cause people do that. Probably, there are specifics of the story he left out, because who cares?

    Wow, JPandarelli manages to be condescending, snarky and wrong all in one go. Well done!

    A. They were not tasked with this, they chose to do this themselves because they had nothing else to do. This isn't their fault, it's a resource management FAIL. You would be a TERRIBLE boss for pink slipping guys who made their own work when their managers weren't making sure they were kept busy. Even if you don't value what they chose to do, it's not their fault they were directionless.

    B. All of your "ways" assume these guys had access to dev tools and/or the developers themselves or that the game allowed saving mid-stream. I think it's pretty obvious that none of those things were available. This sounds like some kind of consol Mario/Sonic knockoff (if not an original) most of which dont allow saving mid level (I'm not aware of one that does, actually). Since it's in multiplayer I'm going to go ahead and say there's no saving.

    C. If this is a big studio project it's quite likely the QA wasn't in the same building, city or even state as the actual programmers AND if done correctly any additional dev tool added to a product (particularly one late in development) would have to go through proper channels or the dev who did it could get in real trouble. Particularly if it caused problems or made it into production.

    D. Also, since this was a late addition to the game it's quite likely all the dev tools were already cut out, or they were disabled for multiplayer to prevent cheaters.

    E. The original Sonic could get over 100 rings on screen without crashing. Frame rate drop, sure, but actual death, no. I'm pretty sure anything made in the last decade targeted at anything more powerful than a handheld could get upwards of 1000 coins before destabilizing. (There are a LOT of factors involved in this, naturally, so your game milage may vary)

    Achievement Unlocked: Snark Backfire

    ExistentialExistenceException: Your thread encountered a NULL pointer and entered a state of non-being.
  • Options
    fortyforty Registered User regular
    I think the original Sonic (and all the Genesis ones) actually had a cap on how many rings would fly out when hit, and it was well below 100. Sure, you had 100 rings, and you dropped to 0 when hit, but there were only maybe 40 rings bouncing around.

Sign In or Register to comment.