Club PA 2.0 has arrived! If you'd like to access some extra PA content and help support the forums, check it out at patreon.com/ClubPA
The image size limit has been raised to 1mb! Anything larger than that should be linked to. This is a HARD limit, please do not abuse it.
Our new Indie Games subforum is now open for business in G&T. Go and check it out, you might land a code for a free game. If you're developing an indie game and want to post about it, follow these directions. If you don't, he'll break your legs! Hahaha! Seriously though.
Our rules have been updated and given their own forum. Go and look at them! They are nice, and there may be new ones that you didn't know about! Hooray for rules! Hooray for The System! Hooray for Conforming!

[Sysadmin] Cause of your issue found: sunspots

191012141537

Posts

  • chrishallett83chrishallett83 Hi! Registered User regular
    Mugsley wrote: »
    This reminds me of a scenario we dealt with during Technical Expert training a couple years ago (the name of the training is wrong, but that's basically what it was).

    Big ass ships use a giant gearbox to transmit power from their engines to the propeller. These gearboxes require oil flow during a power outage while the power system "spins down." There was a ship that was built with the wrong solenoid valve in the oil line (the valve would fail shut instead of fail open). The decision was made to install a UPS for the valve's power so that the gearbox had oil to spin down during a power failure. The ship went to sea and had a power failure. The UPS shit itself, and the gearbox seized. They had to basically dismantle the entire ass end of the ship to replace the gearbox.

    What kind of a fucking idiot doesn't just replace the incorrect solenoid valve and instead puts in place another point of failure?

    DrovekCarpy
  • SeidkonaSeidkona Goddess with a blade Registered User regular
    Mugsley wrote: »
    This reminds me of a scenario we dealt with during Technical Expert training a couple years ago (the name of the training is wrong, but that's basically what it was).

    Big ass ships use a giant gearbox to transmit power from their engines to the propeller. These gearboxes require oil flow during a power outage while the power system "spins down." There was a ship that was built with the wrong solenoid valve in the oil line (the valve would fail shut instead of fail open). The decision was made to install a UPS for the valve's power so that the gearbox had oil to spin down during a power failure. The ship went to sea and had a power failure. The UPS shit itself, and the gearbox seized. They had to basically dismantle the entire ass end of the ship to replace the gearbox.

    What kind of a fucking idiot doesn't just replace the incorrect solenoid valve and instead puts in place another point of failure?

    Idiots chasing short term cost saving at the expense of the future.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    ThegreatcowbowenSiliconStewFeralchrishallett83Echo
  • LD50LD50 Registered User regular
    So we have some pretty serious uptime requirements because we're a level 1 trauma center, and the only one in the state. I'm not familiar with how facilities has backup power set up at the main campus, but we do have a complete replica of production running at a hot site several miles away from the primary datacenter. We test our DR procedures by intentionally forcing a failover to the backup when we want to update prod.

    FeralTL DRCarpy
  • ThegreatcowThegreatcow Lord of All Bacons Washington State - It's Wet up here innit? Registered User regular
    Feral wrote: »
    bowen wrote: »
    the problem in IT is there is so much backlog I do not have time to do disaster recovery procedures and tests like ever

    Oh goddamn this. At my previous job, we kept trying to arrange a generator test to confirm that things were working for our call center (our SLAs basically demand less than 15 minute downtime for our agents woo!) and it kept getting pushed back over and over again.

    Then we finally managed to arrange a test! How you say?

    Well when the Department of Water & Power say they're going to be shutting power off for the whole block due to restringing electricity poles, sounds like as good a time as any!

    Good News: Generator works and kicked in when we lost power.

    Bad News: The BackUPSs we had deployed to the various agents workstations to ensure that their computers don't shutoff during the power transfer between line and generator pretty much all catastrophically failed after 1 minute of battery use and all of the workstations turned off anyway. I guess that will happen when they've been in use for over 6 years without swapping out the battery!

    Ultra Bad News: After presenting the potential bill for replacing the UPS batteries, management decided they'd chance the potential downtime and decided NOT to replace the ups batteries on account of "well the generator came on immediately, they should be able to reboot just fine without issue".

    Conveniently ignoring that several of the older workstations decided to also brick themselves due to the surge part of the UPS also failing during the power transfer and frying their power supplies or mobos.

    My company's largest building has a 300-gallon diesel backup generator feeding all circuits, and there's one circuit that's run through a large UPS and feeds all of the network closets and the main server room.

    We run annual generator and UPS tests, though Seattle power is flaky enough that we have a few major power outages per year.

    We've never had a computer brick after the generator kicks in. I'm not sure if there's something our electricians did, that yours didn't do, to prevent that.

    However, we did attempt to put in small personal APC UPSes at a number of desks. It takes about 5-10 seconds for the generator to kick in, but some folks felt their jobs were important enough that they shouldn't suffer any interruption during a power outage. Despite being brand new and appropriately sized for their loads, the APC UPSes had roughly a 50% failure rate. The UPSes not only failed to maintain power during the brief outage, but they also didn't come back online once the generator was on. And because most of those users refused (or didn't understand) to unplug the UPSes, what should have been a 10 second wait for their computers to come back on became a 20 minute wait while IT scrambled to unplug them ourselves.

    Damn that sounds infuriating. As for the bricking, the only thing I can think of was the building wiring itself was pretty old and from what I heard, a lot of the wiring was done "off the books" so who knows what nightmare fuel was behind those walls.

    It also didn't help that the wired cubicles were in some pretty bad shape. users kept kicking the cubicle walls which caused a lot of the joiner fuses to break/fail so who knows what was going on with that setup.

    Feral
  • wunderbarwunderbar What Have I Done? Registered User regular
    if your flow of power is generator to ups to servers than you should never have an instance of dirty generator power going to a server because the UPS will smooth that out.

    That's actually the exact setup we have at my current place. Our Server room UPS is currently only good for 17 minutes of power (this number is actually getting larger over time as I reduce the number of physical servers we have), but we have a generator that kicks in in the event of a power outage and that takes no more than 5 minutes to get up to speed.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
    FeralTL DR
  • FeralFeral MEMETICHARIZARD along with you if I get drunk well I know I'm gonna be gonna be the man whoRegistered User regular
    wunderbar wrote: »
    if your flow of power is generator to ups to servers than you should never have an instance of dirty generator power going to a server because the UPS will smooth that out.

    That's actually the exact setup we have at my current place. Our Server room UPS is currently only good for 17 minutes of power (this number is actually getting larger over time as I reduce the number of physical servers we have), but we have a generator that kicks in in the event of a power outage and that takes no more than 5 minutes to get up to speed.

    That's what we have.

    The generator powers the whole building but the server room circuit goes through a big facility UPS.

    (Anything that has redundant power supplies has one plug in the UPS-fed circuit and one plug in the non-UPS-fed circuit.)

    When the generator kicks in, the UPS keeps servers and network online. But the human areas lose power for 10 seconds or so.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD along with you if I get drunk well I know I'm gonna be gonna be the man whoRegistered User regular
    It's a pretty good setup.

    Regarding the small APC UPS fiasco:

    I'm partly mad at my boss for caving to pressure

    I'm partly mad from the principle of "poor planning on your part does not constitute an emergency on my part" (which never ever seems to apply to IT)

    I'm partly mad because we have this very nice robust power setup with a big generator and UPS and multiple circuits and redundant power supplies but no that isn't fucking good enough for some people

    It's like a gourmet kitchen cooking up a fine filet mignon steak but some overpaid motherfucker demands ketchup

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    Thegreatcow
  • SeidkonaSeidkona Goddess with a blade Registered User regular
    edited May 9
    Mmmmmm, ketchup.

    *Wraps all of her after dinner mints in gold leaf*

    Seidkona on
    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • wunderbarwunderbar What Have I Done? Registered User regular
    I will now be referring to anywhere in the building but the server room as "human areas"

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
    BahamutZEROThawmusFeralEchoThegreatcow
  • MugsleyMugsley Registered User regular
    Entaru wrote: »
    Mugsley wrote: »
    This reminds me of a scenario we dealt with during Technical Expert training a couple years ago (the name of the training is wrong, but that's basically what it was).

    Big ass ships use a giant gearbox to transmit power from their engines to the propeller. These gearboxes require oil flow during a power outage while the power system "spins down." There was a ship that was built with the wrong solenoid valve in the oil line (the valve would fail shut instead of fail open). The decision was made to install a UPS for the valve's power so that the gearbox had oil to spin down during a power failure. The ship went to sea and had a power failure. The UPS shit itself, and the gearbox seized. They had to basically dismantle the entire ass end of the ship to replace the gearbox.

    What kind of a fucking idiot doesn't just replace the incorrect solenoid valve and instead puts in place another point of failure?

    Idiots chasing short term cost saving at the expense of the future.

    It was interesting because they broke us up into teams to debate potential solutions (replace the valve or add an UPS) with their relevant costs to schedule and time, before telling us what decision was made at the time. Every single team opted to take the short term schedule/budget hit to install the correct valve. We all had different reasonings, but all reached the same decision (at least one team said "no" to the UPS because they were familiar with dealing with UPS).

    From what I recall, the actual scenario involved a ship that was at risk of going over schedule for repairs (and those related costs); which was what drove the UPS decision.

  • LD50LD50 Registered User regular
    wunderbar wrote: »
    I will now be referring to anywhere in the building but the server room as "human areas"

    I'm going to rebuild a wing of our building and bury people in the walls and refer to it as the human area.

    ThawmusFeralwunderbarDarkewolfeLe_Goat
  • wunderbarwunderbar What Have I Done? Registered User regular
    so this is cool. was looking into some SQL performance issues.

    for some reason this one SQL server is stuck at 100mbit on the nic.

    Yea, no wonder calls are slow.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
    SeidkonaFeralmcp
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    wunderbar wrote: »
    bowen wrote: »
    the problem in IT is there is so much backlog I do not have time to do disaster recovery procedures and tests like ever

    This is literally my life right now. I've been told I need to drop everything to write up backup restore testing documents/guidelines by the end of the week, and schedule a partial recovery test by end of May.

    I have about 25 other things I need to be doing but that's the rest of my week
    I have a ticket in our system that my boss put in shortly after I became the backup admin (about a year into my job), which says something like “create a procedure for monthly backup restoration test.” I’ve been the backup admin for 13 years. The boss who put that in retired six years ago.

    bowenDrovekchrishallett83MugsleyThe Escape GoatCarpy
  • EchoEcho Moderator mod
    wunderbar wrote: »
    so this is cool. was looking into some SQL performance issues.

    for some reason this one SQL server is stuck at 100mbit on the nic.

    Yea, no wonder calls are slow.

    I once forgot the unit in a service config and allocated it 100 bytes of memory.

    Echo wrote: »
    Let they who have not posted about their balls in the wrong thread cast the first stone.
    bowenwunderbarMvrckCarpy
  • SeidkonaSeidkona Goddess with a blade Registered User regular
    I got a new job.

    I quit.

    And yet I still just worked all night like a chump.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    ThawmuswunderbarMugsleyMvrckShadowfireDrovekInquisitor77DarkewolfeLe_Goatlwt1973
  • ThawmusThawmus Registered User regular
    edited May 10
    Entaru wrote: »
    I got a new job.

    I quit.

    And yet I still just worked all night like a chump.

    How many more nights of being a chump left on the docket? Before you move on to being a Mac user?

    Thawmus on
    steam_sig.png
    DrovekDarkewolfe
  • LD50LD50 Registered User regular
    Entaru wrote: »
    I got a new job.

    I quit.

    And yet I still just worked all night like a chump.

    Hmm. Are you sure you're familiar with what quitting is?

    ThawmuswunderbarMugsleyFeralShadowfireDrovekJazzInquisitor77DarkewolfeAegisLe_Goat
  • EchoEcho Moderator mod
    LD50 wrote: »
    Entaru wrote: »
    I got a new job.

    I quit.

    And yet I still just worked all night like a chump.

    Hmm. Are you sure you're familiar with what quitting is?

    Add to CV: "Doesn't know when to quit"

    Echo wrote: »
    Let they who have not posted about their balls in the wrong thread cast the first stone.
    DrovekDarkewolfeAustinP0027
  • JazzJazz irregular Un-UKRegistered User regular
    "I don't know the meaning of the word 'quit'!"

    t3x69ku4wq46.jpg

  • SeidkonaSeidkona Goddess with a blade Registered User regular
    Thawmus wrote: »
    Entaru wrote: »
    I got a new job.

    I quit.

    And yet I still just worked all night like a chump.

    How many more nights of being a chump left on the docket? Before you move on to being a Mac user?

    Last day is Wendsday. Then I have a 4.5 day weekend and start the new job.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    Thawmuschrishallett83Le_Goat
  • SeidkonaSeidkona Goddess with a blade Registered User regular
    LD50 wrote: »
    Entaru wrote: »
    I got a new job.

    I quit.

    And yet I still just worked all night like a chump.

    Hmm. Are you sure you're familiar with what quitting is?

    Maybe not?

    It probably doesn't bode well that I just pulled that all nighter because I was the only one qualified to do it.

    Oh well. Not my problem anymore.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    ThawmusLD50Naphtalichrishallett83
  • DarkewolfeDarkewolfe Registered User regular
    Entaru wrote: »
    LD50 wrote: »
    Entaru wrote: »
    I got a new job.

    I quit.

    And yet I still just worked all night like a chump.

    Hmm. Are you sure you're familiar with what quitting is?

    Maybe not?

    It probably doesn't bode well that I just pulled that all nighter because I was the only one qualified to do it.

    Oh well. Not my problem anymore.

    I was serious, the correct response is, "I just quit. If you don't know how to do this yet I guess you'd better stay until you figure it out."

    What is this I don't even.
    DrovekShadowfireJazzSeidkonachrishallett83FeralSwashbucklerXXThawmusMvrckLe_GoatAustinP0027LaOs
  • SeidkonaSeidkona Goddess with a blade Registered User regular
    Last couple days.

    Started wiping my ancillary personal systems. Then I need to write some documentation and get puppet in a good state.

    Wednesday after noon I fully plan to be at the zoo and not on my scheduled one on one.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    ThawmusTL DRNaphtali
  • LD50LD50 Registered User regular
    Our ticketing system just fell off a cliff. Happy Monday.

  • ThawmusThawmus Registered User regular
    LD50 wrote: »
    Our ticketing system just fell off a cliff. Happy Monday.

    I mean...

    Yeah?

    That sounds pretty awesome.

    steam_sig.png
    DrovekwunderbarSeidkonaMvrckThegreatcowFeralJaysonFour
  • wunderbarwunderbar What Have I Done? Registered User regular
    LD50 wrote: »
    Our ticketing system just fell off a cliff. Happy Monday.

    That just means no tickets can come in, which makes it the best monday.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
    ThawmusSeidkonaDrovekShadowfireJaysonFour
  • twmjrtwmjr Registered User regular
    wunderbar wrote: »
    LD50 wrote: »
    Our ticketing system just fell off a cliff. Happy Monday.

    That just means no tickets can come in, which makes it the best monday.

    This entirely depends on what their workaround system is for the ticketing system being down.

    Back in the day ours involved paper and pens and lots of phone calls. We didn't like when the ticketing system went down.

  • ThawmusThawmus Registered User regular
    twmjr wrote: »
    wunderbar wrote: »
    LD50 wrote: »
    Our ticketing system just fell off a cliff. Happy Monday.

    That just means no tickets can come in, which makes it the best monday.

    This entirely depends on what their workaround system is for the ticketing system being down.

    Back in the day ours involved paper and pens and lots of phone calls. We didn't like when the ticketing system went down.

    Man way to harsh on some glorious downtime!

    steam_sig.png
    SeidkonaThegreatcow
  • SeidkonaSeidkona Goddess with a blade Registered User regular
    edited May 13
    I am listening to the VMware admin at my old job talk shit about the cloud without an understanding of anything he is saying.

    Seidkona on
    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • wunderbarwunderbar What Have I Done? Registered User regular
    Entaru wrote: »
    I am listening the VMware admin at my old job talk shit about the cloud without an understanding of anything he is saying.

    sounds like every executive I've ever talked to.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
    DrovekSeidkonaLe_GoatThawmusFeralDarkewolfechrishallett83lwt1973
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    One of my clients called to report the internet was down again. It would appear the fiber got cut again. For those of you keeping score at home, the fiber for this site has been cut 5 times in as many months.

    camo_sig.png
    ThawmusMugsleyRandomHajilechrishallett83Le_GoatCarpyJaysonFour
  • ShadowfireShadowfire Vermont, in the middle of nowhereRegistered User regular
    That_Guy wrote: »
    One of my clients called to report the internet was down again. It would appear the fiber got cut again. For those of you keeping score at home, the fiber for this site has been cut 5 times in as many months.

    Where is it being cut? Side of the building? A squirrel on the line? Is it two dudes in a van that totally work for the power company?

    WiiU: Windrunner ; Guild Wars 2: Shadowfire.3940 ; PSN: Bradcopter
  • ThawmusThawmus Registered User regular
    edited May 13
    Entaru wrote: »
    I am listening to the VMware admin at my old job talk shit about the cloud without an understanding of anything he is saying.

    I work with a guy who shits on VMware all the time whenever someone mentions it, and has never, ever, in his entire life, ever worked with it.

    So what I'm saying is that the universe is in balance.

    Thawmus on
    steam_sig.png
    FeralDrovek
  • FeralFeral MEMETICHARIZARD along with you if I get drunk well I know I'm gonna be gonna be the man whoRegistered User regular
    Nobody here shits on VMware in general, but whenever one of our crappy legacy applications decides to throw a tantrum, one of the applications guys is all "IS THERE SOMETHING WRONG WITH VMWARE!??!?!"

    No, Brad, VMware is not the reason Lotus Notes 1993 is shitting itself today

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    ThawmusDrovekThegreatcowwunderbarLD50RandomHajilechrishallett83Le_GoatCarpyJaysonFour
  • TL DRTL DR Not at all confident in his reflexive opinions of thingsRegistered User regular
    Just swung by a client's satellite office because the folks replacing their carpet cut a patch cable

    Whoever did it actually spliced the cable back together, twisting the strands and applying electrical tape. When I tested the run, it worked fine! What caused their problem is that the client's office staff just yanked everything out of the room in a hurry before the installers started and didn't plug anything back in.

    Con: having to spend 10 minutes standing there scratching my head at the tangled mess of year-over-year halfassed wiring before I figured out what had happened
    Pro: I got to do a new network run, after not doing so for years. That's novel, at least.

    MugsleyFeralBahamutZERO
  • bowenbowen How you doin'? Registered User regular
    edited May 13
    Feral wrote: »
    Nobody here shits on VMware in general, but whenever one of our crappy legacy applications decides to throw a tantrum, one of the applications guys is all "IS THERE SOMETHING WRONG WITH VMWARE!??!?!"

    No, Brad, VMware is not the reason Lotus Notes 1993 is shitting itself today

    Speaking of lotus notes... the last big upgrade I did I pretended that the lotus notes CD was lost when our systems crashed and I didn't have a backup of it and took the verbal abuse from my boss and the office manager about how she had to waste a few hours of her time recreating templates for patient forms in word so I would never have to deal with lotus notes (I think we had 94 or just lotus notes 4?) again.

    bowen on
    not a doctor, not a lawyer, don't @ me
    TL DRwunderbarLD50FeldornShadowfireMvrckLe_Goat
  • bowenbowen How you doin'? Registered User regular
    TL DR wrote: »
    Just swung by a client's satellite office because the folks replacing their carpet cut a patch cable

    Whoever did it actually spliced the cable back together, twisting the strands and applying electrical tape. When I tested the run, it worked fine! What caused their problem is that the client's office staff just yanked everything out of the room in a hurry before the installers started and didn't plug anything back in.

    Con: having to spend 10 minutes standing there scratching my head at the tangled mess of year-over-year halfassed wiring before I figured out what had happened
    Pro: I got to do a new network run, after not doing so for years. That's novel, at least.

    Yeah splicing like that will work, but you'll get a lot of noise and signal loss too. It's only a temporary measure.

    not a doctor, not a lawyer, don't @ me
    ThawmusInfidel
  • mcpmcp Registered User regular
    Somehow setting up Splunk ended up on my desk.

    Easy enough to setup, and get data into.

    Now I'm like, what the fuck do I want to get out of it?

    Is there a repository for dashboards and shit? I looked around gosplunk.com, but there's not a whole lot there.

    walrus.png
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    bowen wrote: »
    Feral wrote: »
    Nobody here shits on VMware in general, but whenever one of our crappy legacy applications decides to throw a tantrum, one of the applications guys is all "IS THERE SOMETHING WRONG WITH VMWARE!??!?!"

    No, Brad, VMware is not the reason Lotus Notes 1993 is shitting itself today

    Speaking of lotus notes... the last big upgrade I did I pretended that the lotus notes CD was lost when our systems crashed and I didn't have a backup of it and took the verbal abuse from my boss and the office manager about how she had to waste a few hours of her time recreating templates for patient forms in word so I would never have to deal with lotus notes (I think we had 94 or just lotus notes 4?) again.
    This sounds like you mean Lotus 1-2-3, not Notes. Notes/Domino is a database platform, not a document editor. (Well, around the fairly recent Notes 9, they integrated a customized OpenOffice into the Notes client, but I’m sure that’s not what you meant.)

    I’m going to say it again: Notes is fine! Microsoft did a damn good job convincing people otherwise right about the time that most of the worst things about Notes were gone. Many many golf course decisions with slimy reps.

  • FeralFeral MEMETICHARIZARD along with you if I get drunk well I know I'm gonna be gonna be the man whoRegistered User regular
    We don't actually use Lotus anything here. I was using it as a euphemism.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
Sign In or Register to comment.