As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

[Sysadmin] Nightmare fuel

1356799

Posts

  • iTunesIsEviliTunesIsEvil Cornfield? Cornfield.Registered User regular
    Feral wrote: »
    Entaru wrote: »
    wunderbar wrote: »
    twmjr wrote: »
    you know it's a good day when you bring three new WAN links up in one afternoon with no issues/carrier involvement to resolve anything

    this has never happened before and I feel like I've won a championship or something

    I don't believe you.

    You also think it's Friday.

    http://isitfriday.org

    "Click to enable Flash Player"

    oh, you cads.

  • chamberlainchamberlain Registered User regular
    Let's see what's in the cabinet that no one ever opens:

    5yejj80srwzi.jpg

    Impossible.

    sq3kitk3t7yx.jpg

    Anyone need a copy of Windows 95?

  • bowenbowen How you doin'? Registered User regular
    I mean I wouldn't pay for it, but it'd be cool actually having win95 usb edition on CD someday when I want to take a trip down memory lane without having to find it on the tubes.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    Yo dawg, we found DOS 6.22 floppies the other day when cleaning out our software cabinet. We kept them.

  • jungleroomxjungleroomx It's never too many graves, it's always not enough shovels Registered User regular
    edited November 2017
    Welp, we finally got our pristine virtual server environment enabled, denied access to developers, and started on setting up a default customer setup for reference.

    About 10 minutes in, we get an email from a high muckity-muck that we are no longer allowed to do it for unspecified reasons.

    Oh, look at that, an it business analyst spot that pays double what I do now and I fit the quals pretty well.

    *updates resume*

    jungleroomx on
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    Welp, we finally got our pristine virtual server environment enabled, denied access to developers, and started on setting up a default customer setup for reference.

    About 10 minutes in, we get an email from a high muckity-muck that we are no longer allowed to do it for unspecified reasons.

    *updates resume*
    *whispers* you guys he's still here...

  • jungleroomxjungleroomx It's never too many graves, it's always not enough shovels Registered User regular
    Welp, we finally got our pristine virtual server environment enabled, denied access to developers, and started on setting up a default customer setup for reference.

    About 10 minutes in, we get an email from a high muckity-muck that we are no longer allowed to do it for unspecified reasons.

    *updates resume*
    *whispers* you guys he's still here...

    Ah ok, I can fix that.

  • CogCog What'd you expect? Registered User regular
    edited November 2017
    Holy shit you guys

    this fucking place

    They have 24 TB of storage in DAS off their ESX hosts but only 12TB of it was actually provisioned. Failed drive in the array. No hot spares were configured.

    Everything was out of warranty

    still backing up to a single-tape drive

    One of their DCs is in a site that has an assigned subnet but it has an IP that doesn't match that subnet, so replication is all fucked up

    There are random printers and folders shared of off random file servers and dcs everywhere. Users complain that network resources and mapped drives randomly appear and disappear

    They have a DHCP scope that has a fucking absurd amount of reservations in it including apparently just some random workstations, reservations made on IP addresses outside the goddamn scope, and a reservation for one of their file servers

    I logged in to said file server and checked... yep, sure enough, it's set to DHCP

    but what's this? A second NIC? Set to a static IP address

    within the fucking DHCP scope

    that has no reservation or exclusion :rotate:

    lets just see... Yeah, Bad_address. It's conflicting with something. No wonder your shares and printers randomly disappear.

    It might just be easier to advise them to burn the building down.

    Cog on
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    I like to put the taskbar on the left and set it to auto-hide but only on servers that are primarily accessed by RDP.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • CogCog What'd you expect? Registered User regular
    The previous IT guy had handed out instructions for mapping drives after connecting to the VPN, but one of the drives would never map for anyone and he never figured it out. I saw in 5 seconds that he mispelled the name of the shared folder on the instructions.

  • AiouaAioua Ora Occidens Ora OptimaRegistered User regular
    Cog wrote: »
    The previous IT guy had handed out instructions for mapping drives after connecting to the VPN, but one of the drives would never map for anyone and he never figured it out. I saw in 5 seconds that he mispelled the name of the shared folder on the instructions.

    :tell_me_more:

    life's a game that you're bound to lose / like using a hammer to pound in screws
    fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
    that's right we're on a fucked up cruise / God is dead but at least we have booze
    bad things happen, no one knows why / the sun burns out and everyone dies
  • SiliconStewSiliconStew Registered User regular
    Feral wrote: »
    I like to put the taskbar on the left and set it to auto-hide but only on servers that are primarily accessed by RDP.

    Ok, now put your RDP window on your right screen so you can enjoy the annoyance you've just created for yourself.

    Just remember that half the people you meet are below average intelligence.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Feral wrote: »
    I like to put the taskbar on the left and set it to auto-hide but only on servers that are primarily accessed by RDP.

    Ok, now put your RDP window on your right screen so you can enjoy the annoyance you've just created for yourself.

    Nah, I do all of my administration through Powershell sessions and RSAT.

    People who only use GUIs should be punished.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Also, I'm totally trolling and do not take those posts seriously.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • bowenbowen How you doin'? Registered User regular
    15k on a server

    dell guy called like
    3 hours before it shipped
    but it was like end of the day for us and I thought he was trying to upsell me on storage or some shit so I let it roll to voice mail

    nope he wanted to upgrade our shipping and support for free but I needed to respond within like that 40 minute window

    boss was upset but like bro I get a dozen sales calls a day I can't spend all day dealing with vendors, and a 40 minute window was not enough

    We can wait an extra 4 days for the server, and it still baffles my mind why this guy can't upgrade our support component.. I can obviously add it after the fact.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • SiliconStewSiliconStew Registered User regular
    bowen wrote: »
    15k on a server

    dell guy called like
    3 hours before it shipped
    but it was like end of the day for us and I thought he was trying to upsell me on storage or some shit so I let it roll to voice mail

    nope he wanted to upgrade our shipping and support for free but I needed to respond within like that 40 minute window

    boss was upset but like bro I get a dozen sales calls a day I can't spend all day dealing with vendors, and a 40 minute window was not enough

    We can wait an extra 4 days for the server, and it still baffles my mind why this guy can't upgrade our support component.. I can obviously add it after the fact.

    Was this on 10/31? I think that's the end of Dell's 3rd Quarter, so it may be related to sales quotas or discounts that may have been expiring.

    Just remember that half the people you meet are below average intelligence.
  • bowenbowen How you doin'? Registered User regular
    bowen wrote: »
    15k on a server

    dell guy called like
    3 hours before it shipped
    but it was like end of the day for us and I thought he was trying to upsell me on storage or some shit so I let it roll to voice mail

    nope he wanted to upgrade our shipping and support for free but I needed to respond within like that 40 minute window

    boss was upset but like bro I get a dozen sales calls a day I can't spend all day dealing with vendors, and a 40 minute window was not enough

    We can wait an extra 4 days for the server, and it still baffles my mind why this guy can't upgrade our support component.. I can obviously add it after the fact.

    Was this on 10/31? I think that's the end of Dell's 3rd Quarter, so it may be related to sales quotas or discounts that may have been expiring.

    11/1

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Non-troll post about Cisco shenanigans. @twmjr

    I replaced one of our old Catalyst 4500s with a new 3850 stack last night. Specifically I set both to HSRP with "standby ip" then forced the 3850 stack to HSRP active.

    Remember from my prior adventures that the spanning tree configs on those old 4500s were completely fucked. BTW, so were the EIGRP configs. EIGRP was completely broken because some genius basically flipped a coin to decide which routers across the old organization should be stubs, which shouldn't, which routers had redistribute static, which shouldn't, etc.

    The 4500s also had VLANs and subnets configured that were no longer in use, and a ton of other legacy garbage.

    So before the cutover last night, I migrated over only the VLANs and subnets and static routes we still use. I dropped EIGRP and used static for everything. None of this was well documented so it took a lot of educated guesswork and wiresharking over the last several months to figure out which subnets/routes/etc were safe to abandon.

    After changing the active router to the 3850 stack, I pinged and tracerouted everything I could possibly think of. Tracerouted workstation to server, server to workstation, workstation to Internet, remote office to datacenter, server to VOIP phones, etc. I played with packet sizes, got my laptop on a mobile hotspot and logged in with VPN, did some large downloads from the Internet, etc.

    Everything looked fantastic.

    So I unplugged old 4500 and powered it down. Re-ran a few ping and tracerouted tests and everything still looked good, so I texted my boss and a couple of coworkers and then went home.

    ...until this morning... (continued...)

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited November 2017
    This morning I got the following text message from a coworker:

    "PCs on the 169 subnet can't get on network"

    We don't have a 169 subnet.

    Now if you know the first fucking thing about Windows you know that if it can't get a DHCP address, it will autoassign an address starting with 169. Why literally nobody else in the department at the time couldn't identify 169.xxx.xxx.xxx as a Windows autoconfiguration address, and therefore a DHCP problem, is a story in and of itself. But this is a story about Cisco, not my coworkers, so I digress.

    I figured it was something stupid, like I forgot to declare ip helper-address on a VLAN or something like that. So I remoted in from home and looked at the 3850 configs...

    ...nope, I remembered all my ip helper-addresses.

    So I did the usual debug commands like debug ip dhcp server packets and debug ip udp, and I also logged into our DHCP server to see if it had any interesting events.

    debug ip dhcp server packets produced no output. At all. Zilch zero nada.

    debug ip udp showed DHCP request packets (UDP port 67) coming in from workstations, but none of them were getting forwarded.

    DHCP server logs (predictably, at this point) showed no DHCP packets being received.

    So I drove into the office, opened a case with Cisco support, and continued to troubleshoot. We're now about an hour into the problem with multiple people across multiple departments unable to get on the network, and nobody else in my department knows enough about TCP/IP in Windows to identify a 169. IP address but i'm digressing again sorry...

    While waiting for Cisco to call me back, I comb through the old 4500 configs to see if there's anything I missed. Any DHCP-related commands or any routing-related commands. Nope, nothing. I also looked through some of our other Catalysts across the network to see if there's anything configured on those that I might have missed. Nope, not a thing.

    More Googling and I come across a forum post from somebody who had the same problem and said that the command service dhcp fixed it. So, fine, fuck it, I try it.

    Suddenly the dhcp debug starts to display forwarding events and the DHCP server starts to receive DHCP requests. I can see workstations lighting up across the building.

    Note that service dhcp is not declared on any other Catalyst in our network nor was it declared on the old 4500.

    A little while later, Cisco calls me back and I run it by the tech and he's like

    "You know, you're the second customer this week with that exact problem. You're right, you shouldn't have to do "service dhcp." It should be on by default. I think it's a bug in our 3850 firmware but I'm not sure."

    ...

    cisco



    cisco plz

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • donavannjdonavannj Registered User regular
    Aioua wrote: »
    donavannj wrote: »
    Why the hell is this one user profile acting like a roaming profile on this specific server even though it's not set as one and doesn't act this way on any other machine and no other profile acts this way on this specific server.

    because windows

    have you deleted/renamed the profile folder and deleted the associated registry keys* so it re-makes it from scratch?

    *in HKLM:\Software\Microsoft\Windows NT\CurrentVersion\ProfileList
    donavannj wrote: »
    Aioua wrote: »
    donavannj wrote: »
    Why the hell is this one user profile acting like a roaming profile on this specific server even though it's not set as one and doesn't act this way on any other machine and no other profile acts this way on this specific server.

    because windows

    have you deleted/renamed the profile folder and deleted the associated registry keys* so it re-makes it from scratch?

    *in HKLM:\Software\Microsoft\Windows NT\CurrentVersion\ProfileList

    The thing is, none of those on this specific machine exist unless I'm explicitly signed in as the user account. I sign out from the account and log in as an administrator account and they disappear from the registry. It's not set to act this way and doesn't act this way anywhere else. And, again, it's the only profile affected by this problem on this machine.

    Did someone mistakenly stick their account in the local Guests (or Domain Guests) group? Members in those groups will have their profile folders and registry keys deleted on logout.

    In case you were wondering, we found the cause. Remote Desktop Services has a "feature" you can turn on for collections called User Profile Disks. This setting had been turned on back in April by someone and they were starting to corrupt, which was starting to spread to other profiles. Turning this setting off fixed it.

    steam_sig.png
  • SiliconStewSiliconStew Registered User regular
    It's a feature in that user profile disks are used if you have multiple RDS hosts in a farm and want your users to be able to log into any of the hosts while keeping their same profile.

    Just remember that half the people you meet are below average intelligence.
  • CogCog What'd you expect? Registered User regular
    Aioua wrote: »
    Cog wrote: »
    The previous IT guy had handed out instructions for mapping drives after connecting to the VPN, but one of the drives would never map for anyone and he never figured it out. I saw in 5 seconds that he mispelled the name of the shared folder on the instructions.

    :tell_me_more:

    This place wants the work to fix the major issues to be done on a weekend so they don't have business hour downtime.

    Would be a pretty power move to just flip them to a whole new domain over the weekend and have them come in and everything is fucking flawless.
    Would be more likely they come in and i'm passed out on the server room floor and the esx host is purple screened and nothing works and there's a thousand photo copies of my ass in the recycling and the trash can in the men's room is on fire

  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    Welp, we finally got our pristine virtual server environment enabled, denied access to developers, and started on setting up a default customer setup for reference.

    About 10 minutes in, we get an email from a high muckity-muck that we are no longer allowed to do it for unspecified reasons.

    *updates resume*
    *whispers* you guys he's still here...

    Ah ok, I can fix that.
    Nah, c'mon, stick around, you're okay.

  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    bowen wrote: »
    bowen wrote: »
    15k on a server

    dell guy called like
    3 hours before it shipped
    but it was like end of the day for us and I thought he was trying to upsell me on storage or some shit so I let it roll to voice mail

    nope he wanted to upgrade our shipping and support for free but I needed to respond within like that 40 minute window

    boss was upset but like bro I get a dozen sales calls a day I can't spend all day dealing with vendors, and a 40 minute window was not enough

    We can wait an extra 4 days for the server, and it still baffles my mind why this guy can't upgrade our support component.. I can obviously add it after the fact.

    Was this on 10/31? I think that's the end of Dell's 3rd Quarter, so it may be related to sales quotas or discounts that may have been expiring.

    11/1
    They had a promotion for extending warranty on old servers and it was through 11/1 according to our warranty specialist dude, yeah.

  • bowenbowen How you doin'? Registered User regular
    bowen wrote: »
    bowen wrote: »
    15k on a server

    dell guy called like
    3 hours before it shipped
    but it was like end of the day for us and I thought he was trying to upsell me on storage or some shit so I let it roll to voice mail

    nope he wanted to upgrade our shipping and support for free but I needed to respond within like that 40 minute window

    boss was upset but like bro I get a dozen sales calls a day I can't spend all day dealing with vendors, and a 40 minute window was not enough

    We can wait an extra 4 days for the server, and it still baffles my mind why this guy can't upgrade our support component.. I can obviously add it after the fact.

    Was this on 10/31? I think that's the end of Dell's 3rd Quarter, so it may be related to sales quotas or discounts that may have been expiring.

    11/1
    They had a promotion for extending warranty on old servers and it was through 11/1 according to our warranty specialist dude, yeah.

    New server though!

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • CogCog What'd you expect? Registered User regular
    Fucking sales weasels man, who knows.

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Cog wrote: »
    Aioua wrote: »
    Cog wrote: »
    The previous IT guy had handed out instructions for mapping drives after connecting to the VPN, but one of the drives would never map for anyone and he never figured it out. I saw in 5 seconds that he mispelled the name of the shared folder on the instructions.

    :tell_me_more:

    This place wants the work to fix the major issues to be done on a weekend so they don't have business hour downtime.

    That's literally not going to happen

    It is literally impossible

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    If they're lucky after a year of weekly or twice weekly visits with a lot of painstaking reconfiguration, they'll see their issues slowly improve.

    They are completely insane if they think their problems can be fixed without business impact.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • bowenbowen How you doin'? Registered User regular
    Yeah I always tell my boss, "sure we can limp along for several weeks with me fixing shit here and there, or I can rip the band aid off for 30-60 minutes during lunch"

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • CogCog What'd you expect? Registered User regular
    edited November 2017
    I mean, I can probably unfuck AD replication and Re-IP the servers that are in the DHCP scope, and get drive & printer mapping via GPO over a weekend.

    The one that's really fucked though is the DC in the site in Sites & Services that has the IP that doesn't match the defined subnet. That's a bitch.

    And it's not going to "fix all the problems" regardless. It might just eliminate some of the big pain points.

    Nothing's going to "fix" things until we set up a whole new domain and jettison this old dumpster fire.

    EDIT: Oh, and having everyone stop being local admins of everything. I can probably work that out, but that will cause some downtime because there's just no way to test everything.

    Cog on
  • CogCog What'd you expect? Registered User regular
    Most of the time when we on-board a client into one of our managed infrastructure services, one of our pre-conditions is that we will be shit-canning your old AD. It's nearly never worth the effort, and almost everyone's domain sucks ass anyway.

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Cog wrote: »
    Most of the time when we on-board a client into one of our managed infrastructure services, one of our pre-conditions is that we will be shit-canning your old AD. It's nearly never worth the effort, and almost everyone's domain sucks ass anyway.

    I can't imagine doing that for every client but okay

    If somebody tried to do that here they'd quickly end up in thisisfine.jpg

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • CogCog What'd you expect? Registered User regular
    Feral wrote: »
    Cog wrote: »
    Most of the time when we on-board a client into one of our managed infrastructure services, one of our pre-conditions is that we will be shit-canning your old AD. It's nearly never worth the effort, and almost everyone's domain sucks ass anyway.

    I can't imagine doing that for every client but okay

    If somebody tried to do that here they'd quickly end up in thisisfine.jpg

    Statistically speaking, your AD environment is probably garbage.

  • CogCog What'd you expect? Registered User regular
    edited November 2017
    Also we spin out template VMs and as soon as they boot up we have a enormous powershell script that installs AD DS and promotes the server and generates the OU structure and all of the general service accounts and things we always use. It goes from zero to fully functioning domain in about 20 minutes.

    Cog on
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Cog wrote: »
    Feral wrote: »
    Cog wrote: »
    Most of the time when we on-board a client into one of our managed infrastructure services, one of our pre-conditions is that we will be shit-canning your old AD. It's nearly never worth the effort, and almost everyone's domain sucks ass anyway.

    I can't imagine doing that for every client but okay

    If somebody tried to do that here they'd quickly end up in thisisfine.jpg

    Statistically speaking, your AD environment is probably garbage.

    It was definitely garbage when I started.

    At this point it's only about 20% garbage.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • wunderbarwunderbar What Have I Done? Registered User regular
    edited November 2017
    You people and your Windows 95's. This is actually one of my more prized possessions among my IT/comptuer stuff. This was handed down to me by a friend of my dad years ago, as DOS 2.10 is a bit before my time. I first learned computers on DOS 6 and Win 3.1.

    I put it in an imgur gallery since I took 12 pictures of it. It's the retail package of IBM Disk Operating System 2.10, provided by Microsoft. It includes a manual with all of the DOS commands.

    https://imgur.com/gallery/6OnlJ

    wunderbar on
    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • LD50LD50 Registered User regular
    I want to point out that windows 95 was a real weird OS.

  • bowenbowen How you doin'? Registered User regular
    I never had a problem with it?

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • SeidkonaSeidkona Had an upgrade Registered User regular
    I still have my copies of Beos and os/2 warp.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • bowenbowen How you doin'? Registered User regular
    I remember our school was had IPX with OS/2 computers at one point

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
This discussion has been closed.