Options

[Sysadmin] Nightmare fuel

1545557596099

Posts

  • Options
    jungleroomxjungleroomx It's never too many graves, it's always not enough shovels Registered User regular
    Feral wrote: »
    I kinda don't care too much about server names as long as they're scalable and I can get a reasonable idea of what a server is doing from the name.

    DC01 or ADS01 or DOMAIN01 are all fine as far as I'm concerned.

    I just get annoyed when I'm going through security logs and I stumble across something like "junebug32" and I'm like "what the fuck is junebug32?"

    "Looks like that server is running something called Junebug 3.2."
    "What the fuck is Junebug?"
    "I dunno. Let's ask the whole department. Does anybody know what Junebug is?"
    "I dunno but I think maybe James does. He's out of the office today."
    "I think it might be something for HR?"

    Had a client with server names like

    "UNDERTABLE"

    or

    "LEFTCLOSET"

    The guy who set it up as such as just as much fun to work with as you'd think.

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Apothe0sis wrote: »
    What's your preference for Junebug32?

    WINAPPSEHR03?

    Depends on what Junebug32 is. I just made that up.

    If it's an HR benefits package, then "HRBENEFITS01" is fine. Maybe HR01 or BENEFITS01. That doesn't tell me what the server does, but it gives me a hint at who is going to be mad if it breaks.

    City code like SEA-HR01 or a suffix for virtual vs physical like SEA-V-HR01 is okay too.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    wunderbar wrote: »
    Feral wrote: »
    I kinda don't care too much about server names as long as they're scalable and I can get a reasonable idea of what a server is doing from the name.

    DC01 or ADS01 or DOMAIN01 are all fine as far as I'm concerned.

    I just get annoyed when I'm going through security logs and I stumble across something like "junebug32" and I'm like "what the fuck is junebug32?"

    "Looks like that server is running something called Junebug 3.2."
    "What the fuck is Junebug?"
    "I dunno. Let's ask the whole department. Does anybody know what Junebug is?"
    "I dunno but I think maybe James does. He's out of the office today."
    "I think it might be something for HR?"

    our old server naming scheme was/is greek gods. Atlas, cerberus, apollo, artemis, hades, hephaestus are all servers I've worked on in the past 2 days.

    One of the very first things I did here was start retiring servers with Star Trek names and replacing them with grownup names.

    That included but isn't limited to servers named KIRK, SPOCK, UHURA, MCCOY, PIKE, PICARD, SULU, SCOTTY, CHEKOV, ARCHER, RIKER, DATA, and LAFORGE.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    Capt HowdyCapt Howdy Registered User regular
    What's bugging me most is that I can't (not allowed) to figure out why the Veritas backups aren't backing up the Oracle databases, yet. Nor can I try to figure out why the raid drive refuses to accept a hard drive they hot-swapped a few months ago after one failed.

    I'm being given a list of problems that I can't work on yet. The admin who set everything up wasn't a note taker, and from what little bit I have seen, was a bit of a Dr Frankenstein with how he configured everything. Pretty sure when it's all said and done I'll wind up nuking the servers from orbit by doing fresh reinstalls of both backup servers. Hopefully with something other than Veritas.

    Steam: kaylesolo1
    3DS: 1521-4165-5907
    PS3: KayleSolo
    Live: Kayle Solo
    WiiU: KayleSolo
  • Options
    CogCog What'd you expect? Registered User regular
    So this thing finally showed up. I don’t even remember how long ago it was that we saw this.

    l2hwm5l4xphv.jpeg

  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Cog! You're alive!

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    CogCog What'd you expect? Registered User regular
    wunderbar wrote: »
    Cog! You're alive!

    For some values of ‘alive’.

  • Options
    Gilbert0Gilbert0 North of SeattleRegistered User regular
    Must have been a new batch. I got mine Friday too :)

  • Options
    ArcSynArcSyn Registered User regular
    Oh wow, I got mine early-mid December!

    4dm3dwuxq302.png
  • Options
    DarkewolfeDarkewolfe Registered User regular
    I usually don't go into details, but seriously. I have a team that started freaking out about not being informed about patching-driven node reboots on an always on cluster. You know, one of the reasons you move to clustering.

    What is this I don't even.
  • Options
    lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Installed some G3 Pro Cameras yesterday. I love the resolution and the zoom.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Cog wrote: »
    wunderbar wrote: »
    Cog! You're alive!

    For some values of ‘alive’.

    We hadn't heard from you in a few weeks. I was starting to think you were just really engrossed in some new install guide.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    StraygatsbyStraygatsby Registered User regular
    New job is exciting! So much to learn!
    New job is terrifying! So much damage I can do!

    Right now I'm baby sea turtle making a beach run for the ocean, but between me and the water is that endless hallway from The Shining.

    And then when I get to the water, I can still be eaten alive any second.




    Good times!?


  • Options
    InfidelInfidel Heretic Registered User regular
    Darkewolfe wrote: »
    I usually don't go into details, but seriously. I have a team that started freaking out about not being informed about patching-driven node reboots on an always on cluster. You know, one of the reasons you move to clustering.

    If we're talking about a cluster not able to survive rolling reboots, then yeah, what the fuck are you even doing.

    OrokosPA.png
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Oh my god, one VM in my highly redundant farm was rebooted!

    *flails arms like Kermit the frog*

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FFFF Once Upon a Time In OaklandRegistered User regular
    I...ok this is going to be dumb so I'm just going to apologize in advance.

    DNS aging/scavenging. Is there any reason it would or should not be enabled in a relatively typical environment? I'm starting to discover that it's not enabled (via it causing my group some issues with bad records). We asked that it be enabled in one of our areas and the update rate was set to 1 minute, which also seems bad.

    Is...is all of the above bad? Am I taking crazy pills? I sort of feel like I might be.

    Huh...
  • Options
    FeldornFeldorn Mediocre Registered User regular
    That update rate... but yes, turn on scavenging.

    I spent a couple days when I started here cleaning up all the old static records after getting scavenging working.

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    What do you mean by "update rate?"

    Like, scavenging refresh rate? Yeah, 1 minute is crazypantswtf.

    Sometimes you might have a DNS record with a 1 minute TTL for some high-availability stuff.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    I just found out/voluntold to stay after hours tomorrow to babysit a jr sysadmin who is doing some upgrades on a couple systems we've put him in charge of. He's nervous has he's never done an after hours server upgrade of this scale before (version upgrade on $system) and wants some backup if things go totally sideways. On one hand it means I have to put in like a 12 hour day tomorrow.

    On the other hand, 4 hours of 1.5 banked time to basically sit and watch him work, and only have to do something if he completely fucks something up. If he just mildly fucks something up he has to fix it himself, and hell even if he completely fucks something up it's still on him to fix, just with my help.

    as I was typing this I was slightly annoyed about having to do it but then I remembered the first time I ever had to do something similar I also had someone hanging around on standby in case I asploded something badly, so I guess I'm paying it forward.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    FFFF Once Upon a Time In OaklandRegistered User regular
    Feral wrote: »
    What do you mean by "update rate?"

    Like, scavenging refresh rate? Yeah, 1 minute is crazypantswtf.

    Sometimes you might have a DNS record with a 1 minute TTL for some high-availability stuff.

    That's what I'm assuming. "I have set entries of refresh record low so it update in a minute for *the domain we use*"

    That's the direct quote from the network admin when asked wtf was going on in the first place (we had loads of old records of machines that were not physically present anymore). I suspect it's only actually set up on the zone that we're in, but I couldn't say for sure.

    We just wanted to be able to find a machine by it's name. :?

    Huh...
  • Options
    twmjrtwmjr Registered User regular
    wunderbar wrote: »
    I just found out/voluntold to stay after hours tomorrow to babysit a jr sysadmin who is doing some upgrades on a couple systems we've put him in charge of. He's nervous has he's never done an after hours server upgrade of this scale before (version upgrade on $system) and wants some backup if things go totally sideways. On one hand it means I have to put in like a 12 hour day tomorrow.

    On the other hand, 4 hours of 1.5 banked time to basically sit and watch him work, and only have to do something if he completely fucks something up. If he just mildly fucks something up he has to fix it himself, and hell even if he completely fucks something up it's still on him to fix, just with my help.

    as I was typing this I was slightly annoyed about having to do it but then I remembered the first time I ever had to do something similar I also had someone hanging around on standby in case I asploded something badly, so I guess I'm paying it forward.

    man, I just had to risk the wrath of the guy who was trying to drink his beer if I screwed up. must be nice to have someone waiting to help!

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Literally the worst thing about Microsoft Outlook is it's lack of animated gif support.

    This is seriously handicapping my ability to manage with memes.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    DarkewolfeDarkewolfe Registered User regular
    Infidel wrote: »
    Darkewolfe wrote: »
    I usually don't go into details, but seriously. I have a team that started freaking out about not being informed about patching-driven node reboots on an always on cluster. You know, one of the reasons you move to clustering.

    If we're talking about a cluster not able to survive rolling reboots, then yeah, what the fuck are you even doing.

    No, it's completely capable of surviving. It's been rebooted a thousand times without the user group knowing. They just lost their shit when they found out there was a node reboot because they persist in refusing to learn how anything works while simultaneously holding strong opinions about it.

    What is this I don't even.
  • Options
    SiliconStewSiliconStew Registered User regular
    FF wrote: »
    I...ok this is going to be dumb so I'm just going to apologize in advance.

    DNS aging/scavenging. Is there any reason it would or should not be enabled in a relatively typical environment? I'm starting to discover that it's not enabled (via it causing my group some issues with bad records). We asked that it be enabled in one of our areas and the update rate was set to 1 minute, which also seems bad.

    Is...is all of the above bad? Am I taking crazy pills? I sort of feel like I might be.

    Yes, it probably should be on, and no it shouldn't need to be that aggressive.

    There's 3 different periods involved in scavenging. You've got a "no refresh" period, a "refresh" period, and "scavenging" period.

    A live DNS client tries to refresh it's DNS registration every 24 hours. A "refresh" means its IP address hasn't changed. The DNS server prevents "refresh" registrations between the client's last valid registration timestamp and timestamp + "no refresh" period, typically a couple days. After the "no refresh" period is up, DNS clients are allowed to update their registrations any time during the "refresh" period. Once they do, the timestamp is updated and the client goes back to the start of their "no refresh" period.

    If a DNS client fails to update its registration within last timestamp + "no refresh" period + "refresh" period, it's presumed dead and becomes eligible to scavenge. The record will be removed once the scavenging process runs again, which happens every time the "scavenging period" is up, again typically every few days. So worst case, a record will be removed after the total of "no refresh" + "refresh" + "scavenge" periods. Best case, it will be removed after "no refresh" + "refresh" periods.

    Fast scavenging (that is, low refresh periods) might make sense if your environment has a really high number of short-lived clients, but in my opinion, you don't want clients to enter scavenging eligibility in less than 48 hours. Since they may only re-register every 24 hours you want to give them time to do so.

    How often the scavenging (that is, deletion) actually runs (every few days vs every few minutes) generally isn't that important other than for performance reasons. I couldn't see ever needing to do it more often than every few hours. Again, in a large short-lived client environment, you might want it to run more often to keep things clean.

    Additionally, depending on how DHCP updates DNS, DHCP lease times should typically match your "no refresh" + "refresh" times so they both age out at about the same time to cut down on duplicate IP entries in DNS.

    Just remember that half the people you meet are below average intelligence.
  • Options
    RadiationRadiation Registered User regular
    Darkewolfe wrote: »
    Infidel wrote: »
    Darkewolfe wrote: »
    I usually don't go into details, but seriously. I have a team that started freaking out about not being informed about patching-driven node reboots on an always on cluster. You know, one of the reasons you move to clustering.

    If we're talking about a cluster not able to survive rolling reboots, then yeah, what the fuck are you even doing.

    No, it's completely capable of surviving. It's been rebooted a thousand times without the user group knowing. They just lost their shit when they found out there was a node reboot because they persist in refusing to learn how anything works while simultaneously holding strong opinions about it.

    I'm shocked to see this attitude of people working for/with/around the government.
    SHOCKED!

    I thought you said you had a pretty solid team technically speaking?

    PSN: jfrofl
  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    A few survey questions:
    1. Do you have Contractor WiFi?
    2. Do you filter your Contractor WiFi?
    3. This is the big one: Do you allow employees to use said WiFi (or another WiFi)?
    4. Do you filter employees if they do have WiFi access?

  • Options
    ThawmusThawmus +Jackface Registered User regular
    What's Contractor Wifi?

    Twitch: Thawmus83
  • Options
    a5ehrena5ehren AtlantaRegistered User regular
    Feral wrote: »
    Literally the worst thing about Microsoft Outlook is it's lack of animated gif support.

    This is seriously handicapping my ability to manage with memes.

    Teams has built-in Giphy search, FWIW. Not that I can get anyone here to actually use it.

  • Options
    SeidkonaSeidkona Had an upgrade Registered User regular
    Thawmus wrote: »
    What's Contractor Wifi?

    WiFi that does none of the work but takes all the credit?

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • Options
    jungleroomxjungleroomx It's never too many graves, it's always not enough shovels Registered User regular
    edited March 2018
    Thawmus wrote: »
    What's Contractor Wifi?

    WiFi that claims it can push a gigabyte/s through with $50 devices but ends up not being able to?

    jungleroomx on
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Thawmus wrote: »
    What's Contractor Wifi?

    Wifi with a sales team capable of selling BS products and services to C-level people who don't understand what wifi is.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    Thawmus wrote: »
    What's Contractor Wifi?
    Like, hey, we have this wifi here for Contractors to use while they're on-site, which is segregated off from the internal network. Guest WiFi but they have to sign a mostly worthless document saying they won't mess up our stuff.

  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    I have to remember that a lot of you guys work for MSPs rather than internal IT, and may not have this particular issue.

  • Options
    ThawmusThawmus +Jackface Registered User regular
    Thawmus wrote: »
    What's Contractor Wifi?
    Like, hey, we have this wifi here for Contractors to use while they're on-site, which is segregated off from the internal network. Guest WiFi but they have to sign a mostly worthless document saying they won't mess up our stuff.

    Ah, okay, guest wifi, got you. That'd be a "no" across the board, though I feel strongly that we should be a "yes" on #1 and 3. Also we get sales vendors who want to do demos all the time and need WiFi and my hands are tied on delivering it. It's frustrating.

    Like, I can leverage about 600 Mbps of actual DIA service to pretty much wherever the fuck I want, you can't tell me this would drain resources.

    Twitch: Thawmus83
  • Options
    MugsleyMugsley DelawareRegistered User regular
    Wifi that underbids a job, then piles on charges based on "emergent" issues (that were totally known when you started)?

  • Options
    twmjrtwmjr Registered User regular
    1. Do you have Contractor WiFi? Guest, yes - it's sponsored (meaning an internal employee has to create an account for the visitor) with some exceptions -- e.g. contractors with internal company IDs can access it directly with their ID.
    2. Do you filter your Contractor WiFi? Yes.
    3. This is the big one: Do you allow employees to use said WiFi (or another WiFi)? No - there is a plan to allow this for personal devices (i.e. non-work smartphones etc.) on a separate SSID (same back end infrastructure), but it's not something we're doing today.
    4. Do you filter employees if they do have WiFi access? Yes, employees would still be filtered on the proposed new solution.

  • Options
    SeidkonaSeidkona Had an upgrade Registered User regular
    Ever feel like everyone coming to you with a problem is like a kid who broke their toy and wants you to fix it. Out stretched arms and tears included..

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    twmjr wrote: »
    1. Do you have Contractor WiFi? Guest, yes - it's sponsored (meaning an internal employee has to create an account for the visitor) with some exceptions -- e.g. contractors with internal company IDs can access it directly with their ID.
    2. Do you filter your Contractor WiFi? Yes.
    3. This is the big one: Do you allow employees to use said WiFi (or another WiFi)? No - there is a plan to allow this for personal devices (i.e. non-work smartphones etc.) on a separate SSID (same back end infrastructure), but it's not something we're doing today.
    4. Do you filter employees if they do have WiFi access? Yes, employees would still be filtered on the proposed new solution.
    Yeah, I mean, that's where I'm headed basically. We have some users employees who were using a contractor's username and password, who then complained TO US IN THE IT DEPT that it quit working when we deleted the contractor (due to them going out of business and leaving the premises).

    I mean, personally, I don't care if it is filtered or not, because like, people can use the data on their phones to get to porn or whatever, but my boss is a bit paranoid about liability.

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    1) Yes, we have contractor wifi.
    2) we have some basic antivirus and anti-malware filtering at our firewall. It blocks a few things. We don't have a separate web filter on our contractor wifi.
    3) we have a separate SSID for employee-owned devices, but it runs on the same infrastructure as the contractor wifi.
    4) See #2.

    Honestly, I think this situation works out pretty well for us.

    It also part of the reason we can get away with strict web filtering on our company-owned devices. People can use the employee/contractor wifi on their phones with minimal interference.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    We also turn on guest isolation and we put a per-device bandwidth limit so one guy torrenting the entirety of Game of Thrones won't fuck up everybody else.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
This discussion has been closed.