As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

[sysadmin] on-call schedule - Always you

145791020

Posts

  • SiliconStewSiliconStew Registered User regular
    edited January 2022
    Our EOL stuff is mostly due to mergers and ERP archives where they never want to pay for the data conversion to the current system, do not want to continue to pay for software maintenance of the old system to keep it updated or have support for migrating it to a newer OS, but they still need to occasionally reference accounting data for the next decade. All we can really do is network isolate and restrict account access to the extent possible and even keep the systems powered off unless a request comes in to access them temporarily.

    Otherwise it's the occasional vendor-supplied equipment who still think XP is all the OS you'll ever need and no their stuff won't run without full admin permissions and you'd be foolish to ask. Though we've mostly fixed this over the last couple years through IT's efforts to convince management that million dollar shipments and severe penalties for their delay dependent on a decade old PC isn't a great business practice.

    SiliconStew on
    Just remember that half the people you meet are below average intelligence.
  • DarkewolfeDarkewolfe Registered User regular
    edited January 2022
    I mostly agree but I also think MFA is close to universally required now. Turns out using two related text string hashes as your only security is iffy.

    Darkewolfe on
    What is this I don't even.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Coincidentally, we don't currently have MFA on our hosted chat system. It's not Slack or Teams, but it's very similar and it's provided by our cloud VOIP company. It works very well.

    ...usually. Today, users are intermittently getting MFA prompts when attempting to log in to it. We didn't enable MFA on it (yet) but it's been on our radar.

    Yay! Cloud!

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • NaphtaliNaphtali Hazy + Flow SeaRegistered User regular
    Feral wrote: »
    Coincidentally, we don't currently have MFA on our hosted chat system. It's not Slack or Teams, but it's very similar and it's provided by our cloud VOIP company. It works very well.

    ...usually. Today, users are intermittently getting MFA prompts when attempting to log in to it. We didn't enable MFA on it (yet) but it's been on our radar.

    Yay! Cloud!

    sounds cloudy with a change of mfaballs today

    Steam | Nintendo ID: Naphtali | Wish List
  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    How I hate doing tech support for my relatives.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • ThawmusThawmus +Jackface Registered User regular
    God fucking dammit if your site isn't on my network and you ask me to generate a CSR for you, you deserve everything you've got coming to you.

    Why would I trust you to know what the fuck you're doing with the keys I give you if you can't even generate a CSR?

    Twitch: Thawmus83
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Thawmus wrote: »
    God fucking dammit if your site isn't on my network and you ask me to generate a CSR for you, you deserve everything you've got coming to you.

    Why would I trust you to know what the fuck you're doing with the keys I give you if you can't even generate a CSR?

    Coincidentally I've been training folks in my department how to do CSRs.

    Because until I started doing so, I was the only person in my IT department who understood how to generate a CSR and install a cert.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    “But Doctor, I am the Senior Sys Admin.”

  • wunderbarwunderbar What Have I Done? Registered User regular
    Feral wrote: »
    Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.

    the only caveat I'll add to that is in the past when you could get a cert for 5 years it's just not something you had to do very often. Especially if you get a wildcard cert for a domain. I worked a job where managing certs was in my job description and I never ended up having to do it at that job because the certs didn't expire until about 6 months after I left there. So like... I didn't have to deal with certificate renewals for 6+ years.

    Now that you can only get SSL certs for a year it should be a thing we deal with on a regular basis, so, you know, remember how to do it.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • ThawmusThawmus +Jackface Registered User regular
    wunderbar wrote: »
    Feral wrote: »
    Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.

    the only caveat I'll add to that is in the past when you could get a cert for 5 years it's just not something you had to do very often. Especially if you get a wildcard cert for a domain. I worked a job where managing certs was in my job description and I never ended up having to do it at that job because the certs didn't expire until about 6 months after I left there. So like... I didn't have to deal with certificate renewals for 6+ years.

    Now that you can only get SSL certs for a year it should be a thing we deal with on a regular basis, so, you know, remember how to do it.

    Kinda. Renewals, absolutely. But you don't need CSR's until you get new certs. So people still end up out of practice.

    Someone was asking me the other day how to generate a CSR on Windows.

    So I'm sure you heard some laughter off in the distance that day. Sorry. That was me.

    Twitch: Thawmus83
  • SiliconStewSiliconStew Registered User regular
    Like most procedures, renewing certs is solved with a bit of documentation.

    Just remember that half the people you meet are below average intelligence.
  • FeldornFeldorn Mediocre Registered User regular
    Like most procedures, renewing certs is solved with a bit of documentation.

    This.

    Everything I need to do once per year I make a page for in our team OneNote so that I don't have to slog through finding the one blog that actually explained it correctly, only to find it's now a Parked Domain :cry:

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited January 2022
    Like most procedures, renewing certs is solved with a bit of documentation.

    i would love it if i could get people to consistently read and follow documentation

    let alone apply a modicum of independent thought when reality diverges from the documentation

    "what does invalid key size mean?"

    (it means you skipped the part in the documentation that told you to change the key size from the default 1024 to 2048)

    "okay my cert still isn't working. halp."

    (well, I just looked up the knowledgebase for the software that I am not the SME in, you're the supposed SME in, and it explicitly says that certificates must be in PFX format, and yours is in PEM format. so maybe fix that.)

    "how do I fix that?"

    (redownload it from the CA)

    "how do I do that?"

    oh ffs...

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited January 2022
    documentation: How to Make a Peanut Butter & Jelly Sandwich

    1) Open a jar of peanut butter.
    2) Obtain a slice of bread.
    3) Using a butter knife, apply peanut butter smoothly to one side of that slice of bread.
    4) Place that slice of bread on a plate for now.
    5) Open a jar of jelly.
    6) Obtain a NEW slice of bread.
    7) Using a butter knife, apply jelly smoothly to one side of the new slice of bread.
    8) Put both slices of bread together.

    Result:

    tltsxr4rd2n6.png

    "feral, halp. the users are complaining that their fingers are sticky!"

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    wunderbar wrote: »
    Feral wrote: »
    Getting requests from sysadmins with "senior" in their titles to help them install SSL certs on systems I don't manage was causing me to die a little inside.

    the only caveat I'll add to that is in the past when you could get a cert for 5 years it's just not something you had to do very often. Especially if you get a wildcard cert for a domain. I worked a job where managing certs was in my job description and I never ended up having to do it at that job because the certs didn't expire until about 6 months after I left there. So like... I didn't have to deal with certificate renewals for 6+ years.

    Now that you can only get SSL certs for a year it should be a thing we deal with on a regular basis, so, you know, remember how to do it.

    The best practice now is that it should be scripted wherever you can do so.

    Most (all?) of the major CAs have APIs now and instructions on how you can script the renewals.

    I haven't gotten around to scripting them in my environment yet, but it's on the agenda.

    But even if we can script it across all of our Windows and Linux VMs, there's still going to be oddball (virtual and physical) appliances and other oneoffs that will need to be manual. And for those, I'm not writing up documentation on each and every snowflake system, so I need people to understand the concepts and make independent decisions.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • NosfNosf Registered User regular
    Check this shit out:

    We merged with another smaller org, 70ish users. They have a couple 2008 servers and a 2010 exchange server. Yikes. Ok, let's get them onto our 2016 on prem Exchange and then down the road we all go to 365. I'll need to pre-migrate all the email and then repoint the MX and then do my final migration. We want to blast the new remote agent onto all those machines so we just take a day, do them all by hand and say hello to all the users.

    Oh, someone else controls your domain reg? Ok, let's take back control of that. Ok, so who controls your apparently Cloudflare DNS? You don't know? We spend 2 weeks digging and never find it? National head office who obviously controls it has no record and repeatedly states they don't? Cool, cool. Ok, we can add you to our DNS, get all the records we know of because you're small and your setup is very simplistic and then flip the name servers day of, and they'll propogate out over a few hours. Wait, didn't you have a cert on your OWA? I know you did, because the migration tool requires that. Oh, you had an undocumented reverse proxy setup in Cloudflare with the cert on it.

    ...

    Are you kidding me?

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    so many yikes

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • MyiagrosMyiagros Registered User regular
    edited January 2022
    I've been running into too many instances of software throwing "System.OutOfMemoryException" errors recently. I've come across two different programs that use SQL on the backend doing it, and now a video wall program. Seems to have all started happening since mid-late December and all of the software vendors have no answers.

    Edit: It's BitDefender causing it: https://community.bitdefender.com/en/discussion/89709/out-of-memory-error

    Myiagros on
    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited January 2022
    VMware support: "Please run this list of curl commands on the appliance to diagnose the problem."

    > curl -I https://www.vmware.com
    HTTP/1.1 200 OK

    > curl -I https://cloud.vmware.com
    HTTP/1.1 200 OK

    > curl -I https://fml.vmware.com
    HTTP/1.1 200 OK

    > curl -I https://cloud.fml.wtf.lol.bbq.vmware.com
    HTTP/1.1 200 OK

    > curl -I https://the.vmware.appliance.my.trouble.ticket.is.for.feral.com
    Failed to connect to 10.69.4.20 port 443 after 1024 ms: Connection refused

    VMware support: "The problem is that the appliance can't connect to the vmware cloud. please make sure your firewall is not blocking it."

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    This VMware virtual appliance runs its own internal kubernetes cluster. During the setup process, the setup scripts configure the kubernetes cluster with its own internal IP schema, and there's an nginx instance inside kubernetes that accepts traffic from outside kubernetes.

    I'm not knowledgeable about kubernetes, but I can at least tell that when I SSH into the VM, run any kind of networking tests from the OS, the networking works... except when trying to connect to the nginx instance running inside kubernetes.

    This appliance is supposed to bridge the gap between a cloud VDI pod and an on-premises VDI pod. We have no cloud VDI. We only need this appliance for one stupid purpose: to enable the Horizon licenses we've already bought and paid for.

    Yaaayyy cloud

    It's so reliable

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    VMware, by email: "We've opened a Zoom session for you. Please join at..."
    Me, 15 minutes later, sees email. Attempts to join.
    Immediately receives another email: "Zoom session closed due to lack of customer response."

    This fucking industry.

    This whole fucking industry.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited January 2022
    OH FOR FUCKS SAKE

    Finally get on the Zoom call, the technician opens up a console session on the appliance and from the appliance's command line:
    curl -I https://localhost/vmware/stuff
    Failed to connect to localhost port 443 after 1024 ms: Connection refused
    

    VMware technician: "This shows us that the appliance does not have external connectivity"

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • DrovekDrovek Registered User regular
    Feral wrote: »
    OH FOR FUCKS SAKE

    Finally get on the Zoom call, the technician opens up a console session on the appliance and from the appliance's command line:
    curl -I https://localhost/vmware/stuff
    Failed to connect to localhost port 443 after 1024 ms: Connection refused
    

    VMware technician: "This shows us that the appliance does not have external connectivity"

    a57j9d6hqlcy.png

    steam_sig.png( < . . .
  • bowenbowen How you doin'? Registered User regular
    Feral wrote: »
    OH FOR FUCKS SAKE

    Finally get on the Zoom call, the technician opens up a console session on the appliance and from the appliance's command line:
    curl -I https://localhost/vmware/stuff
    Failed to connect to localhost port 443 after 1024 ms: Connection refused
    

    VMware technician: "This shows us that the appliance does not have external connectivity"

    Shit like this makes me able to get rid of my imposter syndrome.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    "okay but you need to have command-line access, right? through SSH or straight to the console?"

    yeah but if you combine this with an arbitrary code execution technique, maybe one that is also very commonplace on linux like OH I DUNNO LOG4J

    i'll be in a literal blanket and pillow fort with my cat and a teddy bear and a bottle of whiskey if anybody needs me

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • LD50LD50 Registered User regular
    Feral wrote: »
    "okay but you need to have command-line access, right? through SSH or straight to the console?"

    yeah but if you combine this with an arbitrary code execution technique, maybe one that is also very commonplace on linux like OH I DUNNO LOG4J

    i'll be in a literal blanket and pillow fort with my cat and a teddy bear and a bottle of whiskey if anybody needs me

    I was complaining to my friend that computers were a mistake yesterday. I want to go hide in the woods.

  • ThawmusThawmus +Jackface Registered User regular
    What I can't understand is who the fuck uses pkexec and why

    Twitch: Thawmus83
  • twmjrtwmjr Registered User regular
    critical service: im going to go ahead and break at 4:30; that good for you? you weren't hoping to get off work, right?

  • InfidelInfidel Heretic Registered User regular
    Surprise visit to the data centre today because somehow one of the servers locking up was breaking the firewall/VPN preventing remote access! :rotate:

    OrokosPA.png
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    the good news is that on full VMs with real grown-up OSes you should be able to yum or apt yourself a new pkexec or polkit and be happy

    you can also remove pkexec or polkit in a lot of cases and be happy

    the problem is going to be, again, fucking virtual appliances and other stuff running some form of linux (possibly embedded, maybe iot) where the admin has to rely on the vendor for patching

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    and of course the virtual paperwork and all the questions

    does our vulnerability scanner pick up pwnkit?

    yes, dad

    have you done a vuln scan for pwnkit?

    yeessss, dad

    are you mitigating...

    yessss daaaaaaad

    are your tracking the mitigations...

    yyyyyyessssssss daaaaadddddd gawd leave me alone

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • bowenbowen How you doin'? Registered User regular
    great now my day tomorrow is going to be annoying, thanks feral

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • CarpyCarpy Registered User regular
    Good week to take the OSCP exam though

  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Filling out HIPAA audit forms is the most mind number experience imaginable. It's dozens of pages of questions like that, clearly written by a lawyer instead of a tech.

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    That_Guy wrote: »
    Filling out HIPAA audit forms is the most mind number experience imaginable. It's dozens of pages of questions like that, clearly written by a lawyer instead of a tech.

    I've done HIPAA, Sarbanes-Oxley, and financial regulatory exams & auditing.

    HIPAA is pretty bad. Financial industry audits are the same flavor, but even worse.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • ThawmusThawmus +Jackface Registered User regular
    Whereas I sit here without such garbage, kinda wishing I had such garbage so that I could use it to enforce some sensible policy.

    Twitch: Thawmus83
  • bowenbowen How you doin'? Registered User regular
    Feral wrote: »
    That_Guy wrote: »
    Filling out HIPAA audit forms is the most mind number experience imaginable. It's dozens of pages of questions like that, clearly written by a lawyer instead of a tech.

    I've done HIPAA, Sarbanes-Oxley, and financial regulatory exams & auditing.

    HIPAA is pretty bad. Financial industry audits are the same flavor, but even worse.

    The "I'm not storing unencyprted credit cards" 200 question quiz is probably the worst one to do every year.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited January 2022
    Thawmus wrote: »
    Whereas I sit here without such garbage, kinda wishing I had such garbage so that I could use it to enforce some sensible policy.

    oh, i definitely prefer working in regulated industries.

    the bureaucracy and paperwork is awful, specially for PCI, but at least regulated companies take cybersecurity and reliability seriously

    at least, more often and more seriously than unregulated companies

    i'd rather have a predictable but shitty PCI or HIPAA or Dept of Treasury exam once or twice a year than have to convince a computer-illiterate CEO why the latest ransomware outbreak was an indirect result of him cutting IT headcount down to me and an intern

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
Sign In or Register to comment.