As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

[Sysadmin] Solarwinds compromised.

17980818385

Posts

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Also, today, I had to explain to our new IT helpdesk manager what "TLS" means

    rnzt65z3fzq6.png

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    IncenjucarBlackDragon480lwt1973
  • BlackDragon480BlackDragon480 Bluster Kerfuffle Master of Windy ImportRegistered User regular
    SSL 3 has only been deprecated for like 6 years, can't expect them to have gotten up on it's replacement yet.

    First they came for the Muslims and we said...NOT TODAY MOTHERFUCKERS!
  • ThawmusThawmus Registered User regular
    I gotta say I don't know what TLS stands for off the top of my head.

    steam_sig.png
  • Inquisitor77Inquisitor77 2 x Penny Arcade Fight Club Champion A fixed point in space and timeRegistered User regular
    Team Liquid Sucks

    bowenBlackDragon480Thawmus
  • zagdrobzagdrob Registered User regular
    edited March 3
    Thawmus wrote: »
    I gotta say I don't know what TLS stands for off the top of my head.

    Even if I didn't know what the acronym stood for I would still take .5 seconds googling it before I showed my ass.

    Edit - in a professional setting ofc.

    zagdrob on
  • FeldornFeldorn Mediocre Registered User regular
    zagdrob wrote: »
    Thawmus wrote: »
    I gotta say I don't know what TLS stands for off the top of my head.

    Even if I didn't know what the acronym stood for I would still take .5 seconds googling it before I showed my ass.

    Edit - in a professional setting ofc.

    You show your ass in professional meetings?

    Damn, I still stop at snarky comments.

    Genshin Impact: 600428730
    steam_sig.png
    FeralDrovekBlackDragon480Mugsley
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    SSL 3 has only been deprecated for like 6 years, can't expect them to have gotten up on it's replacement yet.
    You’re making a pretty big assumption that they know what SSL was.

    BlackDragon480
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    It wasn't so much that he needed to know what TLS stands for, or that they needed to know that it was just SSL's new-ish name.

    It's that he didn't know, conceptually, what TLS does or why you need it.

    To his credit, once I explained it, he understood it immediately. He's a smart guy, I just don't think he's ready to be an IT helpdesk manager. It's basically this problem again:
    Feral wrote: »
    I talk shit about these folks but they are, sincerely, hard-working and smart. We just hire people and promote people into positions where they are unsuited. We tell ourselves, "Oh, we'll train them later," and then we never do. Then they end up extremely busy, but mostly because of inefficiencies and technical debt either they generated, or other people (who are also promoted into positions in which they're unsuited) generated.

    If I'm Dr. Cox in this scenario, then imagine that 10% of the "doctors" at this (unregulated) hospital have MDs. Because MDs are expensive. The other 90% of people with "physician" or "doctor" title are actually nurses. Then the people with "nurse" titles don't actually have nursing degrees, they were just front desk workers or janitors who expressed an interest in medicine so we threw them in to nursing jobs with no training. Almost everybody is working a job 1-2 levels above their training & experience.

    We hired somebody to be an IT helpdesk manager whose only professional experience was being a helpdesk technician... in college. He's only had the one job, and he's never worked helpdesk outside of college.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    lwt1973
  • AiouaAioua Ora Occidens Ora OptimaRegistered User regular
    now see, I wouldn't even necessarily mind if the help desk manager wasn't particularly techy (really just need enough to not get bullshitted), as long as they were good at managing
    keep the queue short and the users happy and the projects delivered on time? no worries


    but ofc if they're just... fresh outta college maybe not gonna be that either

    life's a game that you're bound to lose / like using a hammer to pound in screws
    fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
    that's right we're on a fucked up cruise / God is dead but at least we have booze
    bad things happen, no one knows why / the sun burns out and everyone dies
    FeralBlackDragon480ThawmusDizzy D
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Six months ago before new helpdesk manager was hired, I was talking to my boss about how we really need somebody on our team who can do desktop engineer tasks. Y'know, manage Windows images, Windows Updates, can at least find their way around DISM and MDT and WDS with a flashlight and a map, basic Powershell, doesn't stare at me blankly when I tell them to silently push out an MSI, that sort of thing.

    My boss agreed, and said they were going to find somebody with that experience for their next hire.

    The next hire was the new helpdesk manager, and when he started I asked him what his desktop experience and skills were like. And he responded "well, this is my first job after college and I have a lot to learn about that, so I hope you don't mind me asking lots of questions"

    me:

    giphy.gif

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    BlackDragon480MugsleyFeldorn
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Thawmus wrote: »
    I gotta say I don't know what TLS stands for off the top of my head.

    I do. But that's only because the boilerplate text in all of the vulnerability reports that our vulnerability scanner outputs actually spells out the phrase, "Transport Layer Security"

    so when it finds for example a VM accepting HTTPS connections with TLS 1.0, it produces a nice slick report that says "Vulnerability: SERVER069 supports Transport Layer Security (TLS) 1.0. Transport Layer Security (TLS) 1.0 and 1.1 are deprecated. Remediation: Enable Transport Layer Security (TLS) 1.2 or higher, then disable Transport Layer Security (TLS) 1.0 and 1.1. The exact procedure to accomplish this is dependent on your operating system or application."

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular


    Transport Layer Security and Transport Layer Security starring Transport Layer Security aaaand TransportLayerSecurity Transport Layer Transport Layer Security TransportLayer Security Secuuurrrrrity

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    BlackDragon480Thawmus
  • DarkewolfeDarkewolfe Registered User regular
    I was... not expecting that in this thread.

    What is this I don't even.
  • SeidkonaSeidkona Had an upgrade Registered User regular
    Alrighty then. . . .

    Lol

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • bowenbowen How you doin'? Registered User regular
    Feral wrote: »
    Six months ago before new helpdesk manager was hired, I was talking to my boss about how we really need somebody on our team who can do desktop engineer tasks. Y'know, manage Windows images, Windows Updates, can at least find their way around DISM and MDT and WDS with a flashlight and a map, basic Powershell, doesn't stare at me blankly when I tell them to silently push out an MSI, that sort of thing.

    My boss agreed, and said they were going to find somebody with that experience for their next hire.

    The next hire was the new helpdesk manager, and when he started I asked him what his desktop experience and skills were like. And he responded "well, this is my first job after college and I have a lot to learn about that, so I hope you don't mind me asking lots of questions"

    me:

    giphy.gif

    Y'all are essentially looking for like a system admin II for a help desk manager, you're going to have a hard time because absolutely no one in their right mind wants to do that job even a little bit... especially not a system admin.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
    ThawmusDizzy DBlackDragon480zagdrobMvrck
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    bowen wrote: »
    Feral wrote: »
    Six months ago before new helpdesk manager was hired, I was talking to my boss about how we really need somebody on our team who can do desktop engineer tasks. Y'know, manage Windows images, Windows Updates, can at least find their way around DISM and MDT and WDS with a flashlight and a map, basic Powershell, doesn't stare at me blankly when I tell them to silently push out an MSI, that sort of thing.

    My boss agreed, and said they were going to find somebody with that experience for their next hire.

    The next hire was the new helpdesk manager, and when he started I asked him what his desktop experience and skills were like. And he responded "well, this is my first job after college and I have a lot to learn about that, so I hope you don't mind me asking lots of questions"

    me:

    giphy.gif

    Y'all are essentially looking for like a system admin II for a help desk manager, you're going to have a hard time because absolutely no one in their right mind wants to do that job even a little bit... especially not a system admin.

    Oh, I'm under no delusions that a good help desk manager is also going to be a good desktop engineer and vice versa.

    It's more a representation of the general trend: we need people with certain technical skills, but we habitually fail to hire people with those skills and we fail to hire people who are close enough to those skills to learn them in a timely manner.

    Even when I get agreement from management that we need a certain type of technical hire, it doesn't manifest because between the conversation where IT says "we need somebody with X tech skills" and the final offer we make to our candidates, we discover that people with real tech skills don't want to make the 20%-33% under market we offer them.

    If we had hired a network engineer instead and my boss came to me and said "we hired Peter to get all of the Cisco and networking stuff off your plate. I want you to refocus on Windows desktop instead" I would have accepted that.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    bowen
  • bowenbowen How you doin'? Registered User regular
    we discover that people with real tech skills don't want to make the 20%-33% under market we offer them.

    fucking oof

    And for the worst job of the bunch too.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • MyiagrosMyiagros Registered User regular
    I'm kind of in the same boat. I cover so many things - network admin, server admin, help desk, sales, etc, and I get paid well for it but I'm at the point where I'm dealing with at least 15-20 clients regularly and my queue has gone from an average of 20 open tickets to nearly 40. We need another guy but they either have no experience so my boss doesn't want to take the chance on them, or they have all the experience but don't want to wear every hat and within a month or two they are already gone.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
    Feral
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    bowen wrote: »
    we discover that people with real tech skills don't want to make the 20%-33% under market we offer them.

    fucking oof

    And for the worst job of the bunch too.

    @bowen

    Yeah. When I started at this job, I negotiated my salary aggressively, up to the very top of the range they were offering. The top end of their range was exactly median for the job in Seattle at the time. It was more or less like this, with only minor fudging for storytelling purposes:

    "How much money do you want?"
    "I can't work for you for less than $85k."
    "Our salary range for this position is $60k-80k."
    "That's not realistic. $85k is median, and I'm better than your median candidate. $85k is my floor."
    "Will you take $80k?"
    "85."
    "How about $80k, with eligibility for a $5k annual performance bonus?"
    "No, base salary is 85, and I don't need a bonus on top of that."
    "Well, I'm not authorized to hire at 85, but let me talk to our CEO."

    A week later:

    "We're offering you the job at 85."

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    ThawmusSeidkonaDrovekBlackDragon480bowenSpecial KMugsleyFeldornLaOs
  • ThawmusThawmus Registered User regular
    Yeah after my salary has tripled over the past 3 years the basic lesson I've learned is that I was never being paid enough.

    steam_sig.png
    FeralSeidkonaDarkewolfeInfidelRadiationschussBlackDragon480bowenMugsleyLaOs
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Thawmus wrote: »
    Yeah after my salary has tripled over the past 3 years the basic lesson I've learned is that I was never being paid enough.

    Yep. (Very few of us are. And I think that's sort of generally true of everybody in a lot of fields.)

    The lesson for me was that not everybody is going to have comparable chutzpah. An organization that lowballs me will also lowball everybody else, and that's not a recipe for retaining solid talent.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    IncenjucarDarkewolfeFeldorn
  • SeidkonaSeidkona Had an upgrade Registered User regular
    Yeah. I took my job for over their ceiling (by 5k) and made it clear I was doing them a favor for it.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    DarkewolfeFeralFeldorn
  • DarkewolfeDarkewolfe Registered User regular
    Same. This is where my whole advocacy of "interview for jobs you don't really want and be a dick about salary" comes from. It was a life changing revelation to realize how much more I could be making.

    What is this I don't even.
    DrovekFeralIncenjucarSeidkonaShadowfireRadiationBendery It Like BeckhamBlackDragon480Special KMugsley
  • CarpyCarpy Registered User regular
    If any of y'all are running on-prem Exchange and haven't heard go patch your stuff.

    https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/

    Feldornlwt1973
  • NosfNosf Registered User regular
    Just applied CU23 (3-1/2 hours of fucking sweating) and going to apply the patch. Ran the compromise test scripts yesterday, we didn't appear to have been hit.

    FeralFeldorn
  • LD50LD50 Registered User regular
    That is slightly terrifying. Luckily I don't think we're affected as we're hybrid and use the cloud hosted outlook web access and not the self hosted one.

    FeralCarpy
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    You know it's a good day when your email to the C-level executive team and board contain a phrase, only slightly paraphrased, "I do not see evidence that we have been compromised by a nation-state-level advanced persistent threat. However, I must emphasize that uncovering activity by nation-state level adversaries is above my pay grade."

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    IncenjucarDizzy DDrovekShadowfireMugsleyFeldornThawmusLaOszagdrobRadiation
  • NosfNosf Registered User regular
    The actual SU took about 30 minutes or so? Services were slow to come up. I ran the compromise tools a couple times - pre patch it didn't come back with anything. There's a newer one, it came back with about 9 lines - all of which were during the CU install, and one suspicious .zip which I investigated and confirmed it was part of our backup agent's install. Given that the lines referenced were all at midnight today, the 6th, I'd believe those are just a result of the CU install which took place between 10pm on the 5th and 1:30 am on the 6th.

    I found some useful powershell scripts on reddit as well that parse your .aspx for a couple code snippets - didn't come back with anything. All the .aspx files I found dated 5/2019 and appeared to be the usual login/logoff etc.

    Feral
  • MugsleyMugsley Registered User regular
    We're still trying to find someone to help me with my work, in part because the position is considered 'essential,' and so >50% of the work is in the office (high level, I manage multiple contractors doing work on our site that has persistent testing taking place)

    We tried to find someone internally last year and I think the mandatory office work was half the reason we didn't get any applications. The other half was that 4 of us wrote the position description and it ended up a bit bloated (so it could have looked overwhelming)

  • FeldornFeldorn Mediocre Registered User regular
    I’m glad we’re in O365 for this.

    Still patched our internal server used for a couple hybrid tasks, but it isn’t accessible from the internet.

    Installing the latest CU, we were at n-1, still broke OWA and ECP. Spent a couple hours searching for fixes only to find there were included powershell scripts that fixed it.

    Genshin Impact: 600428730
    steam_sig.png
  • taliosfalcontaliosfalcon Registered User regular
    edited March 6
    Bit late to the party with this but re:helpdesk manager; I don’t expect them to know anything technical. The biggest requirement for that job is to be able to get through the day without slitting your wrists at the soul crushing existence your life has become. No one goes into IT with the hope of ending up working at a helpdesk and if you hit manager it’s now a career and not just a temporary stop. May god have mercy on their souls. And being a helpdesk manager basically means you didn’t have the technical aptitude to graduate to anything past helpdesk . I was offered a helpdesk manager job 5 ish years ago (was a field tech at the time) and my response was to say no, quit, and switch to another company doing dev ops it horrified me so much

    taliosfalcon on
  • NosfNosf Registered User regular
    We can't do O365, we use a provincial mail / TLS deal to ensure privacy between orgs and it doesn't work with 365. Dunno if it ever will. Kind of a drag I guess.

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Nosf wrote: »
    We can't do O365, we use a provincial mail / TLS deal to ensure privacy between orgs and it doesn't work with 365. Dunno if it ever will. Kind of a drag I guess.

    A lot of companies do hybrid, where they use O365 as the front-end for employees and as redundant highly-available mailbox servers, but also maintain their own internal on-prem Exchange for those weird-ass use cases.

    (That's what we were moving towards until our board realized that O365 costs money.)

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    Dizzy DLD50
  • NosfNosf Registered User regular
    Yeah, I think we're looking at hybrid eventually - we don't pay for o365 that we do have - tnx charity.

  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Don't you love it when a billion dollar company doesn't have a standard naming convention so your XREF looks like someone got bored partway through and started abbreviating and misspelling things?

    Or is that just me.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
    FeralBlackDragon480Dizzy D
  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    KB5000802 can blue screen if you are printing so that's always a fun thing.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • MyiagrosMyiagros Registered User regular
    edited March 10
    Google-fu is failing me on this one. Looking for a way to deploy a GPO that will give all users a default set of internet shortcuts in Chrome. It's for a terminal server so I'm trying to make it the least hands-on possible for the users.

    Edit: Dig through the GPO settings long enough and you find things. It's under Google > Google Chrome > Managed Bookmarks

    Myiagros on
    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
    Feral
  • Dizzy DDizzy D NetherlandsRegistered User regular
    Just wanted to say that there is a Chrome Managed Bookmarks GPO in the adm template (and it takes some trying to get it to work properly, but I saw in the quote that you already found it yourself).

    Steam/Origin: davydizzy
    Feral
  • MyiagrosMyiagros Registered User regular
    edited March 10
    30 minutes of mucking about with the JSON file and it works! Very handy, definitely going to push Chrome Enterprise on new builds and use this.

    Myiagros on
    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
    Dizzy DFeral
  • yotesyotes Registered User regular
    Just to remind everyone, this could happen to you:

    c36bu4fatjs1.jpg

    https://www.lalsace.fr/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-au-port-du-rhin

    A DC would never completely burn down, they said, they've got fire retardant measures, they said. Also no hoster would ever claim to have DCs that are differently named but built very closely together so having your stuff hosted in differently named zones would at least keep your one pod safe from a fire in another pod, they said.

    Thankfully I only had non-critical stuff there but if I'd had anything productive there that was business critical (and we had to rely on some trick to minimize costs, what could ever happen, they said... it just costs double and extra to have your data safe and accessible, we just can't afford it just do what you can) I would really have hated today.

    This is probably a good point in time to convince my boss that maybe I should have some time to look into our backups and if they actually work and also can be restored.

    [SIGPIC][/SIGPIC]
    ThawmusFeralDrovekBlackDragon480LD50CarpyRadiation
Sign In or Register to comment.