We're going to be adding some advertisements to the forums! If you notice any weirdness around this or spot bad/inappropriate ads, please make a thread in the bugs forum.

[Sysadmin] Solarwinds compromised.

17778798183

Posts

  • RadiationRadiation Registered User regular
    I think this position is a, at best, sideways type move. Job sounds neat, but doesn't seem like it will allow me the growth current job is leveraging currently. Also this was the easiest time it was for me to be super confident in an interview.
    The interview started with "How familiar are you with this tool", and my answer was "I've been working on this tool as a primary function of my job for the past 4 years. So pretty familiar."
    It also helps that I've done a few tech training classes for the tool and was able to go into instructor mode when they started asking questions about how roles in the tool and different.

    PSN: jfrofl
  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    I'm looking for a way to do a search in a Windows' folder for certain keywords and for lacking certain keywords then spit it out in a .txt doc.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • RadiationRadiation Registered User regular
    https://adamtheautomator.com/powershell-grep/

    Looks like allmatches for the match
    and not sure about the ones that don't contain.

    PSN: jfrofl
    Feral
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Radiation wrote: »
    https://adamtheautomator.com/powershell-grep/

    Looks like allmatches for the match
    and not sure about the ones that don't contain.

    wrap it in a foreach loop that iterates through the filesystem

    foreach file system object, do select-string

    if select-string produces a null result, add the file system object's path & name to a custom object

    pseudocode, off the cuff, do not try to eat this raw
    $path = \\path\to\files\
    $files = Get-ChildItem $path
    $hits = @()
    $misses = @()
    
    foreach ($file in $files) {
      $filehits = $file | Select-String -Pattern "fuck", "shit"
      if (($filehits.Length) -eq 0) {
        $misses += $file
        } else {
        $hits += $file
        }
      }
    

    don't @ me if it doesn't work, i'm not gonna debug it for you

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    lwt1973MugsleyRadiation
  • MyiagrosMyiagros Registered User regular
    New lockdown measures means people taking shit home that they usually wouldn't, like pin pads for doing payments... sigh

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
    Feldorn
  • FeldornFeldorn Mediocre Registered User regular
    Myiagros wrote: »
    New lockdown measures means people taking shit home that they usually wouldn't, like pin pads for doing payments... sigh

    Hope their home network is PCI certified...

    Genshin Impact: 600428730
    steam_sig.png
  • EchoEcho Moderator mod
    Darkewolfe wrote: »
    Obligatory: Don't generate a shit ton of logs that literally no one will ever look at for the sake of meeting a compliance check. If they're worth logging they're worth alerting on and reviewing.

    Possible counterpoint, because I don't have enough practical experience yet: make a canonical log line as the end result of processing a request/whatever, let Loki chew on it.

    tl;dr I've been shouting a lot about structured logging at EchoCorp, and settled on logfmt. It's a structured log format where you get key-value pairs that look like this:
    ts=1610844139 level=error msg="pants have been pooped" handler=PoopyPants method=GET path=/foo/bar status=500 duration=2.66773469
    

    Then you can do Loki queries like
    {app="my-service"} | logfmt | status=500 | duration>2s
    

    to get requests that generated a Status 500 and took over two seconds, etc.

    Then write alerts on that.

    Of course, could just as easily do this with Prometheus or some other system to get metrics instead of logs, but let's say you also add a request ID to logs (because of course we have a big distributed microservice architecture), to track the error between half a dozen services. That'll be a cardinality too big for metrics. So append the request ID to the log query to find shit.

    ...and then add some tracing on top of that with the request ID as your trace ID.

    Echo wrote: »
    Let they who have not posted about their balls in the wrong thread cast the first stone.
  • SeidkonaSeidkona Had an upgrade Registered User regular
    If it is not actionable it is useless.

    People ignore noise.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    FeralThawmus
  • DarkewolfeDarkewolfe Registered User regular
    I'll agree with Seidkona, with the clarification of "If you're adding it to something like a searchable SIEM and there are actually an audit team who might someday pull those logs, fine.

    But don't just fill up some local fucking log dump forever till we have to remember to wipe the logs every X period of time and no one ever fucking checked it.

    What is this I don't even.
    SeidkonaFeral
  • SeidkonaSeidkona Had an upgrade Registered User regular
    Darkewolfe wrote: »
    I'll agree with Seidkona, with the clarification of "If you're adding it to something like a searchable SIEM and there are actually an audit team who might someday pull those logs, fine.

    But don't just fill up some local fucking log dump forever till we have to remember to wipe the logs every X period of time and no one ever fucking checked it.

    This is actionable though. It isn't always going to be us who it is actionable to.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • LD50LD50 Registered User regular
    Log everything
    to /dev/null

    SeidkonaDarkewolfeDrovekCarpyMugsleyDizzen
  • EchoEcho Moderator mod
    Seidkona wrote: »
    If it is not actionable it is useless.

    People ignore noise.

    People, yeah. Hence the automatic alerting.

    Echo wrote: »
    Let they who have not posted about their balls in the wrong thread cast the first stone.
  • SeidkonaSeidkona Had an upgrade Registered User regular
    Echo wrote: »
    Seidkona wrote: »
    If it is not actionable it is useless.

    People ignore noise.

    People, yeah. Hence the automatic alerting.

    Amazingly those are noise too that people will ignore.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    FeralNaphtaliThawmus
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Alerts should be actionable.

    Logs can be merely informative.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    IncenjucarInfidelCarpyNaphtaliThawmusEchoiTunesIsEvilSeidkona
  • RadiationRadiation Registered User regular
    Clearly the problem is not logs, but people.

    PSN: jfrofl
    Mugsley
  • ThawmusThawmus Registered User regular
    My boss recently added a script to message my phone whenever a certain backup finished at 9 PM.

    He did not do it as a practical joke. It was not a prank.

    He wasn't even drunk or high when he did this.

    steam_sig.png
  • EchoEcho Moderator mod
    Feral wrote: »
    Alerts should be actionable.

    Logs can be merely informative.

    We don't have the full system in place yet, but when we do my first experiment will be to use logs as a form of metrics. You basically have these tiers of details:
    • Metrics: a pile of numbers giving histograms, averages etc, but nothing very specific.
    • Logs: Can give detailed stuff like request IDs, plus whatever info/data you want - can be either freeform search by squishy human brains, or very indexable for querying with some system.
    • Tracing: Very detailed trace through the code the request passes through, between multiple services.

    So my experiment will be to see how much of the metrics bit I can shoehorn into logging, since newfangled stuff like Loki lets me do some very interesting queries on data from logs that would otherwise require a metrics system like Prometheus.

    Obviously all metrics can't go into logging - logs give me per-request data, metrics give me aggregated performance.

    Echo wrote: »
    Let they who have not posted about their balls in the wrong thread cast the first stone.
    Feral
  • SeidkonaSeidkona Had an upgrade Registered User regular
    Thawmus wrote: »
    My boss recently added a script to message my phone whenever a certain backup finished at 9 PM.

    He did not do it as a practical joke. It was not a prank.

    He wasn't even drunk or high when he did this.

    Welp he had a nice long life...

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • EchoEcho Moderator mod
    Thawmus wrote: »
    My boss recently added a script to message my phone whenever a certain backup finished at 9 PM.

    He did not do it as a practical joke. It was not a prank.

    He wasn't even drunk or high when he did this.

    Automated response at 10:47 pm: "Thanks for letting me know!"

    Echo wrote: »
    Let they who have not posted about their balls in the wrong thread cast the first stone.
    LD50ThawmusIncenjucarDrovekDarkewolfeMugsleyMvrckRadiationFeralDizzen
  • ThawmusThawmus Registered User regular
    He retired last week so guess what the first thing I did this morning was?

    steam_sig.png
    InfidelIncenjucarDrovekCarpyDarkewolfeMugsleyNaphtaliMvrckRadiationFeral
  • MyiagrosMyiagros Registered User regular
    Microsoft Bookings has just made my job way less stressful.

    15 new VPN users to set up due to new lockdown orders. Called the first 5 and only one of them answered the call. Took the time from calling the others and set up a Bookings calendar instead. Stress-free VPN setup and scheduling from now on. Phone tag is the absolute worst.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
    Incenjucar
  • DrovekDrovek Registered User regular
    Thawmus wrote: »
    He retired last week so guess what the first thing I did this morning was?

    Find out his personal phone number?

    steam_sig.png
    EchoRadiationTamerBill
  • ThawmusThawmus Registered User regular
    Naw I didn't like that he did that but truth be told I seriously fucking miss him and was missing him last week before he walked out the door.

    Everybody you work with sucks in some capacity, some worse than others, to be sure. Whether his reasons were correct or not he built a linux-only IT environment and it's been a fucking blast to work in it.

    Now I'm in charge and I'm being quickly reminded of my extremely high capacity to make incredibly bad decisions. That and if I don't have someone, anyone, trying to direct my effort toward something meaningful it gets pretty meaningless and shoddy in a hurry.

    steam_sig.png
    Feral
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Sometimes I feel like I have a good relationship with my CIO, and sometimes I get requests that are just WTF.

    "What controls can we put in place to prevent people from printing company documents at home?"

    What, you mean like when they use VDI? (our primary WFH system) We can just prevent the VDI client from mapping local printers. Easy.

    "No, when they're on a non-company device."

    Me: *stares at the chat window*

    Me: ...

    Are you worried that somebody is going to log in to some cloud-based system that we don't control, using their personal computer, and print out documents from there? I, uh, don't think there's much we can do about that, chief.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    and, uh, if you're really worried about that, maybe rethink the boner you have for the cloud

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
  • IncenjucarIncenjucar Not a Fictional Character Seattle, WARegistered User regular
    At the end of the day people can just bust out a camera and photograph the screen.

  • bowenbowen How you doin'? Registered User regular
    Incenjucar wrote: »
    At the end of the day people can just bust out a camera and photograph the screen.

    This request is very "We may have done some illegal shit and someone internally might report us, how do we stop it?"

    I've seen it a few times and it starts like that and almost always ends with "and that's how I made a lot of money this quarter at the expense of breaking some rules from several regulating bodies and I didn't think they'd notice!"

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
    IncenjucarDrovek
  • IncenjucarIncenjucar Not a Fictional Character Seattle, WARegistered User regular
    bowen wrote: »
    Incenjucar wrote: »
    At the end of the day people can just bust out a camera and photograph the screen.

    This request is very "We may have done some illegal shit and someone internally might report us, how do we stop it?"

    I've seen it a few times and it starts like that and almost always ends with "and that's how I made a lot of money this quarter at the expense of breaking some rules from several regulating bodies and I didn't think they'd notice!"

    I mean there are ways that, in certain scenarios, you can make it a pain in the ass to print off documents, but then the question is why are you letting people you don't trust access those documents anyway.

    bowenGilgaronThawmusSeidkonaDrovek
  • LD50LD50 Registered User regular
    We make it intentionally difficult to print to a personal device, not because we are trying to stop bad actors, but because people are fucking stupid and if we didn't make it difficult there would be thousands of PHI containing printouts scattered over every surface of peoples homes/in estate sales/blowing in the wind/whatever.

    Dizzy DschussFeral
  • MugsleyMugsley Registered User regular
    Maybe I'm missing something but people can just email themselves the docs and print from their personal email(?)

  • LD50LD50 Registered User regular
    Mugsley wrote: »
    Maybe I'm missing something but people can just email themselves the docs and print from their personal email(?)

    If you email doesn't get flagged by our DLP system, sure.

    FeralDrovek
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    LD50 wrote: »
    Mugsley wrote: »
    Maybe I'm missing something but people can just email themselves the docs and print from their personal email(?)

    If you email doesn't get flagged by our DLP system, sure.

    Exactly. We have a robust DLP system that consumes data from multiple sources, including but not limited to software agents that live on all domain-joined PCs (including VDI) and monitor website uploads, print jobs, and emails.

    And we're not really worried about somebody using a cell phone to take a screenshot of a sensitive document. It's tuned more towards situations like somebody dumps our entire CRM to CSV just before quitting to work for a competitor.

    We know we can't stop one-off small data leaks of single pieces of information. But combined with defense in depth (like firewall rules that block unauthorized outgoing FTP & SSH), we have a fighting chance to prevent Equifax-esque massive data exfiltration.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
  • MvrckMvrck Registered User regular
    So I just had a roomie move in, and through the act of getting her computer set up, us both being home all the time watching stuff on different devices, and me needing to do larger file transfers for work, we are nearing our data cap this month for Comcast (because they are just the fucking worst). I could set up a PieHole to track usage (I think), but are there any free Windows alternatives that can track your entire network usage?

    I was looking at Glasswire, but full network tracking is locked behind the subscription wall.

    PwH4Ipj.jpg
  • shadowaneshadowane Registered User regular
    Mvrck wrote: »
    So I just had a roomie move in, and through the act of getting her computer set up, us both being home all the time watching stuff on different devices, and me needing to do larger file transfers for work, we are nearing our data cap this month for Comcast (because they are just the fucking worst). I could set up a PieHole to track usage (I think), but are there any free Windows alternatives that can track your entire network usage?

    I was looking at Glasswire, but full network tracking is locked behind the subscription wall.

    Can't you track your usage in your comcast account? I'm pretty sure mine tells me how much I've used for the month. Though you can't alert on that or anything so maybe you are looking for something more useful than just knowing the number.

    Rich on Beer - I talk about drinking beer. You read about it.
  • MvrckMvrck Registered User regular
    shadowane wrote: »
    Mvrck wrote: »
    So I just had a roomie move in, and through the act of getting her computer set up, us both being home all the time watching stuff on different devices, and me needing to do larger file transfers for work, we are nearing our data cap this month for Comcast (because they are just the fucking worst). I could set up a PieHole to track usage (I think), but are there any free Windows alternatives that can track your entire network usage?

    I was looking at Glasswire, but full network tracking is locked behind the subscription wall.

    Can't you track your usage in your comcast account? I'm pretty sure mine tells me how much I've used for the month. Though you can't alert on that or anything so maybe you are looking for something more useful than just knowing the number.

    Yeah, I'm trying to more track our average streaming usage to figure out how much of this push has been work vs recreation and where in recreation our major usage is coming. I can always swing by the office for large file transfers (talking 80 gigs of video) from now on, but also to see where the breakdown is really coming from TV vs computer usage (we have one of the 4k Chromecasts, but only a 1080p TV, so if it's doing dumb shit like pulling down 4k video for it, that would be good to know too). Previously I was averaging about 580gb a month, so almost doubling that from one extra person seems surprising, but not impossible.

    PwH4Ipj.jpg
  • SiliconStewSiliconStew Registered User regular
    Anyone have recommendations for software you've used to scan contents of files in a standard windows SMB fileshare and alert if identifying information such as social security numbers, credit card numbers, etc are found? Sort of as a compliment to what you can get in O365. One of our business partners had a security incident that has spooked our upper management into giving us the budget to actually implement many of the security items we've been asking them for.

    Just remember that half the people you meet are below average intelligence.
  • Dizzy DDizzy D NetherlandsRegistered User regular
    Anyone have recommendations for software you've used to scan contents of files in a standard windows SMB fileshare and alert if identifying information such as social security numbers, credit card numbers, etc are found? Sort of as a compliment to what you can get in O365. One of our business partners had a security incident that has spooked our upper management into giving us the budget to actually implement many of the security items we've been asking them for.

    Is this kinda what you are looking for https://docs.microsoft.com/en-us/compliance/regulatory/gdpr-for-office-servers ? This one is based on Azure (so you need to have a subscription, but it can scan network shares and on-prem servers). I know Symantec has made something similar, but I haven't worked with their version.

    Steam/Origin: davydizzy
    Feral
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Anyone have recommendations for software you've used to scan contents of files in a standard windows SMB fileshare and alert if identifying information such as social security numbers, credit card numbers, etc are found? Sort of as a compliment to what you can get in O365. One of our business partners had a security incident that has spooked our upper management into giving us the budget to actually implement many of the security items we've been asking them for.

    It's a feature of the DLP package we use (Forcepoint) but if that's all you want to do, Forcepoint is like buying a bulldozer to swat a fly.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    Feldorn
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Application engineer, to me: "Hey, do you have a minute to get on a call? I'm trying to figure out a problem with a vendor."

    Me: Sure. Give me a rundown of the problem?

    App: "REST calls from Contoso [a vendor I've never heard of] aren't working. We're wondering if it's the firewall."

    Me: Who is Contoso, where are the REST calls coming from, what is the destination of the rest call, and what happens when you try?

    App: "The REST URL is https://rest.contoso.com/api [Oh, so it's not REST calls from Contoso, it's REST calls to Contoso.] They're coming from our Widgets server."

    Me: Okay, so what happens?

    App: "It just closes the connection."

    Me: Well, it's highly unlikely to be the firewall [because reasons], but I'll check. In the meantime, are you RDPed into the server?

    App: "yeah."

    Me: Try browsing to the REST URL from any browser installed on it.

    App: "okay."

    Me: FYI you can conference me in any time.

    App: "Okay"

    App: ...

    App: "We're looking at the code now."

    Me: What code?

    App: "The REST code."

    Me: Okay, so what happened when you tried it from a browser?

    App: ...

    App: ...

    Me: ...

    App: "We needed to add an auth key to the HTTPS call."

    Me: That's fine, but it wouldn't explain your SSL connection just getting closed. What happened when you tried it from a browser?

    App: It didn't work right.

    Me: Define 'didn't work right.'

    App: *posts screenshot*

    App: *screenshot shows a browser window, contents are a JSON string, and right in the middle is the error INVALID AUTH KEY*

    Me: So you got an auth error when you tried it in a browser, and your script didn't contain an auth key. Is that right?

    App: "Yeah."

    giphy.gif

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
    the "no true scotch man" fallacy.
    IncenjucarschussDrovekThawmusDarkewolfeFeldornInfidelBlackDragon480boweniTunesIsEvilPeewilwt1973
Sign In or Register to comment.