As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

[Sysadmin] Routing to null

1767779818299

Posts

  • Options
    SiliconStewSiliconStew Registered User regular
    edited November 2020
    SniperGuy wrote: »
    Myiagros wrote: »
    Make sure you have backups, and then make sure that they can restore properly. Especially when learning the ropes, there is nothing worse than the feeling of dread when you make a mistake and then have to figure out when your last usable backup is, and then figuring out how to restore it. Veeam is a good option as it is very flexible, it is also free to use the Community Edition and allows for image, VM, file, etc backups.

    We actually have Veeam! We've only got the one VMWare AD server, but we have Veeam making regular automatic backups of the whole thing. Hoping to get a second server for redundancy soon.

    Backups are always good but domain controllers have special considerations. Just keep in mind that once you have more than one domain controller, you should really never do a restore of an individual domain controller. Especially when they are virtual and so easily replaced, it's not worth the problems it will cause. Domain Controllers keep track of sync updates between themselves and if you restore one, it will know it can't get back in sync and best case it shuts off its ability to do anything with AD (users can't log in against it, etc), worst case it overwrites recent changes with it's outdated info and things start breaking across your environment. If something breaks on one DC, just leave it broke and shut it down, stand up a new vm and add it as a new domain controller to the domain, then just delete the broken VM. You'd then do a manual cleanup of AD to remove the references to the broken DC, you can find the proper steps to do so on the internet.

    If you have a really severe issue with AD that really does need restoration (and I've never run across such a situation that would truly warrant this), kill all your domain controllers but one and only restore that one, preferably your FSMO Master Role holder. Then set up new domain controllers to replace all the other ones. That way you have a single copy of the "truth" of your AD post restoration that gets replicated to all the new DC's without issue.

    SiliconStew on
    Just remember that half the people you meet are below average intelligence.
  • Options
    Dizzy DDizzy D NetherlandsRegistered User regular
    Enable Active Directory Recycle Bin if you haven't done so already and create a test-account and delete and restore it, so you have done it before you ever need to do it during production time.

    Steam/Origin: davydizzy
  • Options
    lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Quick question, what's the best and easy PDF software that you use? We need something to just add a couple of things to PDF's per day so it doesn't have to have all the bells and whistles.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • Options
    DarkewolfeDarkewolfe Registered User regular
    Just pay for the Adobe product.

    What is this I don't even.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited November 2020
    lwt1973 wrote: »
    Quick question, what's the best and easy PDF software that you use? We need something to just add a couple of things to PDF's per day so it doesn't have to have all the bells and whistles.

    They all suck for different flavors of suckitude. Foxit and Nitro don't suck too horribly much, usually.

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    I'm in a slow argument with one of our web hosting companies where one of their techs included this line, trying to justify why they don't have load balancers in their infrastructure:
    Our engineering team used to run load balancers, but they turned out to be clunky and failure-prone.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    SiliconStewSiliconStew Registered User regular
    Feral wrote: »
    I'm in a slow argument with one of our web hosting companies where one of their techs included this line, trying to justify why they don't have load balancers in their infrastructure:
    Our engineering team used to run load balancers, but they turned out to be clunky and failure-prone.

    Sounds like you need to be in a fast argument about switching web hosting companies.

    Just remember that half the people you meet are below average intelligence.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited November 2020
    Feral wrote: »
    I'm in a slow argument with one of our web hosting companies where one of their techs included this line, trying to justify why they don't have load balancers in their infrastructure:
    Our engineering team used to run load balancers, but they turned out to be clunky and failure-prone.

    Sounds like you need to be in a fast argument about switching web hosting companies.

    This is the new hosting company that we just switched (one particular service) over to.

    Their argument is "we put everything on AWS so we don't need load balancers"

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    schussschuss Registered User regular
    Feral wrote: »
    Feral wrote: »
    I'm in a slow argument with one of our web hosting companies where one of their techs included this line, trying to justify why they don't have load balancers in their infrastructure:
    Our engineering team used to run load balancers, but they turned out to be clunky and failure-prone.

    Sounds like you need to be in a fast argument about switching web hosting companies.

    This is the new hosting company that we just switched (one particular service) over to.

    Their argument is "we put everything on AWS so we don't need load balancers"

    Let me guess, their stuff is stood up on EC2 instances in AWS just like they were on-prem...

  • Options
    That_GuyThat_Guy I don't wanna be that guy Registered User regular
    lwt1973 wrote: »
    Quick question, what's the best and easy PDF software that you use? We need something to just add a couple of things to PDF's per day so it doesn't have to have all the bells and whistles.

    Several of my clients use Bluebeam but it's still not as good as Adobe Acrobat.

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    schuss wrote: »
    Feral wrote: »
    Feral wrote: »
    I'm in a slow argument with one of our web hosting companies where one of their techs included this line, trying to justify why they don't have load balancers in their infrastructure:
    Our engineering team used to run load balancers, but they turned out to be clunky and failure-prone.

    Sounds like you need to be in a fast argument about switching web hosting companies.

    This is the new hosting company that we just switched (one particular service) over to.

    Their argument is "we put everything on AWS so we don't need load balancers"

    Let me guess, their stuff is stood up on EC2 instances in AWS just like they were on-prem...

    That's how it looks.

    The initial problem that led us down this rabbit hole is that we have certain other vendors who need us to give them the static IPs assigned to this particular service, and this hosting company refuses to do that because "compute nodes have dynamic IPs"

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    We'll probably end up throwing a simple proxy up on one of our other hosting providers to deal with this.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    SiliconStewSiliconStew Registered User regular
    Feral wrote: »
    Feral wrote: »
    I'm in a slow argument with one of our web hosting companies where one of their techs included this line, trying to justify why they don't have load balancers in their infrastructure:
    Our engineering team used to run load balancers, but they turned out to be clunky and failure-prone.

    Sounds like you need to be in a fast argument about switching web hosting companies.

    This is the new hosting company that we just switched (one particular service) over to.

    Their argument is "we put everything on AWS so we don't need load balancers"

    Ah, of course! Those jokers at Amazon put Elastic Load Balancers, Application Load Balancers, and Network Load Balancers in the AWS menus just to trick people. But they won't fall for that ruse!

    Just remember that half the people you meet are below average intelligence.
  • Options
    LD50LD50 Registered User regular
    That_Guy wrote: »
    lwt1973 wrote: »
    Quick question, what's the best and easy PDF software that you use? We need something to just add a couple of things to PDF's per day so it doesn't have to have all the bells and whistles.

    Several of my clients use Bluebeam but it's still not as good as Adobe Acrobat.

    Nitro is better than acrobat pro.

  • Options
    MyiagrosMyiagros Registered User regular
    LD50 wrote: »
    That_Guy wrote: »
    lwt1973 wrote: »
    Quick question, what's the best and easy PDF software that you use? We need something to just add a couple of things to PDF's per day so it doesn't have to have all the bells and whistles.

    Several of my clients use Bluebeam but it's still not as good as Adobe Acrobat.

    Nitro is better than acrobat pro.

    We've been pushing Kofax PowerPDF, I find it runs better than the other alternatives.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Hey billion dollar company, you might want to switch from Internet Explorer for your VPN access at some point.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • Options
    DrovekDrovek Registered User regular
    Currently on "ass slightly warm but not quite on fire" over this one

    Fun times

    steam_sig.png( < . . .
  • Options
    SiliconStewSiliconStew Registered User regular
    Drovek wrote: »
    Currently on "ass slightly warm but not quite on fire" over this one

    Fun times

    Yeah, we've got a LoB application that uses Kinesis that we can't get data to or from right now. At least it's a slow time of the year for that business unit so for us the impact is about as low as it could be for that level of outage.

    Just remember that half the people you meet are below average intelligence.
  • Options
    V1mV1m Registered User regular
    lwt1973 wrote: »
    Hey billion dollar company, you might want to switch from Internet Explorer for your VPN access at some point.

    "It won't increase my bonus, and it literally costs nothing to make you work 20 hour days for a month to fix any problems."

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Other than using ALTools/EventCombMT, does anybody have any sick strats for tracking down the source of an account lockout?

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FeldornFeldorn Mediocre Registered User regular
    edited December 2020
    In AD? If event 4740 in the PDC doesn’t list the caller computer then I think I once enabled netlogon logging and found out that way to get the IP.

    If you know where it’s coming from but not why... I do not.

    Feldorn on
  • Options
    Dizzy DDizzy D NetherlandsRegistered User regular
    If you have AD FS (and a recent version of it), you can configure extranet smart lockout which helps a bit to narrow down where the lockouts are coming from, but otherwise it's mostly Event Comb.

    Steam/Origin: davydizzy
  • Options
    FeldornFeldorn Mediocre Registered User regular
    Dizzy D wrote: »
    If you have AD FS (and a recent version of it), you can configure extranet smart lockout which helps a bit to narrow down where the lockouts are coming from, but otherwise it's mostly Event Comb.

    ADFS 4 (2016) or higher.

    You'd still see the correct caller computer name if it was from ADFS though.

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Big Solarwinds breach.

    An advanced persistent threat (state-level actor backed by Russia, most likely) infiltrated Solarwinds's digital supply chain and injected a trojan into Solarwinds patches. If you've downloaded or updated anything from Solarwinds for the Orion framework (such as, but not limited to, Network Performance Monitor) in 2020 you might be affected.

    https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
    SolarWinds has just been made aware our systems experienced a highly sophisticated, manual supply chain attack on SolarWinds® Orion® Platform software builds for versions 2019.4 HF 5 and 2020.2 with no hotfix or 2020.2 HF 1. We have been advised this attack was likely conducted by an outside nation state and intended to be a narrow, extremely targeted, and manually executed attack, as opposed to a broad, system-wide attack. We recommend taking the following steps related to your use of the SolarWinds Orion Platform.

    https://www.solarwinds.com/securityadvisory

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    DarkewolfeDarkewolfe Registered User regular
    https://cyber.dhs.gov/ed/21-01/

    Assume advanced persistent threat on anything managed by solarwinds and assume compromise of all accounts due to kerberoasting.

    Basically, if you used Solarwinds you need to reimage and rebuild everything you own.

    What is this I don't even.
  • Options
    SiliconStewSiliconStew Registered User regular
    Always enjoy butt clenching news like this. Looks like we dodged the bullet. We use a couple solarwinds things, but not Orion.

    Just remember that half the people you meet are below average intelligence.
  • Options
    jungleroomxjungleroomx It's never too many graves, it's always not enough shovels Registered User regular
    Hahah wow holy shit

  • Options
    SeidkonaSeidkona Had an upgrade Registered User regular
    I couldn't think of a good sarcastic title so "just the facts ma'am" this time.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • Options
    RadiationRadiation Registered User regular
    Whole lot of people are having a bad day today.
    *pours one out for other security folk*

    PSN: jfrofl
  • Options
    FeldornFeldorn Mediocre Registered User regular
    I think our Orion install is old enough to not have been affected...

    I'm sure it has other problems though

  • Options
    AthenorAthenor Battle Hardened Optimist The Skies of HiigaraRegistered User regular
    SANS.org is doing an emergency webcast on this situation. If you are interested:

    https://youtu.be/4tmlZCk2gCg

    He/Him | "A boat is always safest in the harbor, but that’s not why we build boats." | "If you run, you gain one. If you move forward, you gain two." - Suletta Mercury, G-Witch
  • Options
    jungleroomxjungleroomx It's never too many graves, it's always not enough shovels Registered User regular
    My old company had a lot of Solarwinds stuff at the datacenter.

    Oh man I hope shit is hitting the fan.

  • Options
    DarkewolfeDarkewolfe Registered User regular
    Having your former boss e-mail this morning to ask if you'll come back. A+. Also lol no.

    What is this I don't even.
  • Options
    DarkewolfeDarkewolfe Registered User regular
    edited December 2020
    Also, let me point out that Solarwinds stock is only down 17% after a day trading on this news.

    What in the actual fuck. Maybe they're assuming the money the company gets from overtime from Solarwinds professional services to mitigate their own vulnerability will make it better?

    Is this a good strategy? Unleash a devastating vuln on your customers and then make bank having them pay you to fix it?

    Darkewolfe on
    What is this I don't even.
  • Options
    RadiationRadiation Registered User regular
    Down 17% because it's bad, but those contracts aren't going to dry up for a while at least.

    PSN: jfrofl
  • Options
    AthenorAthenor Battle Hardened Optimist The Skies of HiigaraRegistered User regular
    edited December 2020
    Darkewolfe wrote: »
    Also, let me point out that Solarwinds stock is only down 17% after a day trading on this news.

    What in the actual fuck. Maybe they're assuming the money the company gets from overtime from Solarwinds professional services to mitigate their own vulnerability will make it better?

    Is this a good strategy? Unleash a devastating vuln on your customers and then make bank having them pay you to fix it?

    There's no sign Solarwinds is doing this. They've already filed SEC filings about how bad this is, and there's no signs they are gonna try to make money off this.

    Edit: I should note I'm a Solarwinds Fanboy and I don't want them getting too hurt.. but hopefully this makes them a better company because more and more legit complaints are coming out about how they are handling this.

    Athenor on
    He/Him | "A boat is always safest in the harbor, but that’s not why we build boats." | "If you run, you gain one. If you move forward, you gain two." - Suletta Mercury, G-Witch
  • Options
    That_GuyThat_Guy I don't wanna be that guy Registered User regular
    edited December 2020
    Y'all got any suggestions for a SATA RAID card that supports 16tb drives? The built in VROC raid controller in this T40 server I have won't work with 16tb drives. I just need a RAID1 on these 2 drives for a backup appliance. I was doing some research and the card needs LBA48 to support the drives I intend to use.

    After doing a bunch of research, I think I'm going to give this one a try.
    https://www.amazon.com/High-Point-RocketRAID-640L-PCI-Express/dp/B0034CQR4O

    That_Guy on
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Athenor wrote: »
    Darkewolfe wrote: »
    Also, let me point out that Solarwinds stock is only down 17% after a day trading on this news.

    What in the actual fuck. Maybe they're assuming the money the company gets from overtime from Solarwinds professional services to mitigate their own vulnerability will make it better?

    Is this a good strategy? Unleash a devastating vuln on your customers and then make bank having them pay you to fix it?

    There's no sign Solarwinds is doing this. They've already filed SEC filings about how bad this is, and there's no signs they are gonna try to make money off this.

    Edit: I should note I'm a Solarwinds Fanboy and I don't want them getting too hurt.. but hopefully this makes them a better company because more and more legit complaints are coming out about how they are handling this.

    @Athenor

    good lord, why?

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    AthenorAthenor Battle Hardened Optimist The Skies of HiigaraRegistered User regular
    Feral wrote: »
    Athenor wrote: »
    Darkewolfe wrote: »
    Also, let me point out that Solarwinds stock is only down 17% after a day trading on this news.

    What in the actual fuck. Maybe they're assuming the money the company gets from overtime from Solarwinds professional services to mitigate their own vulnerability will make it better?

    Is this a good strategy? Unleash a devastating vuln on your customers and then make bank having them pay you to fix it?

    There's no sign Solarwinds is doing this. They've already filed SEC filings about how bad this is, and there's no signs they are gonna try to make money off this.

    Edit: I should note I'm a Solarwinds Fanboy and I don't want them getting too hurt.. but hopefully this makes them a better company because more and more legit complaints are coming out about how they are handling this.

    Athenor

    good lord, why?

    Stockholm syndrome? We used to run SCOM, but it was horrible and bloated. And while Solarwinds is also bloated, at least we got to make it from the ground up and tune it to what we needed. It's been great actually having visibility into our systems.

    He/Him | "A boat is always safest in the harbor, but that’s not why we build boats." | "If you run, you gain one. If you move forward, you gain two." - Suletta Mercury, G-Witch
  • Options
    DarkewolfeDarkewolfe Registered User regular
    Does anyone remember who Moe is? I don't.

    What is this I don't even.
This discussion has been closed.