As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
We're funding a new Acquisitions Incorporated series on Kickstarter right now! Check it out at https://www.kickstarter.com/projects/pennyarcade/acquisitions-incorporated-the-series-2

[Sysadmin] Nightmare fuel

1181921232499

Posts

  • CogCog What'd you expect? Registered User regular
    Infidel wrote: »
    bowen wrote: »
    I've had to block a lot of domains recently, somehow people are getting ahold of my email and asking me to buy shit.

    It's all local too, like none of it is random nigerian princes. It's all companies that exist in syracuse and they want me to buy their garbage they're selling. And I can find the sales person on linkedin working for them too.

    Sounds like your email got onto a business registry somewhere!

    I mean, he did make a major purchase recently.

  • AiouaAioua Ora Occidens Ora OptimaRegistered User regular
    speaking of security team annoyances

    got a spam/phishing looking mail

    check out the links, they're going to suspicious URLs

    forward to the [email protected] email like you're supposed to

    get a mail back in a minute from that address all like "good job, this was a phishing test mail, congrats on passing"


    um

    afaik we haven't ever done phishing tests (though I've certainly thought we should!)

    and... if this was a test mail it's a real bad one cause I checked out the link urls and none of them are going to, like a company that runs those or anything internal, so they're not gathering data

    so either they have a real bad a useless phishing test
    or
    they had a real test at one point and stupidly set up an autoreply and never turned it off
    or
    they're fraudulently bad at their jobs and the reporting box is entirely for show

    life's a game that you're bound to lose / like using a hammer to pound in screws
    fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
    that's right we're on a fucked up cruise / God is dead but at least we have booze
    bad things happen, no one knows why / the sun burns out and everyone dies
  • jungleroomxjungleroomx It's never too many graves, it's always not enough shovels Registered User regular
    So uh.

    Had a client (a legacy one, a Catholic school) get an odd version of ransomware on their mail server.

    I say odd because it didn't propagate across the entire network.

    So, we blow out the entire thing by removing the infected drives, put in some clean ones, and right in the middle of me installing Exchange server, I get booted.

    I log back in immediately and apparently interrupted someone copy/pasting a thing called "vpn.exe"

    None of the other tech companies they deal with were logging in that day, neither was anyone at the school.

    So, yeah, passed that shit on up the ladder to a security company. That's outside of my wheelhouse.

    Windows servers by default allow 2 RDP sessions, so being kicked out and/or connecting back to an existing session means someone else was using your exact same credentials. AKA, your account has been compromised.

    Yeah, I know.

    It's the domain admin credentials for the school.

    You know, the ones they write on a post-it in the server room and stick on the side of one of the machines.

    wunderbarFeral
  • wunderbarwunderbar What Have I Done? Registered User regular
    So uh.

    Had a client (a legacy one, a Catholic school) get an odd version of ransomware on their mail server.

    I say odd because it didn't propagate across the entire network.

    So, we blow out the entire thing by removing the infected drives, put in some clean ones, and right in the middle of me installing Exchange server, I get booted.

    I log back in immediately and apparently interrupted someone copy/pasting a thing called "vpn.exe"

    None of the other tech companies they deal with were logging in that day, neither was anyone at the school.

    So, yeah, passed that shit on up the ladder to a security company. That's outside of my wheelhouse.

    Windows servers by default allow 2 RDP sessions, so being kicked out and/or connecting back to an existing session means someone else was using your exact same credentials. AKA, your account has been compromised.

    Yeah, I know.

    It's the domain admin credentials for the school.

    You know, the ones they write on a post-it in the server room and stick on the side of one of the machines.

    you mean that's not best practice?

    c5e.jpg

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
    Apothe0sis
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Aioua wrote: »
    speaking of security team annoyances

    got a spam/phishing looking mail

    check out the links, they're going to suspicious URLs

    forward to the [email protected] email like you're supposed to

    get a mail back in a minute from that address all like "good job, this was a phishing test mail, congrats on passing"


    um

    afaik we haven't ever done phishing tests (though I've certainly thought we should!)

    and... if this was a test mail it's a real bad one cause I checked out the link urls and none of them are going to, like a company that runs those or anything internal, so they're not gathering data

    so either they have a real bad a useless phishing test
    or
    they had a real test at one point and stupidly set up an autoreply and never turned it off
    or
    they're fraudulently bad at their jobs and the reporting box is entirely for show

    I might misunderstand your point.

    Why is that a bad test?

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • AiouaAioua Ora Occidens Ora OptimaRegistered User regular
    Feral wrote: »
    Aioua wrote: »
    speaking of security team annoyances

    got a spam/phishing looking mail

    check out the links, they're going to suspicious URLs

    forward to the [email protected] email like you're supposed to

    get a mail back in a minute from that address all like "good job, this was a phishing test mail, congrats on passing"


    um

    afaik we haven't ever done phishing tests (though I've certainly thought we should!)

    and... if this was a test mail it's a real bad one cause I checked out the link urls and none of them are going to, like a company that runs those or anything internal, so they're not gathering data

    so either they have a real bad a useless phishing test
    or
    they had a real test at one point and stupidly set up an autoreply and never turned it off
    or
    they're fraudulently bad at their jobs and the reporting box is entirely for show

    I might misunderstand your point.

    Why is that a bad test?

    the link is going to an outside domain we don't control, and not one that belongs to, like a company that runs phishing tests
    so like not only are they not catching people who click the link, they have no control over what's actually on the other end of the link

    currently the link is dead, there's no DNS entry for the domain
    like if it's the case that it's a real 'test' then my suspicion is they took a REAL FUCKING PHISHING MAIL and copied it without editing anything
    and the link just happens to be dead because it's old and got shut down

    life's a game that you're bound to lose / like using a hammer to pound in screws
    fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
    that's right we're on a fucked up cruise / God is dead but at least we have booze
    bad things happen, no one knows why / the sun burns out and everyone dies
    Feldorn
  • ArcSynArcSyn Registered User regular
    wunderbar wrote: »
    So uh.

    Had a client (a legacy one, a Catholic school) get an odd version of ransomware on their mail server.

    I say odd because it didn't propagate across the entire network.

    So, we blow out the entire thing by removing the infected drives, put in some clean ones, and right in the middle of me installing Exchange server, I get booted.

    I log back in immediately and apparently interrupted someone copy/pasting a thing called "vpn.exe"

    None of the other tech companies they deal with were logging in that day, neither was anyone at the school.

    So, yeah, passed that shit on up the ladder to a security company. That's outside of my wheelhouse.

    Windows servers by default allow 2 RDP sessions, so being kicked out and/or connecting back to an existing session means someone else was using your exact same credentials. AKA, your account has been compromised.

    Yeah, I know.

    It's the domain admin credentials for the school.

    You know, the ones they write on a post-it in the server room and stick on the side of one of the machines.

    you mean that's not best practice?

    c5e.jpg

    Yeah, obviously you are supposed to give your normal login domain admin privileges because it just makes everything easier.

    4dm3dwuxq302.png
  • DarkewolfeDarkewolfe Registered User regular
    For the first time in my life I'm in a group of sysadmins who are all decidedly older and more insular. Up until now I'd always been in tech groups that trended younger.

    And so the other thing I'm witnessing for the first time is the true, batshit crazy, conspiracy theorist, awful human beings that make up most of our industry. Holy shit I hate these guys.

    What is this I don't even.
    SeidkonaAiouaKakodaimonos
  • Gilbert0Gilbert0 North of SeattleRegistered User regular
    Anyone on here using Azure for anything? Any advice on what to start to read up on / research first?

  • SiliconStewSiliconStew Registered User regular
    Gilbert0 wrote: »
    Anyone on here using Azure for anything? Any advice on what to start to read up on / research first?

    You'd need to be more specific. Azure encompasses a huge variety of different products and services.

    Just remember that half the people you meet are below average intelligence.
    DrovekFeral
  • Gilbert0Gilbert0 North of SeattleRegistered User regular
    Oh very aware, and that's kind of the problem. We're starting from scratch and will be setting it all up. So basically everything. Setting up EA portal, the Azure Portal, figuring out billing better, reports, role based access, setting up divisions. Starting to roll out some devices with InTune / AutoPilot. Short term / first services are Mail/Exchange/SharePoint. Eventually full Office365, Teams, storage, VMs.

  • CogCog What'd you expect? Registered User regular
    So, I'm kicking off an Exchange upgrade and migration today.

    Anyone else having "fun"?

    Apothe0sis
  • KakodaimonosKakodaimonos Code fondler Helping the 1% get richerRegistered User regular
    Darkewolfe wrote: »
    For the first time in my life I'm in a group of sysadmins who are all decidedly older and more insular. Up until now I'd always been in tech groups that trended younger.

    And so the other thing I'm witnessing for the first time is the true, batshit crazy, conspiracy theorist, awful human beings that make up most of our industry. Holy shit I hate these guys.

    So I work for a trading firm. It's been pretty funny watching all the IT & devs go absolutely bonkers about the Bitcoin futures and the professional traders are all avoiding that shit like the plague.

    VladimerApothe0sis
  • ArcSynArcSyn Registered User regular
    Cog wrote: »
    So, I'm kicking off an Exchange upgrade and migration today.

    Anyone else having "fun"?

    I'm trying to figure out why my VOIP recorder is just getting static. Wireshark captures it clearly at the server.

    4dm3dwuxq302.png
  • twmjrtwmjr Registered User regular
    codec mismatch?

  • ArcSynArcSyn Registered User regular
    edited December 2017
    twmjr wrote: »
    codec mismatch?

    That's my guess.
    Used to work, but now it doesn't.

    ArcSyn on
    4dm3dwuxq302.png
  • CogCog What'd you expect? Registered User regular
    That feeling when your server boots up for the first time and you realize you accidentally installed it as a Core server.

    FeralFeldorn
  • CogCog What'd you expect? Registered User regular
    ArcSyn wrote: »
    twmjr wrote: »
    codec mismatch?

    That's my guess.
    Used to work, but now it doesn't.

    It's always DNS.

    Aiouabowen
  • CogCog What'd you expect? Registered User regular
    edited December 2017
    Sophos sent us a thing.

    I thought you guys would appreciate it.

    Spoiled because it's fucking massive.
    TzGuSht.jpg

    EDIT: I actually even turned it into a clickthrough image because it's way bigger than the boards will display.

    Cog on
    ArcSyniTunesIsEvilbowenAiouaInfidelwunderbarSeidkonaShadowfireMugsley
  • bowenbowen How you doin'? Registered User regular
    Nice.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • ReznikReznik Registered User regular
    Any of you folks know if popping a laptop open for a RAM upgrade will trigger bitlocker? Google is giving vague and/or contradictory answers.

    I'm gonna suspend it anyway but it would be nice to know. The people who run this thing have an axe to grind against the group I support so I need to take every precaution to avoid dealing with them.

    Do... Re.... Mi... Ti... La...
    Do... Re... Mi... So... Fa.... Do... Re.... Do...
    Forget it...
    Aioua
  • bowenbowen How you doin'? Registered User regular
    no, it shouldn't

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • AiouaAioua Ora Occidens Ora OptimaRegistered User regular
    bowen wrote: »
    no, it shouldn't

    it shouldn't but if you don't have the bitlocker key?

    like I mean, you are looking at it right now and are extremely certain it is correct and up-to-date?

    might as well suspend before making hardware or firmware changes

    and you know what, suspend it anyway while you're at it

    life's a game that you're bound to lose / like using a hammer to pound in screws
    fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
    that's right we're on a fucked up cruise / God is dead but at least we have booze
    bad things happen, no one knows why / the sun burns out and everyone dies
    Darkewolfe
  • AiouaAioua Ora Occidens Ora OptimaRegistered User regular
    we have laptops that will throw bitlocker if you you plug them into the wrong kind of docking station

    it's a fickle mistress

    life's a game that you're bound to lose / like using a hammer to pound in screws
    fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
    that's right we're on a fucked up cruise / God is dead but at least we have booze
    bad things happen, no one knows why / the sun burns out and everyone dies
    Darkewolfe
  • SeidkonaSeidkona Had an upgrade Registered User regular
    Shouldn't and won't are two different things.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
    FeralDarkewolfe
  • CogCog What'd you expect? Registered User regular
    A place I used to work at, I had a super shitstain for a manager. He was completely incompetent, both as an IT tech, and as a manager. Got the position only because he was good buddies with a guy up the ladder, who was also borderline criminally incompetent.

    So that guy subsequently, years later, worked here at my current job up until about a year before I did. He was assigned the as the primary tech at the client where I'm currently the primary tech.

    Last weekend one of their servers started spitting out alerts via the iDRAC. I opened a case with Dell and while they were looking at the case history of that server, they saw that they'd dispatched that same part for that same server a year and a half ago.

    We did some digging on various part numbers and things, and what we eventually figured out was, my dipshit ex-boss actually opened a case on the same issue the last time the part was acting up. But when he received the part - which happened to be the god damned server motherboard - he replaced the motherboard in the wrong fucking server.

    Vindication tastes so sweet.

    bowenSeidkonawunderbarDrovekShadowfirea5ehrenLaOsInfidelMugsleyRandomHajileMvrckjungleroomx
  • bowenbowen How you doin'? Registered User regular
    amazing

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
    Drovek
  • CogCog What'd you expect? Registered User regular
    bowen wrote: »
    amazing

    Ladies.

  • MugsleyMugsley DelawareRegistered User regular
    Cog wrote: »
    God dammit.

    Look.

    I'm a big fan of security, but there comes a point where you've locked yourself down so much that basic program functions can't basically function programs anymore because you're up every single network packet's asshole with two hands and a flashlight.

    Welcome to every day of my life on Navy systems.

  • DarkewolfeDarkewolfe Registered User regular
    Cog wrote: »
    God dammit.

    Look.

    I'm a big fan of security, but there comes a point where you've locked yourself down so much that basic program functions can't basically function programs anymore because you're up every single network packet's asshole with two hands and a flashlight.

    There's an environment where this is true, but just as often it's that you asshats have done such a poor job on everything leading up to this that stringent network security is the only way to have any modicum of security at all.

    No, you don't need a fucking priveleged account, persons A-Q.

    What is this I don't even.
    Feral
  • taliosfalcontaliosfalcon Registered User regular
    edited December 2017
    We have a client, who is a City, and keeps DDOSing themselves by sending out mass emails to 18K+ people referencing 200+ MB PDFs and giant training videos on their site. Que 6k concurrent users trying to d/l it at once when their average concurrent users are ~300 and ololol there goes the web servers. And then they dont learn and a month later do it again. Rinse and repeat

    taliosfalcon on
    steam xbox - adeptpenguin
    wunderbarFeldornFeralShadowfireDonovan Puppyfucker
  • bowenbowen How you doin'? Registered User regular
    This is why email is not file storage.

    The better thing to do is make a blog post with the PDF's content and videos on your website. This will drive sales to your website too as customers will browse through other shit you're offering and maybe make more purchases.

    This also let's them bookmark and find it easier rather than it being buried in their email recycle bin before it gets eaten by the retention policy.

    You should bring those points up to them that they could potentially make more money like that. That is what drives policy changes.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
    ArcSynLD50FeralCog
  • ArcSynArcSyn Registered User regular
    Re: Firewalls

    This makes me nuts with app developers. "Turn off the software firewall for our app."
    What?! Give me a port. Or a few ports. Why is this so hard?

    4dm3dwuxq302.png
    FeralDarkewolfe
  • LD50LD50 Registered User regular
    ArcSyn wrote: »
    "Turn off the software firewall for our app."

    No.

    SeidkonaArcSynbowenFeralDarkewolfeShadowfireTraceDonovan PuppyfuckerMvrck
  • SiliconStewSiliconStew Registered User regular
    bowen wrote: »
    This is why email is not file storage.

    The better thing to do is make a blog post with the PDF's content and videos on your website. This will drive sales to your website too as customers will browse through other shit you're offering and maybe make more purchases.

    This also let's them bookmark and find it easier rather than it being buried in their email recycle bin before it gets eaten by the retention policy.

    You should bring those points up to them that they could potentially make more money like that. That is what drives policy changes.

    By the description, they already do that. But their webservers cannot handle the sudden traffic spikes and choke.

    Just remember that half the people you meet are below average intelligence.
  • bowenbowen How you doin'? Registered User regular
    bowen wrote: »
    This is why email is not file storage.

    The better thing to do is make a blog post with the PDF's content and videos on your website. This will drive sales to your website too as customers will browse through other shit you're offering and maybe make more purchases.

    This also let's them bookmark and find it easier rather than it being buried in their email recycle bin before it gets eaten by the retention policy.

    You should bring those points up to them that they could potentially make more money like that. That is what drives policy changes.

    By the description, they already do that. But their webservers cannot handle the sudden traffic spikes and choke.

    Yeah don't actually offer a PDF download directly. They're overhead heavy compared to HTML data that can be gziped and all that.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    This is what CDNs are fooorrrrrrr

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
    bowenDarkewolfe
  • Apothe0sisApothe0sis Have you ever questioned the nature of your reality? Registered User regular
    Well, one of our DBAs just managed to run this on a production database.
    DELETE FROM transactions WHERE transactionID IN (SELECT transactionID FROM transactions)
    
    

    ALWAYS WRAP DELETES IN A TRANSACTION DBA GUY

  • BigityBigity Lubbock, TXRegistered User regular
    LANsweeper, where have you been all my life.

    This thing is damn handy.

  • wunderbarwunderbar What Have I Done? Registered User regular
    Bigity wrote: »
    LANsweeper, where have you been all my life.

    This thing is damn handy.

    YES! We use it extensively where I work. It's our help desk ticketing system and asset management system, as well as a copule other little things I don't even use it for.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
This discussion has been closed.