As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

[sysadmin] on-call schedule - Always you

1356720

Posts

  • InfidelInfidel Heretic Registered User regular
    That_Guy wrote: »
    Infidel wrote: »
    That_Guy wrote: »
    What do you have the IPsec tunnel timeout set to?

    Uh keepalive frequency 10, key lifetime 86400 phase 1 / 43200 phase 2?

    The phase 2 selector is setup at a /16 for these servers so it's weird that one host works but the adjacent one doesn't... which is making me think this probably isn't Fortigate stuff at all really? Hmmmm. Maybe it's just the host networking being fubar? But then a little weird that a Fortigate reboot helps.

    Does the link status actually show "up" on both ends of the tunnel? I'm also wondering if you are using Forticloud and/or if you can access both firewalls remotely via wan when you're having the vpn traffic issue. I'm no expert but I'm happy to compare your settings to mine. I may be able to ping some experts on my team that can answer specific questions.

    Yep, would all show up and work for other hosts over the same tunnel.

    OrokosPA.png
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Infidel wrote: »
    That_Guy wrote: »
    Infidel wrote: »
    That_Guy wrote: »
    What do you have the IPsec tunnel timeout set to?

    Uh keepalive frequency 10, key lifetime 86400 phase 1 / 43200 phase 2?

    The phase 2 selector is setup at a /16 for these servers so it's weird that one host works but the adjacent one doesn't... which is making me think this probably isn't Fortigate stuff at all really? Hmmmm. Maybe it's just the host networking being fubar? But then a little weird that a Fortigate reboot helps.

    Does the link status actually show "up" on both ends of the tunnel? I'm also wondering if you are using Forticloud and/or if you can access both firewalls remotely via wan when you're having the vpn traffic issue. I'm no expert but I'm happy to compare your settings to mine. I may be able to ping some experts on my team that can answer specific questions.

    Yep, would all show up and work for other hosts over the same tunnel.

    I reread the original post a couple of times and did some pondering. Stupid question. Could you have anything on the 2.X network that could be using/broadcasting 1.2? Like maybe a rogue statistically assigned device?

  • InfidelInfidel Heretic Registered User regular
    That_Guy wrote: »
    Infidel wrote: »
    That_Guy wrote: »
    Infidel wrote: »
    That_Guy wrote: »
    What do you have the IPsec tunnel timeout set to?

    Uh keepalive frequency 10, key lifetime 86400 phase 1 / 43200 phase 2?

    The phase 2 selector is setup at a /16 for these servers so it's weird that one host works but the adjacent one doesn't... which is making me think this probably isn't Fortigate stuff at all really? Hmmmm. Maybe it's just the host networking being fubar? But then a little weird that a Fortigate reboot helps.

    Does the link status actually show "up" on both ends of the tunnel? I'm also wondering if you are using Forticloud and/or if you can access both firewalls remotely via wan when you're having the vpn traffic issue. I'm no expert but I'm happy to compare your settings to mine. I may be able to ping some experts on my team that can answer specific questions.

    Yep, would all show up and work for other hosts over the same tunnel.

    I reread the original post a couple of times and did some pondering. Stupid question. Could you have anything on the 2.X network that could be using/broadcasting 1.2? Like maybe a rogue statistically assigned device?

    They're all statically assigned, by me, on these networks. Don't see anything else that could be on it but maybe it's a device blackhole due to ARP caching or similar getting goofed up and I'm not resetting the right devices. Gonna find a time to restart the destination device and network.

    OrokosPA.png
  • wunderbarwunderbar What Have I Done? Registered User regular
    Sigh. You silly americans and your weird Thursday holiday. Just realized the vendor I'm waiting to hear from probably won't be getting back to me until Monday.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    A lot of our clients take the whole week off.

  • wunderbarwunderbar What Have I Done? Registered User regular
    Gave notice today, leaving this startup that's a complete mess of an organization to go work for a company where i'll be a sysadmin in a microsoft shop with the task/directive of doing an on-prem to cloud infrastructure migration next year.

    I took a job at the current place after being unemployed for almost a year, and just took whatever I could, and it ended up not being a good fit. Who knows, maybe I'll hate the new place, but at least the work is more in my wheelhouse and less "whatever random administrative shit we tell you to do."

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    It's a worker's market out there. Get you that $$$.

  • wunderbarwunderbar What Have I Done? Registered User regular
    That_Guy wrote: »
    It's a worker's market out there. Get you that $$$.

    It wasn't even about money to me. i'm making about 4% more than I was at my soon to be last job. I care much more about fit and the actual work. Will the new place be better? I'm not 100% sure, but I know I needed a change. At worst, I'll spend 2022 leading a cloud migration project and get to stick that on my resume for future opportunity.

    This year feels like a bit wasted, as I can't really say I have added too many things to my resume this year, except for maybe using a Mac as a work computer for the first time, and administering the messes that are Google Workspace and Slack.

    I can't believe I'm saying this, but I'm actually looking forward to going back to managing a Microsoft environment instead of a startup mentality of "someone signed up for [random service] 18 months ago because they just needed a thing" and now we have 50 of those services and even after almost a year here I still don't have a full handle on all of the "things" we use.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Let's say I wanted to buy a professional-level or prosumer-level firewall for my home network, and a separate AP. I don't want something that will stop working if I stop paying a yearly maintenance contract. Any recommendations?

    (I already have some ideas of my own but i'm interested in y'alls')

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    one thing i'm considering is getting a minipc with multiple NICs and throwing pfSense on it
    PfSense partners with Netgate for prebuilt pfSense appliances but I don't know how reliable their hardware is

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Feral wrote: »
    Let's say I wanted to buy a professional-level or prosumer-level firewall for my home network, and a separate AP. I don't want something that will stop working if I stop paying a yearly maintenance contract. Any recommendations?

    (I already have some ideas of my own but i'm interested in y'alls')

    FortiGate with a FortiAP or a Ruckus R510/R610/R710. You can flash the Ruckus AP wit unleashed firmware. You can get used FortiGates and Ruckus APs on ebay super cheap.

  • ThawmusThawmus +Jackface Registered User regular
    Honestly firewall-wise if you want something robust just build one yourself and throw OPNsense on it.

    I've been replacing our pfSense firewalls with OPNsense and haven't looked back.

    I have Amplifi for home wifi and I don't like it and don't recommend it.

    Twitch: Thawmus83
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    That_Guy wrote: »
    Feral wrote: »
    Let's say I wanted to buy a professional-level or prosumer-level firewall for my home network, and a separate AP. I don't want something that will stop working if I stop paying a yearly maintenance contract. Any recommendations?

    (I already have some ideas of my own but i'm interested in y'alls')

    FortiGate with a FortiAP or a Ruckus R510/R610/R710. You can flash the Ruckus AP wit unleashed firmware. You can get used FortiGates and Ruckus APs on ebay super cheap.

    Fortigate requires an active contract to download firmware updates. That means you can't patch exploitable CVEs without paying for a subscription. That's a deal-breaker for me.

    I don't expect a device to receive custom blocklists or AV signatures without a subscription, but paywalling critical security patches is a practice I don't abide.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Thawmus wrote: »
    Honestly firewall-wise if you want something robust just build one yourself and throw OPNsense on it.

    I've been replacing our pfSense firewalls with OPNsense and haven't looked back.

    I have Amplifi for home wifi and I don't like it and don't recommend it.

    Neat. I might do that.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • MugsleyMugsley DelawareRegistered User regular
    Does r/homelab have any popular hardware?

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    r/homelab says that the Ruckus Unleashed don't require a subscription for firmware updates, so that's a good option on APs (thanks, mugsley and that_guy)

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Feral wrote: »
    That_Guy wrote: »
    Feral wrote: »
    Let's say I wanted to buy a professional-level or prosumer-level firewall for my home network, and a separate AP. I don't want something that will stop working if I stop paying a yearly maintenance contract. Any recommendations?

    (I already have some ideas of my own but i'm interested in y'alls')

    FortiGate with a FortiAP or a Ruckus R510/R610/R710. You can flash the Ruckus AP wit unleashed firmware. You can get used FortiGates and Ruckus APs on ebay super cheap.

    Fortigate requires an active contract to download firmware updates. That means you can't patch exploitable CVEs without paying for a subscription. That's a deal-breaker for me.

    I don't expect a device to receive custom blocklists or AV signatures without a subscription, but paywalling critical security patches is a practice I don't abide.

    There's nothing preventing you from downloading the firmware and applying it manually.

  • FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    Oh! Hmm. Interesting.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • JaysonFourJaysonFour Classy Monster Kitteh Registered User regular
    Drovek wrote: »
    If you can have powerwashing simulators and trucking simulators, why not IT Support simulators?

    I mentioned over in the Steam thread I was giving this a good playthrough, and well... to quote from my post about some of the problems you end up getting to tackle in that expansion...

    …is it wrong of me to ask, out loud, on reading some of the ongoing issues with some of the workers in the game that you always need to fix (the guy who always hits his computer in frustration when something happens, to the guy who every fucking week has something catch fire in his PC, to the guy who wanted enough power in his case fan for it to be able to slice salami, or the marketing guru who tried to put a t-shirt through the printer (percussive maintenance on that one FTW), for example), “…how in the hell are some of you even able to feed yourselves and drive to work without incident? I mean, really…”

    …real-life people in a company have to be smarter than this… right?



    And yes, I'm looking at you, lady who misplaced her computer right before a big project has to be done by and requires a whole new PC to be built and set up in a few days, and also at the young crypto nut who siphoned off a million bucks from the budget to use as a prize for a crypto-hacking scavenger hunt full of overly complex puzzles... that he lost the answers to... and also to the young intern who thinks IT would be easy because he's watched countless YouTube videos on the subject and was trying to show off to that cute girl in his division about "hot-swapping CPUs" and "overclocking the PSU" and other such things, brings the computer in question to me... and after I fixed it, proceeded to give it back to her and claim HE did it...

    I've come to the opinion IT isn't fucking paid enough for dealing with the stuff that they have to. I mean, really. I'm surprised there isn't the option to open a desk drawer to take a shot out of a flask at times due to how dumb some of these people are... or at least find them and give them a solid Gibbs slap upside the head.

    steam_sig.png
    I can has cheezburger, yes?
  • NosfNosf Registered User regular
    My job is stressful enough as is, why the fuck would I want to to go home and suffer it? (I mean I do, I'm at home now fixing three things remotely at once and one of those things is a defective human.)

  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    And IT simulator would just be 1 person asking the same stupid question over and over again and you having to find a way to not blow your brains out.

  • SeidkonaSeidkona Had an upgrade Registered User regular
    Well new job with a fancy ass title: I made Principal Engineer

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • FeldornFeldorn Mediocre Registered User regular
    That_Guy wrote: »
    And IT simulator would just be 1 person asking the same stupid question over and over again and you having to find a way to not blow your brains out.

    changing the same password every day :rotate:

  • wunderbarwunderbar What Have I Done? Registered User regular
    This is my last week at this job. I've handed off everything except for one piece, so I kinda have nothing to do this week. I literally spend an hour and a half building a TV stand today for something to do.

    I really hope they just don't make me come in on Friday.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • ThawmusThawmus +Jackface Registered User regular
    An IT simulator has to have you spend 35 minutes waiting for someone to realize they're mistyping their brand new password immediately after changing it and confirming it, because they're fucking balls at keyboarding.

    It also needs to have you get in a car and drive for 45 minutes to a site to turn someone's computer on.

    It also needs to have you talk to a user for 85 minutes while you see EOD rapidly approaching, with a whole bunch of shit needing to be done by EOD, knowing that you can't be rude or they could end your employment.

    Twitch: Thawmus83
  • wunderbarwunderbar What Have I Done? Registered User regular
    Thawmus wrote: »
    An IT simulator has to have you spend 35 minutes waiting for someone to realize they're mistyping their brand new password immediately after changing it and confirming it, because they're fucking balls at keyboarding.

    It also needs to have you get in a car and drive for 45 minutes to a site to turn someone's computer on.

    It also needs to have you talk to a user for 85 minutes while you see EOD rapidly approaching, with a whole bunch of shit needing to be done by EOD, knowing that you can't be rude or they could end your employment.

    I did once drive for an hour to plug a monitor in, despite the user telling me for sure that it was plugged in, and was just dead. the user couldn't even be bothered to apologize.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • wunderbarwunderbar What Have I Done? Registered User regular
    anyone else having fun this morning with AWS US-East-1 seemingly disappearing off the face of the earth for a bit?

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Yeah, AWS has been having issues all day. It seems to be indirectly affecting several of our tools too.

  • ThawmusThawmus +Jackface Registered User regular
    I'm simultaneously very sorry you guys are dealing with that and very glad I'm not dealing with that.

    Twitch: Thawmus83
  • wunderbarwunderbar What Have I Done? Registered User regular
    Our dev team is scrambling because I think their backup solution to an AWS issue was that AWS doesn't go down. I don't even think they had set up geographic redundancy on the tools we have that are down.

    But thankfully that's not my problem in my role and I only have a couple of days left working here so it's even less of my problem.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • ThawmusThawmus +Jackface Registered User regular
    wunderbar wrote: »
    Our dev team is scrambling because I think their backup solution to an AWS issue was that AWS doesn't go down. I don't even think they had set up geographic redundancy on the tools we have that are down.

    But thankfully that's not my problem in my role and I only have a couple of days left working here so it's even less of my problem.

    I mean at a certain point if AWS is gonna go down it's gonna go down and you're fucking down and just goddamn accept it and watch an anime or something until it's back. I think sometimes we obsess way too much with uptime, when even these megacorps can't ensure it. Like, AWS has redundancy themselves, that's what you're paying them for. If that redundancy has failed, it's failed, time to just face facts and go to lunch.

    Twitch: Thawmus83
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Thawmus wrote: »
    wunderbar wrote: »
    Our dev team is scrambling because I think their backup solution to an AWS issue was that AWS doesn't go down. I don't even think they had set up geographic redundancy on the tools we have that are down.

    But thankfully that's not my problem in my role and I only have a couple of days left working here so it's even less of my problem.

    I mean at a certain point if AWS is gonna go down it's gonna go down and you're fucking down and just goddamn accept it and watch an anime or something until it's back. I think sometimes we obsess way too much with uptime, when even these megacorps can't ensure it. Like, AWS has redundancy themselves, that's what you're paying them for. If that redundancy has failed, it's failed, time to just face facts and go to lunch.

    Yeah, this. Today has been minimally productive due to AWS going down but who cares. Shit's going to break sometimes. I never want to be so busy that a half day's outage will kill me.

  • Inquisitor77Inquisitor77 2 x Penny Arcade Fight Club Champion A fixed point in space and timeRegistered User regular
    The thing with the cloud that everyone seems content to ignore is that the issues may be less frequent, but when they do happen they are giant clusterfucks. In some ways it's good, because your clients, customers, and users can't really complain when the entire eastern seaboard is without the internet. But in a lot of other ways it is bad because you really shouldn't have the entire eastern seaboard without internet in a modern world that assumes that the internet is always available.

  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    The thing with the cloud that everyone seems content to ignore is that the issues may be less frequent, but when they do happen they are giant clusterfucks. In some ways it's good, because your clients, customers, and users can't really complain when the entire eastern seaboard is without the internet. But in a lot of other ways it is bad because you really shouldn't have the entire eastern seaboard without internet in a modern world that assumes that the internet is always available.

    The internet is being held together by shoestrings and bubblegum. Given the current state of technical literacy of most users, it's a wonder the internet works as well as it does.

  • DrovekDrovek Registered User regular
    Thawmus wrote: »
    wunderbar wrote: »
    Our dev team is scrambling because I think their backup solution to an AWS issue was that AWS doesn't go down. I don't even think they had set up geographic redundancy on the tools we have that are down.

    But thankfully that's not my problem in my role and I only have a couple of days left working here so it's even less of my problem.

    I mean at a certain point if AWS is gonna go down it's gonna go down and you're fucking down and just goddamn accept it and watch an anime or something until it's back. I think sometimes we obsess way too much with uptime, when even these megacorps can't ensure it. Like, AWS has redundancy themselves, that's what you're paying them for. If that redundancy has failed, it's failed, time to just face facts and go to lunch.

    Basically this.

    Our plan is just "wait till AWS fixes stuff." We do have backups outside the regions and could (eventually) move to another region, but that effort is measured in days, so if the AWS outage is going to be a few hours then there's not much to do other than monitor the situation.

    steam_sig.png( < . . .
  • FeldornFeldorn Mediocre Registered User regular
    Move to the cloud they said.

    Eliminate your single points of failure they said.

  • Bendery It Like BeckhamBendery It Like Beckham Hopeless Registered User regular
    It's Wednesday my doods.

    "Hmm, these uninstall scripts aren't setup to detect properly lemme just adjust this so it doesn't impact my deployment today..."

    And I've triggered ~1200 AV reinstalls :+1:

  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Feldorn wrote: »
    Move to the cloud they said.

    Eliminate your single points of failure they said.

    At least when the cloud goes down you have an excuse to slack off. When a local service goes offline everyone's in panic mode.

  • AkimboEGAkimboEG Mr. Fancypants Wears very fine pants indeedRegistered User regular
    One nice thing with my previous employer was that in every contract we signed with clients, even with giant multinationals, we had GCP going down listed an "act of god". That is, we can't be held accountable for it if and when it happens, regardless of SLA.

    Give me a kiss to build a dream on; And my imagination will thrive upon that kiss; Sweetheart, I ask no more than this; A kiss to build a dream on
  • FFFF Once Upon a Time In OaklandRegistered User regular
    Welp, I've put myself in it now, hah. I'm not requesting an answer but maybe if somebody could point me to a good learning resource or two.

    I know the below two echo lines, and ways of calling the timestamp function, are functionally equivalent as in they give the same output, but I don't know why.

    If I'm understanding the commented pair correctly, echo is echoing the date command, then the sentence, and that output of the date command + sentence is being piped to tee that puts it all in a log file, while also letting the output be seen at the console. I'm not sure about maybe having a double use of the echo command though.

    The uncommented pair, is expanding the timestamp function that then tells echo to use the sentence after it as a parameter?
    #!/bin/zsh
     
     timestamp() {
     #      echo `date "+%a %b %d %T"`
             echo `date "+%a %b %d %T"` "${1}"
     }
     LOGFILE="logs/test.log"
     # echo $(timestamp) "This should be the day/time formatting I want" | tee -a $LOGFILE
     timestamp "This should be the day/time formatting I want" | tee -a $LOGFILE
    

    Huh...
Sign In or Register to comment.