Options

[sysadmin] on-call schedule - Always you

1568101122

Posts

  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited January 2022
    hot take: most the requirements of HIPAA or Dept of Treasury should be enforced by regulation on ALL businesses, in my opinion

    (PCI goes too far in a lot of ways, like how bowen said)

    But lot of medical and financial IT regulations are a reasonable minimum security baseline for any business of any size, and any business not following them is negligent, and i'd be fine if the law treated it as negligence

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Feral wrote: »
    Thawmus wrote: »
    Whereas I sit here without such garbage, kinda wishing I had such garbage so that I could use it to enforce some sensible policy.

    oh, i definitely prefer working in regulated industries.

    the bureaucracy and paperwork is awful, specially for PCI, but at least regulated companies take cybersecurity and reliability seriously

    at least, more often and more seriously than unregulated companies

    i'd rather have a predictable but shitty PCI or HIPAA or Dept of Treasury exam once or twice a year than have to convince a computer-illiterate CEO why the latest ransomware outbreak was an indirect result of him cutting IT headcount down to me and an intern

    wait, you get an intern?

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    bowenbowen How you doin'? Registered User regular
    Feral wrote: »
    hot take: most the requirements of HIPAA or Dept of Treasury should be enforced by regulation on ALL businesses, in my opinion

    (PCI goes too far in a lot of ways, like how bowen said)

    But lot of medical and financial IT regulations are a reasonable minimum security baseline for any business of any size, and any business not following them is negligent, and i'd be fine if the law treated it as negligence

    Yeah HIPAA is great and I wish most places treated customer information like that.

    PCI still better than HITECH which can go fuck itself

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    wunderbar wrote: »
    Feral wrote: »
    Thawmus wrote: »
    Whereas I sit here without such garbage, kinda wishing I had such garbage so that I could use it to enforce some sensible policy.

    oh, i definitely prefer working in regulated industries.

    the bureaucracy and paperwork is awful, specially for PCI, but at least regulated companies take cybersecurity and reliability seriously

    at least, more often and more seriously than unregulated companies

    i'd rather have a predictable but shitty PCI or HIPAA or Dept of Treasury exam once or twice a year than have to convince a computer-illiterate CEO why the latest ransomware outbreak was an indirect result of him cutting IT headcount down to me and an intern

    wait, you get an intern?

    iknowright? lucky me

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FFFF Once Upon a Time In OaklandRegistered User regular
    Pitching using Teams to manage and track an AD cleanup project to my boss. The alternative used in the past has been "Uh, I had that in my email somewhere..." We have Teams (and like, every MS365 app as well) so I figured we might as well get used to it and try it.

    Funny how I had to go from being a manager under my old boss, to a sys admin under my new boss to actually get to try some project management. ¯\_(ツ)_/¯

    Huh...
  • Options
    MugsleyMugsley DelawareRegistered User regular
    I work for the DoD and we (still) don't have any password manager program as default loadout. And we aren't allowed to install one.

  • Options
    SiliconStewSiliconStew Registered User regular
    Mugsley wrote: »
    I work for the DoD and we (still) don't have any password manager program as default loadout. And we aren't allowed to install one.

    You must go through a lot of sticky notes then.

    Just remember that half the people you meet are below average intelligence.
  • Options
    BlackDragon480BlackDragon480 Bluster Kerfuffle Master of Windy ImportRegistered User regular
    Mugsley wrote: »
    I work for the DoD and we (still) don't have any password manager program as default loadout. And we aren't allowed to install one.

    Any 8 inch floppy drives from ICBM silos still floating around?

    No matter where you go...there you are.
    ~ Buckaroo Banzai
  • Options
    MugsleyMugsley DelawareRegistered User regular
    Mugsley wrote: »
    I work for the DoD and we (still) don't have any password manager program as default loadout. And we aren't allowed to install one.

    You must go through a lot of sticky notes then.

    I know more than a few people who manage their passwords this way

  • Options
    SiliconStewSiliconStew Registered User regular
    Mugsley wrote: »
    Mugsley wrote: »
    I work for the DoD and we (still) don't have any password manager program as default loadout. And we aren't allowed to install one.

    You must go through a lot of sticky notes then.

    I know more than a few people who manage their passwords this way

    I've got over 1000 passwords in my vault currently, all different. I can't imagine trying to do my job in a secure manner without a password manager.

    Just remember that half the people you meet are below average intelligence.
  • Options
    MyiagrosMyiagros Registered User regular
    edited February 2022
    Nothing is more fun than upgrading an internet connection from 100/100 to 500/500 and then having uploading become saturated when people access files over VPN. Bandwidth is only hitting 70Mb tops, and running a speed test doesn't trigger the issue. Of course it's not the ISP causing the issue, or the firewall causing the issue according to their support. :rotate:

    Myiagros on
    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Lots and lots of crypto sessions on a firewall can cause the CPU to spike, limiting its overall throughput. Not saying that's what's going on here but it's not outside the realm of possibility.

  • Options
    SiliconStewSiliconStew Registered User regular
    That_Guy wrote: »
    Lots and lots of crypto sessions on a firewall can cause the CPU to spike, limiting its overall throughput. Not saying that's what's going on here but it's not outside the realm of possibility.

    We have small remote office firewalls that have similar behavior. The VPN throughput is less than the max throughput of the internet interface due to processing overhead. So even if you do a speed test at that location you might get 200/200 Mbps but traffic over the VPN to the home office caps out around 75/75 Mbps. It's not really been an issue for us since far more traffic is internet destined these days vs the small amount still needed for on-prem things like RDS sessions and those small offices only have a handful of employees on site. Busier sites have higher spec units that either have a higher but still capped VPN throughput or units that the VPN throughput can match the max external throughput.

    Just remember that half the people you meet are below average intelligence.
  • Options
    MyiagrosMyiagros Registered User regular
    Maybe it's that then, it's incredibly frustrating to deal with. We are running a Sonicwall NSa 2650 which should be able to handle it, there's about 30-40 VPN users at any time. I have started shifting people onto a RDS server, may just have to force more to that.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    SiliconStewSiliconStew Registered User regular
    I also seem to remember seeing a discussion somewhere where client VPN performance issues were caused by Windows and/or NIC drivers because they improperly use multithreading. Something like each processor was generating it's own separate sequence numbers, making the sequence as a whole be out of order, causing a large number of retransmits and tanking throughput. As I recall they offered no viable solutions or workarounds.

    Excessive retransmits you'd be able to see in a packet capture.

    Just remember that half the people you meet are below average intelligence.
  • Options
    MyiagrosMyiagros Registered User regular
    I'm reviewing the options in this thread and will be testing disabling some of the GAV settings: https://www.reddit.com/r/sonicwall/comments/drengi/gate_way_anti_virus_causing_throughput_issues/

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Me: So let me get this straight, you want me to try and test it out again and to let me know if it works. But you're on site and aren't going to test it from an outside internet connection because why exactly?
    Them: ...

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • Options
    zagdrobzagdrob Registered User regular
    Good news! Recent breaches have resulted in my (parent) organization taking IT security more seriously and instituting a bunch of new policies to address vulnerabilities.

    Bad news! Most of them are like 'wait, IA wasn't doing this already'.

    Like, one of the things they are talking about is setting up a disciplinary process for IT people who are responsible for a data breach either intentionally or through negligence. How...is that just now something being done?

    Also an obsession with getting all logs of everything into Splunk. Which is good and fair! But also the Splunk team is completely understaffed and overwhelmed and are like a three week backlog to do anything, and all these vulnerability reports require 30 day corrections.

    I just love the whole 'took us three years to get to this, but now that we finally did you have 30 days to remediate'.

  • Options
    ThawmusThawmus +Jackface Registered User regular
    zagdrob wrote: »
    Good news! Recent breaches have resulted in my (parent) organization taking IT security more seriously and instituting a bunch of new policies to address vulnerabilities.

    Bad news! Most of them are like 'wait, IA wasn't doing this already'.

    Like, one of the things they are talking about is setting up a disciplinary process for IT people who are responsible for a data breach either intentionally or through negligence. How...is that just now something being done?

    Also an obsession with getting all logs of everything into Splunk. Which is good and fair! But also the Splunk team is completely understaffed and overwhelmed and are like a three week backlog to do anything, and all these vulnerability reports require 30 day corrections.

    I just love the whole 'took us three years to get to this, but now that we finally did you have 30 days to remediate'.

    Honestly I think that's horseshit because 9/10 that's not IT's fault. This reads to me as a way to scapegoat IT for shitty management decisions that force IT into a corner, which happens fucking always.

    Twitch: Thawmus83
  • Options
    bowenbowen How you doin'? Registered User regular
    Thawmus wrote: »
    zagdrob wrote: »
    Good news! Recent breaches have resulted in my (parent) organization taking IT security more seriously and instituting a bunch of new policies to address vulnerabilities.

    Bad news! Most of them are like 'wait, IA wasn't doing this already'.

    Like, one of the things they are talking about is setting up a disciplinary process for IT people who are responsible for a data breach either intentionally or through negligence. How...is that just now something being done?

    Also an obsession with getting all logs of everything into Splunk. Which is good and fair! But also the Splunk team is completely understaffed and overwhelmed and are like a three week backlog to do anything, and all these vulnerability reports require 30 day corrections.

    I just love the whole 'took us three years to get to this, but now that we finally did you have 30 days to remediate'.

    Honestly I think that's horseshit because 9/10 that's not IT's fault. This reads to me as a way to scapegoat IT for shitty management decisions that force IT into a corner, which happens fucking always.

    "do you think hackers are breaking into our system?"

    "who the fuck knows, I have neither the time nor the budget to monitor our system, and you laughed at me when I wanted to get an intrusion detection system"

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    That_GuyThat_Guy I don't wanna be that guy Registered User regular
    bowen wrote: »
    Thawmus wrote: »
    zagdrob wrote: »
    Good news! Recent breaches have resulted in my (parent) organization taking IT security more seriously and instituting a bunch of new policies to address vulnerabilities.

    Bad news! Most of them are like 'wait, IA wasn't doing this already'.

    Like, one of the things they are talking about is setting up a disciplinary process for IT people who are responsible for a data breach either intentionally or through negligence. How...is that just now something being done?

    Also an obsession with getting all logs of everything into Splunk. Which is good and fair! But also the Splunk team is completely understaffed and overwhelmed and are like a three week backlog to do anything, and all these vulnerability reports require 30 day corrections.

    I just love the whole 'took us three years to get to this, but now that we finally did you have 30 days to remediate'.

    Honestly I think that's horseshit because 9/10 that's not IT's fault. This reads to me as a way to scapegoat IT for shitty management decisions that force IT into a corner, which happens fucking always.

    "do you think hackers are breaking into our system?"

    "who the fuck knows, I have neither the time nor the budget to monitor our system, and you laughed at me when I wanted to get an intrusion detection system"

    The only response I can think of to scapegoating IT for breaches is "I quit, effective immediately." Good luck ever getting another IT person who would take on that kind of responsibility.

  • Options
    ThawmusThawmus +Jackface Registered User regular
    Consequences for negligence and fraud should always start at the very top. But the top doesn't like that and they set the policy, so this is always gonna be bullshit.

    Twitch: Thawmus83
  • Options
    Inquisitor77Inquisitor77 2 x Penny Arcade Fight Club Champion A fixed point in space and timeRegistered User regular
    I recall a piece of fiction where there was some sort of elected official who didn't actually make decisions and just lived a lap of luxury in exchange for being the one who gets executed if the shit hits the fan.

  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    Splunk team? Y’all have a team for that? Where I work it’s just me now that our networking guy took a better job elsewhere.

  • Options
    schussschuss Registered User regular
    Thawmus wrote: »
    zagdrob wrote: »
    Good news! Recent breaches have resulted in my (parent) organization taking IT security more seriously and instituting a bunch of new policies to address vulnerabilities.

    Bad news! Most of them are like 'wait, IA wasn't doing this already'.

    Like, one of the things they are talking about is setting up a disciplinary process for IT people who are responsible for a data breach either intentionally or through negligence. How...is that just now something being done?

    Also an obsession with getting all logs of everything into Splunk. Which is good and fair! But also the Splunk team is completely understaffed and overwhelmed and are like a three week backlog to do anything, and all these vulnerability reports require 30 day corrections.

    I just love the whole 'took us three years to get to this, but now that we finally did you have 30 days to remediate'.

    Honestly I think that's horseshit because 9/10 that's not IT's fault. This reads to me as a way to scapegoat IT for shitty management decisions that force IT into a corner, which happens fucking always.

    Ehhhhhh, S3 bucket negligence is pretty common.

  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    New insurance company fun.

    They asked if we had remediated a specific CVE. The number was from 2014 which was... ok.. sure. I look up the CVE and it was the POODLE vulnerability in SSL 3.0.

    Ok, so... from 2014, SSL 3.0 officially deprecated in 2015 so all should be good.

    Spot check one of our servers with IIS and.... fails. sad;fs;ljkfds;lkjsfa;ljksdfl;kjfdsfal;jk

    After I stop swearing and calm down, go look at our other two IIS servers... and they check out ok.

    So at least I have only one server to fix and not 3+.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    NosfNosf Registered User regular
    edited February 2022
    Once again i have asserted my technical dominance at work, and my position as IT manager because

    I can fucking google a solution to a problem better than anyone else

    the sweet kiss of oblivion cannot claim me soon enough

    edit: porting phone #s to our sip service - send in paperwork - they come back with this crazy spreadsheet i have never seen before, immediately see a tranposed digit - fix and note it, add other info requested

    "hm, better do a once over before mailing back, i feel paranoid"

    oh they weren't going to port the actual main lines for that site, they were going to leave it behind

    ...

    edit 2: To their credit, they called me and confirmed some of the changes and numbers, so good for them!

    Nosf on
  • Options
    electricitylikesmeelectricitylikesme Registered User regular
    I love the experience of having to deploy some vendor's Java application. They're unhelpful as anything about actual deployment concerns or modern architecture, so of course the easiest way to get anything done is to decompile the whole thing with Jetbrains.

    Next up would be to learn to bytecode patch tomcat apps. We're skirting it.

  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    edited March 2022
    The last 90 minutes of my life have been... very interesting.

    APC UPS in the server room started throwing battery replacement needed errors, UPS is inside of warranty period, so call APC to start a warranty claim.

    Due to supply chain issues, I'm told that they do not have replacement batteries in stock, and don't expect new stock of replacement batteries until June. They are, however, willing to send me AN ENTIRE NEW UPS instead. The UPS would be the same model number as our existing one, which I would presume has the same battery in it.

    I questioned that to the person on the phone and she said basically "yeah... we have had a few customers point that out."

    If I could get a replacement UPS in, yank the battery out of that, and then just send the new UPS back I'd do that, but they track the serial numbers on the exchange so I can't do that. So my two options are to get a brand new UPS, which then requires a couple of hours of downtime on the weekend to change out, or to wait until June for a replacement battery.

    The cherry on top of this is that I checked with our reseller and based of their website it looks like they have the battery model I need in stock with same day shipping if ordered before 3pm ET. So I reached out to our account manager to confirm that and if they do I might just spend $400 on a new battery, get the faulty one replaced and then have a spare battery, I guess?

    EDIT: I almost forgot the best part of this: The support person I was on the phone with had an honest to god rooster in the background going full on "cock a doodle do" every 3 minutes during the 45 minutes I was on the phone with her, to the point that she was apologizing for it. What an afternoon.

    wunderbar on
    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    ThawmusThawmus +Jackface Registered User regular
    My guess is that they don't want to have a bunch of open-box battery-less UPS's in stock, so this is what they're doing instead.

    Twitch: Thawmus83
  • Options
    DarkewolfeDarkewolfe Registered User regular
    Large orgs and what you can jam through your ticket software can be real fuckin weird.

    What is this I don't even.
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Thawmus wrote: »
    My guess is that they don't want to have a bunch of open-box battery-less UPS's in stock, so this is what they're doing instead.

    oh, for sure I'm not expeting them to go rip a battery out of a ready to ship UPS in a box. That'd be silly. What's silly is the lack of inventory on the batteries alone when they do seem to have stock of the UPS's those batteries go into.

    The whole situation is unfortunate because my options are down to needing to schedule downtime to replace an entire UPS because of a battery issue, living with a potential battery issue until june, or spending $550CAD on a new battery when the existing one is still under warranty.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    SiliconStewSiliconStew Registered User regular
    wunderbar wrote: »
    Thawmus wrote: »
    My guess is that they don't want to have a bunch of open-box battery-less UPS's in stock, so this is what they're doing instead.

    oh, for sure I'm not expeting them to go rip a battery out of a ready to ship UPS in a box. That'd be silly. What's silly is the lack of inventory on the batteries alone when they do seem to have stock of the UPS's those batteries go into.

    The whole situation is unfortunate because my options are down to needing to schedule downtime to replace an entire UPS because of a battery issue, living with a potential battery issue until june, or spending $550CAD on a new battery when the existing one is still under warranty.

    I'd start the RMA process on the battery and buy the new one now so it can be installed without the downtime for pulling the UPS itself. Keep the battery you eventually get back from RMA on the shelf for when the new one needs to be lifecycled in 3 years or for any possible failures in between. You should be keeping spare UPS batteries on hand anyway to limit the time your business is exposed to downtime from potential power failures. I certainly wouldn't want to rely on never having any power issues for several months waiting on backordered batteries.

    Just remember that half the people you meet are below average intelligence.
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    wunderbar wrote: »
    Thawmus wrote: »
    My guess is that they don't want to have a bunch of open-box battery-less UPS's in stock, so this is what they're doing instead.

    oh, for sure I'm not expeting them to go rip a battery out of a ready to ship UPS in a box. That'd be silly. What's silly is the lack of inventory on the batteries alone when they do seem to have stock of the UPS's those batteries go into.

    The whole situation is unfortunate because my options are down to needing to schedule downtime to replace an entire UPS because of a battery issue, living with a potential battery issue until june, or spending $550CAD on a new battery when the existing one is still under warranty.

    I'd start the RMA process on the battery and buy the new one now so it can be installed without the downtime for pulling the UPS itself. Keep the battery you eventually get back from RMA on the shelf for when the new one needs to be lifecycled in 3 years or for any possible failures in between. You should be keeping spare UPS batteries on hand anyway to limit the time your business is exposed to downtime from potential power failures. I certainly wouldn't want to rely on never having any power issues for several months waiting on backordered batteries.

    Yeah, that was my recommendation to my manager, just waiting to hear back on approval for the spend. the UPS isn't super robust though, just a single battery UPS so it's not like even having a second one on the shelf means that we can extend runtime on the servers attached since I can't swap in a new battery without power to the mains anyway. Regardless I agree having a spare battery on hand in case there is an issue is a good idea.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    MyiagrosMyiagros Registered User regular
    Any suggestions on software to use for file server auditing?

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    You haven't lived until you've heard a fortinet engineer say "huh, I've never seen this before" while he's looking at a firewall that is a 5 hour drive away that we're connected to via a laptop plugged directly in that is tethered to a cell phone that I'm doing a screen share session to.

    GUI was broken, commands in the CLI via putty were taking 30+ seconds to run despite the fact that the device was idling at 0% CPU and 30% memory, and there was nothing else connected to the firewall, we had the WAN and the LAN unplugged. We were trying to use the TFTP instance and were getting a speed of 200 bytes/s. Bytes.....

    Ended up having to terrifingly factory reset it and pray. It came back just fine and then restored from a backup config. Engineer's best guess was the file system got corrupted.

    That was a fun morning.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    That_GuyThat_Guy I don't wanna be that guy Registered User regular
    wunderbar wrote: »
    You haven't lived until you've heard a fortinet engineer say "huh, I've never seen this before" while he's looking at a firewall that is a 5 hour drive away that we're connected to via a laptop plugged directly in that is tethered to a cell phone that I'm doing a screen share session to.

    GUI was broken, commands in the CLI via putty were taking 30+ seconds to run despite the fact that the device was idling at 0% CPU and 30% memory, and there was nothing else connected to the firewall, we had the WAN and the LAN unplugged. We were trying to use the TFTP instance and were getting a speed of 200 bytes/s. Bytes.....

    Ended up having to terrifingly factory reset it and pray. It came back just fine and then restored from a backup config. Engineer's best guess was the file system got corrupted.

    That was a fun morning.

    Hey, at least Fortinet has people that can help you. In Cisco land you don't get customer support. They managed to find a racket where people will pay them to learn how to support their own products.

  • Options
    AiouaAioua Ora Occidens Ora OptimaRegistered User regular
    you can never get away from IT work, the users will always find you no matter how far you run


    I develop this tool that ingests data from a big ol data lake, does some reporting on it
    problem is now I get these program managers who want access to my data, instead of going and getting it from the lake
    Ideally my answer would be no, but ofc i don't get to decide that

    fine, here's an account to my db, it's just boring mysql have fun

    no, now I gotta hold everyone's hand on how they can connect their excel or whatever program, answer all their questions about the db layout, they want me to try and troubleshoot their queries

    mfers you asked for special access and now you want me to do your job for you too

    life's a game that you're bound to lose / like using a hammer to pound in screws
    fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
    that's right we're on a fucked up cruise / God is dead but at least we have booze
    bad things happen, no one knows why / the sun burns out and everyone dies
  • Options
    FeldornFeldorn Mediocre Registered User regular
    That_Guy wrote: »
    wunderbar wrote: »
    You haven't lived until you've heard a fortinet engineer say "huh, I've never seen this before" while he's looking at a firewall that is a 5 hour drive away that we're connected to via a laptop plugged directly in that is tethered to a cell phone that I'm doing a screen share session to.

    GUI was broken, commands in the CLI via putty were taking 30+ seconds to run despite the fact that the device was idling at 0% CPU and 30% memory, and there was nothing else connected to the firewall, we had the WAN and the LAN unplugged. We were trying to use the TFTP instance and were getting a speed of 200 bytes/s. Bytes.....

    Ended up having to terrifingly factory reset it and pray. It came back just fine and then restored from a backup config. Engineer's best guess was the file system got corrupted.

    That was a fun morning.

    Hey, at least Fortinet has people that can help you. In Cisco land you don't get customer support. They managed to find a racket where people will pay them to learn how to support their own products.

    I feel like some Citrix products are right there too.

  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    That_Guy wrote: »
    wunderbar wrote: »
    You haven't lived until you've heard a fortinet engineer say "huh, I've never seen this before" while he's looking at a firewall that is a 5 hour drive away that we're connected to via a laptop plugged directly in that is tethered to a cell phone that I'm doing a screen share session to.

    GUI was broken, commands in the CLI via putty were taking 30+ seconds to run despite the fact that the device was idling at 0% CPU and 30% memory, and there was nothing else connected to the firewall, we had the WAN and the LAN unplugged. We were trying to use the TFTP instance and were getting a speed of 200 bytes/s. Bytes.....

    Ended up having to terrifingly factory reset it and pray. It came back just fine and then restored from a backup config. Engineer's best guess was the file system got corrupted.

    That was a fun morning.

    Hey, at least Fortinet has people that can help you. In Cisco land you don't get customer support. They managed to find a racket where people will pay them to learn how to support their own products.

    Yes, this is reason 2349827349387 why I won't touch Cisco products with a 10 foot pole.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
Sign In or Register to comment.