As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

[sysadmin] on-call schedule - Always you

1679111221

Posts

  • Options
    twmjrtwmjr Registered User regular
    You guys don't have direct Cisco SME contacts for every technology area when you need something? :)

    In more seriousness, I'm assuming this is a business size/relationship thing. Getting support as a small/medium-sized customer must be an entirely different experience.

  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    It mostly depends on how much (or if) you’re willing to pay. We’re a small-ish company but we are allowed to spend in the medium business range sometimes. We classified our networking stuff as critical (especially with the cybersecurity push) so we can pay Cisco and an outside contractor for help.

    Our current problem, however, is that we lost our networking guy and his 12 years of experience in our weird environment, so even explaining the problem can be difficult.

  • Options
    SiliconStewSiliconStew Registered User regular
    It mostly depends on how much (or if) you’re willing to pay. We’re a small-ish company but we are allowed to spend in the medium business range sometimes. We classified our networking stuff as critical (especially with the cybersecurity push) so we can pay Cisco and an outside contractor for help.

    Our current problem, however, is that we lost our networking guy and his 12 years of experience in our weird environment, so even explaining the problem can be difficult.

    Our last small acquisition had that issue. Their current IT staff had been there all of 3 months after the previous group bailed and knew next to nothing about the environment. Even the MSP they were using for some stuff barely knew what they were doing. Basically spent two months learning their environment and doing that MSP's work for them just so we could then convert their environment into ours.

    Just remember that half the people you meet are below average intelligence.
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    edited March 2022
    It mostly depends on how much (or if) you’re willing to pay. We’re a small-ish company but we are allowed to spend in the medium business range sometimes. We classified our networking stuff as critical (especially with the cybersecurity push) so we can pay Cisco and an outside contractor for help.

    Our current problem, however, is that we lost our networking guy and his 12 years of experience in our weird environment, so even explaining the problem can be difficult.

    Our last small acquisition had that issue. Their current IT staff had been there all of 3 months after the previous group bailed and knew next to nothing about the environment. Even the MSP they were using for some stuff barely knew what they were doing. Basically spent two months learning their environment and doing that MSP's work for them just so we could then convert their environment into ours.

    Literally my life. I was hired to replace a retiring sysadmin and on the day I started the manager, who also has sysadmin background and as we're finding out now was doing a lot more day to day administration and break/fix stuff than a manager should have been doing told me he had given his notice the friday before and was gone at the end of that week (this was right before christmas). I had about 4 weeks of overlap with the retiring sysadmin and that was it.

    So we have the Tier 1 guy who has been here for almsot 3 years, then me at 3 months, and the new IT manager at 3 weeks. the new manager was hired to be a manager and not a sysadmin like the last guy (which is good, and I agree with), but I'm drowning here. Had a couple outages in the last couple days that have taken 3x as long to fix as they will the next time because I was figuring out as I went.

    wunderbar on
    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    zerzhulzerzhul Registered User, Moderator mod
    For very qualified people in most fields, "huh, never seen this before" is the best thing for the expert, and the worst thing to hear as the client/user.

  • Options
    LD50LD50 Registered User regular
    zerzhul wrote: »
    For very qualified people in most fields, "huh, never seen this before" is the best thing for the expert, and the worst thing to hear as the client/user.

    My personal favorite is "Oh dear."

  • Options
    zerzhulzerzhul Registered User, Moderator mod
    LD50 wrote: »
    zerzhul wrote: »
    For very qualified people in most fields, "huh, never seen this before" is the best thing for the expert, and the worst thing to hear as the client/user.
    My personal favorite is "Oh dear."
    "This is going to be fun!"

  • Options
    FFFF Once Upon a Time In OaklandRegistered User regular
    zerzhul wrote: »
    LD50 wrote: »
    zerzhul wrote: »
    For very qualified people in most fields, "huh, never seen this before" is the best thing for the expert, and the worst thing to hear as the client/user.
    My personal favorite is "Oh dear."
    "This is going to be fun!"

    I've used a variety but, "Well this is interesting!" and "Oh my!" were the best received.

    Huh...
  • Options
    FeldornFeldorn Mediocre Registered User regular
    as long as it isn't "aww shit"

  • Options
    electricitylikesmeelectricitylikesme Registered User regular
    There are too many people in "security" today who need to be fired because they don't know what they're doing.

    No you muppet: blocking outbound ports other then 80 and 443 doesn't prevent an intruder in anyway from getting data out. It does however mean I can't use the rsync daemon to mirror content efficiently. You have contributed literally nothing other then making normal, non-covert business more difficult and the fact you thought this helped in some way means you're a god damn idiot.

  • Options
    MugsleyMugsley DelawareRegistered User regular
    There are too many people in "security" today who need to be fired because they don't know what they're doing.

    No you muppet: blocking outbound ports other then 80 and 443 doesn't prevent an intruder in anyway from getting data out. It does however mean I can't use the rsync daemon to mirror content efficiently. You have contributed literally nothing other then making normal, non-covert business more difficult and the fact you thought this helped in some way means you're a god damn idiot.

    "But I've got SEC+ certs!"

  • Options
    BlackDragon480BlackDragon480 Bluster Kerfuffle Master of Windy ImportRegistered User regular
    Mugsley wrote: »
    There are too many people in "security" today who need to be fired because they don't know what they're doing.

    No you muppet: blocking outbound ports other then 80 and 443 doesn't prevent an intruder in anyway from getting data out. It does however mean I can't use the rsync daemon to mirror content efficiently. You have contributed literally nothing other then making normal, non-covert business more difficult and the fact you thought this helped in some way means you're a god damn idiot.

    "But I've got SEC+ certs!"

    https://youtu.be/Ht0eYGzT5KM

    No matter where you go...there you are.
    ~ Buckaroo Banzai
  • Options
    NaphtaliNaphtali Hazy + Flow SeaRegistered User regular
    edited March 2022
    So I'm not a networking person in any stretch, and I'm having difficulty googling exactly the issues I'm seeing. I just want to understand what's happening so I can explain it to the customer's IT if I have to.

    long story short:

    windows 7 32bit running old HMI application - network connections to remote SQL and local SQL - both nearly instantaneous. no Windows hosts file being used.

    windows 10 64bit running upgraded HMI application - network connections to remote SQL taking upwards of two seconds each, local SQL instant. after remote server added to Windows hosts file, remote connections are now instantaneous again.

    all these PCs and the SQL Server are running on a closed network that's un-managed to my knowledge. no routing tables or anything like that. everything has static IPv4. SQL Server has all named pipes, TCP on port 1433, etc open. nothing's locked down. as far as I can tell the SQL Server was set up exactly as the old one was, and we saw the performance issues happening against both the new and old server (new Server 2019, old Server 2012?).

    I can't find any specific settings on the win 7 box to explain why they don't run into this issue, registry or networking or otherwise.

    Naphtali on
    Steam | Nintendo ID: Naphtali | Wish List
  • Options
    DarkewolfeDarkewolfe Registered User regular
    Have you double checked that it's not just the SQL server shitting itself because they've done something unholy with their db's?

    What is this I don't even.
  • Options
    NaphtaliNaphtali Hazy + Flow SeaRegistered User regular
    Darkewolfe wrote: »
    Have you double checked that it's not just the SQL server shitting itself because they've done something unholy with their db's?

    Nah, it isn't. It's just for some reason TCP name resolution of the server takes longer on the win 10 boxes in both their live environment and our test (with their new sql node, which we'll install at some point). As soon as you add the server name to the windows hosts file on the win 10 machines, everything's fine again. Fucking weird but I'm sure it's just something I'm not smart enough to know to check for.

    Steam | Nintendo ID: Naphtali | Wish List
  • Options
    LuvTheMonkeyLuvTheMonkey High Sierra Serenade Registered User regular
    Is there any difference in the connection string between the old and new apps?

    Molten variables hiss and roar. On my mind-forge, I hammer them into the greatsword Epistemology. Many are my foes this night.
    STEAM | GW2: Thalys
  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    It’s not DNS
    There’s no way it’s DNS
    It was DNS

  • Options
    RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    edited March 2022
    I’m not saying it is DNS…..but it’s likely that DNS is involved somehow.

    Edit: Two seconds sounds suspiciously like the primary DNS server is failing/unavailable/giving an incorrect IP address and then the PC fails over to the secondary DNS. As you said everything is assigned a static IP address (oofa doofa) then that’s the first thing I would check.

    RandomHajile on
  • Options
    SiliconStewSiliconStew Registered User regular
    2 seconds is the normal timeout for failed DNS lookups. Localhost connections wouldn't have such a DNS resolution problem. Fixing by adding a hosts entry tells you your client has a DNS issue.

    Check the app's connection string is using the FQDN of your SQL server, and not just a short name. Short name lookups may fail or take longer depending on the client's DNS suffix settings. Make sure the DNS suffix settings for the network adapter on the Win10 client are correct and also have all the additional suffix entries if there's multiple domains involved.

    Also make sure the client is actually using the correct, reachable DNS servers.

    Just remember that half the people you meet are below average intelligence.
  • Options
    electricitylikesmeelectricitylikesme Registered User regular
    High probability you've got a setup which had broken IPv6, and Win 7 wasn't trying to use it but Win 10 is.

  • Options
    NaphtaliNaphtali Hazy + Flow SeaRegistered User regular
    edited March 2022
    High probability you've got a setup which had broken IPv6, and Win 7 wasn't trying to use it but Win 10 is.

    There's actually no fixed IPv6 set on the Win 7 boxes but it is enabled on the network card. I have them disabled on the network cards in the Win 10 boxes and on the new SQL Server box, but the Win 7 boxes connect to both new & old SQL instantly without issue. Maybe that's the problem there.

    I don't think there is any DNS involved here (which could also be part of the problem). The only things these boxes have set are IPv4 IP and Subnet, no gateway. No DNS servers are set. Their network is just on an unmanaged switch isolated from everything else.

    Connection string is using a Datasource Name via ODBC, it's setup exactly the same on both old and new boxes. Everything's been angled with the intent that we can piecemeal upgrade their system one machine at a time and have zero issue with new and old stuff intermingling. That's basically the problem I guess, I copied the old setup from the Win 7 machine too closely and/or missed something on the machine I guess. I agree the Win 10 boxes are doing something different in trying to resolve the name into an IP (since adding it to the hosts files seems to have solved that), but not entirely sure why because again I'm a hack and don't fully understand networking since it's usually outside my scope of work.

    FWIW, we didn't design this, we were just tapped to do a simple flip and upgrade. It's never a simple flip and upgrade, ever.

    edit: turning on ipv6 on just the win 10 box (even with it off on sql server) removes the sql connection delay, even with the hosts file empty. :rotate:

    edit edit: for fun, turned off ipv6 on the win 7 box and it still was fine, even without having a configured hosts file. idk my bff jill

    Naphtali on
    Steam | Nintendo ID: Naphtali | Wish List
  • Options
    SiliconStewSiliconStew Registered User regular
    Win10 / Windows Server needs IPv6 enabled. Unbinding it can break multiple things in weird ways in a Windows environment because of all the things using it behind the scenes. Add this to the examples.

    Just remember that half the people you meet are below average intelligence.
  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Christ. 3 completely separate but debilitating outages for us this morning.

    Opened my laptop to 15 messages about our inventory management system being down. Thankfully that just required kicking(restarting) that server. But no one could work, and we have a branch that's a timez one away that couldn't do 80% of their work for the first 90 minutes of their day.

    then the internet went down at another branch location, and was down for almost 2 hours. That was an ISP issue, so nothing for me to do there but still, crappy for that branch.

    Now our phone system is down. We use a voip system through our ISP which just basically resells RingCentral and RingCentral is/was down, but the entire phone system is down. So that's good. (at least no one can call me on the phone to tell me about that! :grin: )

    Imma need a drink.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    ThawmusThawmus +Jackface Registered User regular
    Where the fuck is your desk whiskey?

    Do we need to talk?

    Twitch: Thawmus83
  • Options
    LD50LD50 Registered User regular
    Hey, guess what broke today... (hint: it's DNS!)

  • Options
    wunderbarwunderbar What Have I Done? Registered User regular
    Thawmus wrote: »
    Where the fuck is your desk whiskey?

    Do we need to talk?

    I mean I'm at home, I have access to quite a bit of whiskey

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • Options
    ThawmusThawmus +Jackface Registered User regular
    wunderbar wrote: »
    Thawmus wrote: »
    Where the fuck is your desk whiskey?

    Do we need to talk?

    I mean I'm at home, I have access to quite a bit of whiskey

    Okay, phew! I was worried.

    You made me worried.

    Twitch: Thawmus83
  • Options
    SeidkonaSeidkona Had an upgrade Registered User regular
    It has been a long few months and I have been away because job hunting has been exhausting but I finally landed a pretty good gig and holy hell am I relieved.

    I now work for one of the major infrastructure companies that grew out of a major piece of OSS software.

    Mostly just huntin' monsters.
    XBL:Phenyhelm - 3DS:Phenyhelm
  • Options
    ThawmusThawmus +Jackface Registered User regular
    Sysadmin thread, arise!

    So after a lot of consideration, I've decided to start pursuing changing over our environment to Windows. Hooray!......hooray......

    I am hoping that as I do this that upper management will get some sticker shock and reverse some of their recent decisions, but for now I have to keep us moving forward and make shit work.

    The long and the short is that our bread-and-butter internally-developed software is now going to require Windows, and I'm pretty staunchly against running that in WINE. Shit will break all the time and I will have zero tools to fix it.

    I'm pretty fucking depressed about the whole thing. I'm kinda expecting that even if they do say, "Hey holy shit our costs just exploded let's re-think this" that it will just go back to us running everything in WINE and then I'm going to be biting my nails every day from that point forward.

    I'm not going to pretend our environment is perfect but fuck I am not looking forward to figuring out Microsoft licensing and Azure and DC's and Group Policy and securing workstations and all that shit all over again. I've been so happy away from all that. Goddammit.

    Twitch: Thawmus83
  • Options
    FeldornFeldorn Mediocre Registered User regular
    Is your internally-developed software still actively maintained?

    As someone who does work in Windows the licensing is a pain in the ass. The best part is that it's even hard to find a Microsoft employee who can successfully explain it.

    The thing I hate most about Windows is that even with identical hardware and images things will break differently in many instances.

  • Options
    ThawmusThawmus +Jackface Registered User regular
    Feldorn wrote: »
    Is your internally-developed software still actively maintained?

    As someone who does work in Windows the licensing is a pain in the ass. The best part is that it's even hard to find a Microsoft employee who can successfully explain it.

    The thing I hate most about Windows is that even with identical hardware and images things will break differently in many instances.

    It is, we have developers on staff who maintain it and are moving it to a new platform.

    I don't agree with the direction they're taking and have offered alternatives, but I also don't know programming so these alternatives are easy for them to shoot down since I don't know what I'm talking about.

    I've forced many meetings to take place regarding this, they're committed to this direction. It's depressing.

    Twitch: Thawmus83
  • Options
    FeldornFeldorn Mediocre Registered User regular
    Going in a direction where you lack expertise and support is risky move, Cotton, let's see how it works out.

    Godspeed.

    Actually, are they going full Azure and lifting all the services into the Azure services?

  • Options
    ThawmusThawmus +Jackface Registered User regular
    Feldorn wrote: »
    Going in a direction where you lack expertise and support is risky move, Cotton, let's see how it works out.

    Godspeed.

    Actually, are they going full Azure and lifting all the services into the Azure services?

    Well, that part is ultimately up to me. I'm mostly trying to figure out Azure vs having DC's for the Windows domain, because if we're going to have 100+ windows workstations I'm going to have this shit locked down to the point where you gotta ask permission to move the fucking mouse. I am not wiping viruses all week.

    Twitch: Thawmus83
  • Options
    FeldornFeldorn Mediocre Registered User regular
    Thawmus wrote: »
    Feldorn wrote: »
    Going in a direction where you lack expertise and support is risky move, Cotton, let's see how it works out.

    Godspeed.

    Actually, are they going full Azure and lifting all the services into the Azure services?

    Well, that part is ultimately up to me. I'm mostly trying to figure out Azure vs having DC's for the Windows domain, because if we're going to have 100+ windows workstations I'm going to have this shit locked down to the point where you gotta ask permission to move the fucking mouse. I am not wiping viruses all week.

    Strong web and email filtering are what has provided the most benefit for our organization in that regard.

    We need to be much better about properly locked down workstations but don't have the time or personnel to actually manage that.

    I'd look into Azure Virtual Desktop if it's available. That would put you into the cloud-first zone if it isn't a problem. I don't know how that would work if there are any hybrid (on premises) workloads though.

  • Options
    ThawmusThawmus +Jackface Registered User regular
    Feldorn wrote: »
    Thawmus wrote: »
    Feldorn wrote: »
    Going in a direction where you lack expertise and support is risky move, Cotton, let's see how it works out.

    Godspeed.

    Actually, are they going full Azure and lifting all the services into the Azure services?

    Well, that part is ultimately up to me. I'm mostly trying to figure out Azure vs having DC's for the Windows domain, because if we're going to have 100+ windows workstations I'm going to have this shit locked down to the point where you gotta ask permission to move the fucking mouse. I am not wiping viruses all week.

    Strong web and email filtering are what has provided the most benefit for our organization in that regard.

    We need to be much better about properly locked down workstations but don't have the time or personnel to actually manage that.

    I'd look into Azure Virtual Desktop if it's available. That would put you into the cloud-first zone if it isn't a problem. I don't know how that would work if there are any hybrid (on premises) workloads though.

    Fuck I hadn't even thought about web proxy. I'm gonna have to start looking into a web proxy. Goddammit. I haven't had to do that for a goddamned decade.

    Twitch: Thawmus83
  • Options
    schussschuss Registered User regular
    I mean, it's somewhat strange if you're a mostly windows-less environment and the solution is to move it to a windows-reliant solution. Building things to work in linux first is much more common, as the stacks are a bajillion times cheaper to buy/maintain.

  • Options
    That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Man, this little MSP I work for is turning into such a shitshow. A couple of years ago I went from an account tech to inside technical sales. All of the clients I was previously responsible for were shuffled off to other techs. Things just keep getting worse and worse.

    This week one of my old clients reported half of their campus was down. For 4 FUCKING DAYS the account tech and the service team discussed what should be done but no one ACTUALLY did anything. About all we knew was that the link to the other buildings was hard down and the uplink ports on the switch showed down. Not admin down, just down.

    Today the account tech comes back to the project team area asking for help on what he should do next. We go back and forth and decide that someone should have gone onsite days ago. The account tech doesn't seem interested in going out there, I'm buried under a pile of quote requests and solutions that needs engineering, and no one else on the service team is available. Our Jr project tech and I discuss the problem for a while and he volunteers to go out there. He's out there for a an hour tracing down lines and devices only to find that the UPS powering 2 of the fiber media converters went bad. Luckily I sent him up there with a couple extra UPSs so he was able to get them back online right away.

    For 3 days I was telling people on the service team that I think it's probably just power to the media converters. For 3 days no one did a fucking thing until I took charge and got someone who I'm not even in charge of, who isn't even on the service team, out there. We had the entire campus back online inside of an hour.

    I need to start looking for a new job.

  • Options
    BlazeFireBlazeFire Registered User regular
    That_Guy wrote: »
    Man, this little MSP I work for is turning into such a shitshow. A couple of years ago I went from an account tech to inside technical sales. All of the clients I was previously responsible for were shuffled off to other techs. Things just keep getting worse and worse.

    This week one of my old clients reported half of their campus was down. For 4 FUCKING DAYS the account tech and the service team discussed what should be done but no one ACTUALLY did anything. About all we knew was that the link to the other buildings was hard down and the uplink ports on the switch showed down. Not admin down, just down.

    Today the account tech comes back to the project team area asking for help on what he should do next. We go back and forth and decide that someone should have gone onsite days ago. The account tech doesn't seem interested in going out there, I'm buried under a pile of quote requests and solutions that needs engineering, and no one else on the service team is available. Our Jr project tech and I discuss the problem for a while and he volunteers to go out there. He's out there for a an hour tracing down lines and devices only to find that the UPS powering 2 of the fiber media converters went bad. Luckily I sent him up there with a couple extra UPSs so he was able to get them back online right away.

    For 3 days I was telling people on the service team that I think it's probably just power to the media converters. For 3 days no one did a fucking thing until I took charge and got someone who I'm not even in charge of, who isn't even on the service team, out there. We had the entire campus back online inside of an hour.

    I need to start looking for a new job.

    Ctrl+C, Ctrl+V, email to boss.

  • Options
    That_GuyThat_Guy I don't wanna be that guy Registered User regular
    BlazeFire wrote: »
    That_Guy wrote: »
    Man, this little MSP I work for is turning into such a shitshow. A couple of years ago I went from an account tech to inside technical sales. All of the clients I was previously responsible for were shuffled off to other techs. Things just keep getting worse and worse.

    This week one of my old clients reported half of their campus was down. For 4 FUCKING DAYS the account tech and the service team discussed what should be done but no one ACTUALLY did anything. About all we knew was that the link to the other buildings was hard down and the uplink ports on the switch showed down. Not admin down, just down.

    Today the account tech comes back to the project team area asking for help on what he should do next. We go back and forth and decide that someone should have gone onsite days ago. The account tech doesn't seem interested in going out there, I'm buried under a pile of quote requests and solutions that needs engineering, and no one else on the service team is available. Our Jr project tech and I discuss the problem for a while and he volunteers to go out there. He's out there for a an hour tracing down lines and devices only to find that the UPS powering 2 of the fiber media converters went bad. Luckily I sent him up there with a couple extra UPSs so he was able to get them back online right away.

    For 3 days I was telling people on the service team that I think it's probably just power to the media converters. For 3 days no one did a fucking thing until I took charge and got someone who I'm not even in charge of, who isn't even on the service team, out there. We had the entire campus back online inside of an hour.

    I need to start looking for a new job.

    Ctrl+C, Ctrl+V, email to boss.

    Oh yeah, all that is going into a report one of our guys is writing for the company owners, and the clients involved.

  • Options
    BlazeFireBlazeFire Registered User regular
    That_Guy wrote: »
    BlazeFire wrote: »
    That_Guy wrote: »
    Man, this little MSP I work for is turning into such a shitshow. A couple of years ago I went from an account tech to inside technical sales. All of the clients I was previously responsible for were shuffled off to other techs. Things just keep getting worse and worse.

    This week one of my old clients reported half of their campus was down. For 4 FUCKING DAYS the account tech and the service team discussed what should be done but no one ACTUALLY did anything. About all we knew was that the link to the other buildings was hard down and the uplink ports on the switch showed down. Not admin down, just down.

    Today the account tech comes back to the project team area asking for help on what he should do next. We go back and forth and decide that someone should have gone onsite days ago. The account tech doesn't seem interested in going out there, I'm buried under a pile of quote requests and solutions that needs engineering, and no one else on the service team is available. Our Jr project tech and I discuss the problem for a while and he volunteers to go out there. He's out there for a an hour tracing down lines and devices only to find that the UPS powering 2 of the fiber media converters went bad. Luckily I sent him up there with a couple extra UPSs so he was able to get them back online right away.

    For 3 days I was telling people on the service team that I think it's probably just power to the media converters. For 3 days no one did a fucking thing until I took charge and got someone who I'm not even in charge of, who isn't even on the service team, out there. We had the entire campus back online inside of an hour.

    I need to start looking for a new job.

    Ctrl+C, Ctrl+V, email to boss.

    Oh yeah, all that is going into a report one of our guys is writing for the company owners, and the clients involved.

    I hope that Jr Tech gets some recognition too. What the hell is the account tech for if not to go to site when there are unresolved issues?

Sign In or Register to comment.