As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/

[sysadmin] on-call schedule - Always you

17810121320

Posts

  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    BlazeFire wrote: »
    That_Guy wrote: »
    BlazeFire wrote: »
    That_Guy wrote: »
    Man, this little MSP I work for is turning into such a shitshow. A couple of years ago I went from an account tech to inside technical sales. All of the clients I was previously responsible for were shuffled off to other techs. Things just keep getting worse and worse.

    This week one of my old clients reported half of their campus was down. For 4 FUCKING DAYS the account tech and the service team discussed what should be done but no one ACTUALLY did anything. About all we knew was that the link to the other buildings was hard down and the uplink ports on the switch showed down. Not admin down, just down.

    Today the account tech comes back to the project team area asking for help on what he should do next. We go back and forth and decide that someone should have gone onsite days ago. The account tech doesn't seem interested in going out there, I'm buried under a pile of quote requests and solutions that needs engineering, and no one else on the service team is available. Our Jr project tech and I discuss the problem for a while and he volunteers to go out there. He's out there for a an hour tracing down lines and devices only to find that the UPS powering 2 of the fiber media converters went bad. Luckily I sent him up there with a couple extra UPSs so he was able to get them back online right away.

    For 3 days I was telling people on the service team that I think it's probably just power to the media converters. For 3 days no one did a fucking thing until I took charge and got someone who I'm not even in charge of, who isn't even on the service team, out there. We had the entire campus back online inside of an hour.

    I need to start looking for a new job.

    Ctrl+C, Ctrl+V, email to boss.

    Oh yeah, all that is going into a report one of our guys is writing for the company owners, and the clients involved.

    I hope that Jr Tech gets some recognition too. What the hell is the account tech for if not to go to site when there are unresolved issues?

    We were doing troubleshooting in group chat for all to see. I was sure to loudly congratulate him in front of the boss when he got back.

  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Email from user to everyone in their department and the higher-ups:

    The software doesn't work. It messed this thing up. It's so confusing for everyone and needs to be fixed.

    Me after looking at the issue raised and doing a reply all:

    The user made a mistake when creating <insert example here of what they did> and that's why it looks like it messed up when in reality it is working properly.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    I had a similar one the other day. It ended up being a case where one user didn’t have an updated copy of a mailing list and it missed about a dozen people. They tried to blame the mail server and it went all the way to our VP before coming to me. I don’t usually light people up like that, but I replied and included basically their entire supervision chain.

  • FeldornFeldorn Mediocre Registered User regular
    Hopefully you responded to the email with everyone copied :p

  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Feldorn wrote: »
    Hopefully you responded to the email with everyone copied :p

    If they email me or the people who might've seen the issue, I don't care. When they email everyone just because then it's reply all.

    Had an user email everyone about a VPN issue. I replied all for some troubleshooting and they emailed just me to say they had made a mistake. So I replied all again and quoted the user's email to let everyone know to follow the instructions on connecting to the VPN.

    The most annoying was when a user copied everyone about a small problem. I called them up and walked them through the solution and then asked them why they copied everyone when it was just a small problem. They said that "someone" told them to do that. I asked them who and they paused for a long while before saying they didn't remember. Suuuuuuure you don't.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • NaphtaliNaphtali Hazy + Flow SeaRegistered User regular
    lwt1973 wrote: »
    Feldorn wrote: »
    Hopefully you responded to the email with everyone copied :p

    If they email me or the people who might've seen the issue, I don't care. When they email everyone just because then it's reply all.

    Had an user email everyone about a VPN issue. I replied all for some troubleshooting and they emailed just me to say they had made a mistake. So I replied all again and quoted the user's email to let everyone know to follow the instructions on connecting to the VPN.

    The most annoying was when a user copied everyone about a small problem. I called them up and walked them through the solution and then asked them why they copied everyone when it was just a small problem. They said that "someone" told them to do that. I asked them who and they paused for a long while before saying they didn't remember. Suuuuuuure you don't.

    it was the ghost of christmas past

    Steam | Nintendo ID: Naphtali | Wish List
  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    I just had a regular user email me asking what we are going to do when Windows 10 is out of service in 2025.

    Ummmmm...are you that concerned?

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    The answer is "You will have a new computer by then."

  • wunderbarwunderbar What Have I Done? Registered User regular
    lwt1973 wrote: »
    I just had a regular user email me asking what we are going to do when Windows 10 is out of service in 2025.

    Ummmmm...are you that concerned?

    I get the odd person asking me when we're going to move to Windows 11. But not phrased quite that way That's pretty good.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • lwt1973lwt1973 King of Thieves SyndicationRegistered User regular
    Please don't do the thing again that I asked you not to do three times now after you did it over the last five months.

    Thank you.

    "He's sulking in his tent like Achilles! It's the Iliad?...from Homer?! READ A BOOK!!" -Handy
  • BSoBBSoB Registered User regular
    Got info that a manufacturing batch of DIMMs we have is defective.

    Does anyone know a way to get the full SN of a DIMM out of linux? dmidecode is only giving me the last 8.

    I need the digits that show the manufacturing date.

    My other alterative is to hand check the DIMMs on 90 systems.

  • SiliconStewSiliconStew Registered User regular
    BSoB wrote: »
    Got info that a manufacturing batch of DIMMs we have is defective.

    Does anyone know a way to get the full SN of a DIMM out of linux? dmidecode is only giving me the last 8.

    I need the digits that show the manufacturing date.

    My other alterative is to hand check the DIMMs on 90 systems.

    The dmidecode serial number is an 8 character hex string. It's not the last 8 digits, it's an encoded version of what's printed on the human readable physical label.

    You'd need to use something like decode-dimms to see a manufacture date
    https://www.richud.com/wiki/Ubuntu_See_Live_RAM_Timings_Decode_DIMMS

    Just remember that half the people you meet are below average intelligence.
  • MyiagrosMyiagros Registered User regular
    Manual mail migrations are the devil. PSTs, OSTs, accounts that won't add to Outlook properly in order to export the remainder of the info off the old server. Seriously, how do 18 accounts add to Outlook with no issue, then the rest just.... don't. I ended up adding them to Thunderbird, exporting to .eml, converting those to .msg, importing those into a PST. I'm looking at another 10-15 hours this weekend of converting OSTs and importing to 365.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Myiagros wrote: »
    Manual mail migrations are the devil. PSTs, OSTs, accounts that won't add to Outlook properly in order to export the remainder of the info off the old server. Seriously, how do 18 accounts add to Outlook with no issue, then the rest just.... don't. I ended up adding them to Thunderbird, exporting to .eml, converting those to .msg, importing those into a PST. I'm looking at another 10-15 hours this weekend of converting OSTs and importing to 365.

    I know you're already balls deep in that migration but we've been using MigrationWiz from Bittitan for nearly all of our migrations. You still have to babysit it but it requires a LOT less massaging than manually uploading PSTs to exchange online.

    https://www.bittitan.com/migrationwiz/

  • FeldornFeldorn Mediocre Registered User regular
    We used whatever Dell has to migrate from GroupWise 6 years ago and it was mostly pain free after enough planning.

  • MyiagrosMyiagros Registered User regular
    I'm currently doing cleanups, bunch of PSTs, bunch of OSTs. The import went fine, just stuff all over the place, like having 8 different PST files.

    I'll have to check the BitTitan options if this comes up again. The old server had no server-side access since it was from their web hosting company so that left me with manual for the most part.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    Myiagros wrote: »
    I'm currently doing cleanups, bunch of PSTs, bunch of OSTs. The import went fine, just stuff all over the place, like having 8 different PST files.

    I'll have to check the BitTitan options if this comes up again. The old server had no server-side access since it was from their web hosting company so that left me with manual for the most part.

    Ah, in that case we've used EDBMails. Not having access to the mail server can be a pain.

    https://www.edbmails.com/

  • wunderbarwunderbar What Have I Done? Registered User regular
    my tier one dude is on holidays this week so guess who gets to do help desk tickets all week.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • ThawmusThawmus +Jackface Registered User regular
    wunderbar wrote: »
    my tier one dude is on holidays this week so guess who gets to do help desk tickets all week.

    The receptionist?

    Twitch: Thawmus83
  • wunderbarwunderbar What Have I Done? Registered User regular
    Thawmus wrote: »
    wunderbar wrote: »
    my tier one dude is on holidays this week so guess who gets to do help desk tickets all week.

    The receptionist?

    Based on the conversation I just had with her, if I did that I'm pretty sure the building will be burned to the ground by the end of the week.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • wunderbarwunderbar What Have I Done? Registered User regular
    Between not working for a year, working in a slightly different kind of job for another year, it's been about two and a half years since I have spent any time as doing primarily tier 1 work for more than say, half a day at a time. been lucky to have had the last IT job were there were 2 of us that would take tickets when the tier 1 guy went on holidays, so it was split.

    I get a slightly better appreciation for the tier 1 people and marvel at a) how I ever did it as a job, and b) how Tier 1 people get anything done. It's not like we're overwhelmed with tickets here, I'd say the average is about 8-10 requests per day, absolutely something one person can handle on a day to day basis. couple of them take 5 minutes, I've had a couple I've needed to spend a couple hours on. but I haven't fell behind on it, which is good.

    but the constant interruptions are awful. I have projects/tasks that are part of my actual job that I to this point have barely been able to make any tangible progress on this week because I don't go more than about 45 minutes without some kind of interruption.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • ThawmusThawmus +Jackface Registered User regular
    wunderbar wrote: »
    Between not working for a year, working in a slightly different kind of job for another year, it's been about two and a half years since I have spent any time as doing primarily tier 1 work for more than say, half a day at a time. been lucky to have had the last IT job were there were 2 of us that would take tickets when the tier 1 guy went on holidays, so it was split.

    I get a slightly better appreciation for the tier 1 people and marvel at a) how I ever did it as a job, and b) how Tier 1 people get anything done. It's not like we're overwhelmed with tickets here, I'd say the average is about 8-10 requests per day, absolutely something one person can handle on a day to day basis. couple of them take 5 minutes, I've had a couple I've needed to spend a couple hours on. but I haven't fell behind on it, which is good.

    but the constant interruptions are awful. I have projects/tasks that are part of my actual job that I to this point have barely been able to make any tangible progress on this week because I don't go more than about 45 minutes without some kind of interruption.

    Yeah I just yell at people if I get interrupted a bunch. I know it's unprofessional but I have ADD and I can't keep having my thought process derailed.

    One thing I will say about 8-10 requests a day for Tier 1: This doesn't really give them much room to grow and dig in deeper on an issue. I know it's more efficient for them to escalate but this is kinda an inherent problem with escalation, it keeps Tier 1 folks in Tier 1 forever and they never improve.

    Twitch: Thawmus83
  • wunderbarwunderbar What Have I Done? Registered User regular
    edited May 2022
    Thawmus wrote: »
    wunderbar wrote: »
    Between not working for a year, working in a slightly different kind of job for another year, it's been about two and a half years since I have spent any time as doing primarily tier 1 work for more than say, half a day at a time. been lucky to have had the last IT job were there were 2 of us that would take tickets when the tier 1 guy went on holidays, so it was split.

    I get a slightly better appreciation for the tier 1 people and marvel at a) how I ever did it as a job, and b) how Tier 1 people get anything done. It's not like we're overwhelmed with tickets here, I'd say the average is about 8-10 requests per day, absolutely something one person can handle on a day to day basis. couple of them take 5 minutes, I've had a couple I've needed to spend a couple hours on. but I haven't fell behind on it, which is good.

    but the constant interruptions are awful. I have projects/tasks that are part of my actual job that I to this point have barely been able to make any tangible progress on this week because I don't go more than about 45 minutes without some kind of interruption.

    Yeah I just yell at people if I get interrupted a bunch. I know it's unprofessional but I have ADD and I can't keep having my thought process derailed.

    One thing I will say about 8-10 requests a day for Tier 1: This doesn't really give them much room to grow and dig in deeper on an issue. I know it's more efficient for them to escalate but this is kinda an inherent problem with escalation, it keeps Tier 1 folks in Tier 1 forever and they never improve.

    I don't disagree with that. And it is something I'm going to bring up/pay more attention to. There's a couple things happenin here that we're learning as we go. the "infrastructure" team here used to be the IT manager that was basically a sysadmin, a sysadmin that I'm discovering was doing probably 50% sysadmin and 50% tier 1 work, and a tier 1 guy who was also doing a few other bits here or there.

    In the last 6 months both the manager and sysadmin both moved on. sysadmin retired, manager to go work for a bigger company. the new direction is they want the IT manager to be more of a manager and doing less day to day work, which is a good thing in general because this place lacked a proper IT strategy/vision, but that means the sysadmin, me, has less time to deal with tier 1 work, which has shifted more of it to the tier 1 dude. So while he can manage he's not getting extra time to do skills development.

    After this week I think we need to find a better/happier medium here for that tier 1 guy, but I'm also up to my eyeballs in projects because most of the on prem infrastructure here is on Server 2012, which only has 7 months of support left, and the hybrid exchange environment is an Exchange 2010 on prem box that I need to fire into the sun as soon as possible. EDIT: oh and most of the security/firewalls need to be replaced. And we're opening a new warehouse location that I need to get equipment set up for, and we need to replace our corproate AV since it's through a 3rd party management company we're cutting ties with. and about 5 other things I"m forgetting.

    The previous regime here was really treading water and now I'm coming in to clean it up and... its really "fun"

    wunderbar on
    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • BlazeFireBlazeFire Registered User regular
    I have a Windows 10 machine that currently does not get to see the internet or a WSUS server. It keeps trying to install a speech recognition patch but returns a 0x8024402C error. This prompts a pop-up notification saying the optional feature install failed. I am unable to figure out how to get Windows to stop trying to install that optional feature. I can't find an appropriate solution via Google. Putting it on the internet is not an option.

    Any tips or suggestions?

  • MvrckMvrck Dwarven MountainhomeRegistered User regular
    So my virtual web servers have expanded over the past 3 years from a single one, with a dev server to tooling around into 5 production, 3 dev and a staging testbed. Due to reasons (IE, people being super stingy with budget stuff), those dev servers are also being used as content delivery stores for some internal work, so outside of my actual development work for the applications their production equivalents host, tracking their performance is actually kinda needed now.

    What I'm looking for is recommendations for CPU/Memory/Network monitoring for Debian 11 servers that would ideally be able to be viewed through a web interface. I've seen some options like Cacti and similar, but I was just curious if anyone had something they swore by (or swore at) that was free.

  • That_GuyThat_Guy I don't wanna be that guy Registered User regular
    I think PRTG will do all that. It's got a free version with, like, 10 sensors.

  • DrovekDrovek Registered User regular
    There's also Prometheus + Grafana, but I personally don't have much experience setting those up.

    steam_sig.png( < . . .
  • electricitylikesmeelectricitylikesme Registered User regular
    *sigh* I'm on a call hearing another team tell me they've designed a solution where they put no effort into reducing the transitionary burden. The actual answer here seems to be that no one cared.

  • DrovekDrovek Registered User regular
    Company going through layoffs these couple of days.

    Really hilarious* to have the sword of Damocles hanging above while seeing people get "the notification" without rhyme nor reason.

    * Situation might not be actually hilarious. Consult your Doctor if problems persist.

    steam_sig.png( < . . .
  • MyiagrosMyiagros Registered User regular
    Looks like it's time to learn how to migrate an entire network to Azure, should be fun.

    iRevert wrote: »
    Because if you're going to attempt to squeeze that big black monster into your slot you will need to be able to take at least 12 inches or else you're going to have a bad time...
    Steam: MyiagrosX27
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    Pretty cool to have our Fortune 250 parent company mess up something on an integration with out server and then I get to diagnose the problem for them.

    Long story short:
    It’s not DNS
    There’s no way it’s DNS
    It was DNS

  • wunderbarwunderbar What Have I Done? Registered User regular
    it's literally always DNS

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    edited May 2022
    Usually with this company it’s the two firewalls between us, but I checked our side as soon as I heard there was a problem. Typically the rules are the same, so then I actually looked at what their screenshots showed. Turns out they set up an entirely new DNS infrastructure a couple weeks ago and it doesn’t conditionally forward to ours like it should. Whoopsie!

    RandomHajile on
  • DrovekDrovek Registered User regular
    Drovek wrote: »
    Company going through layoffs these couple of days.

    Really hilarious* to have the sword of Damocles hanging above while seeing people get "the notification" without rhyme nor reason.

    * Situation might not be actually hilarious. Consult your Doctor if problems persist.

    Well, didn't get the axe but my Lead did as well as a friend in another team.

    Good times ahead. /s

    steam_sig.png( < . . .
  • FeldornFeldorn Mediocre Registered User regular
    Usually with this company it’s the two firewalls between us, but I checked our side as soon as I heard there was a problem. Typically the rules are the same, so then I actually looked at what their screenshots showed. Turns out they set up an entirely new DNS infrastructure a couple weeks ago and it doesn’t conditionally forward to ours like it should. Whoopsie!

    Recently promoted a couple new DCs that are also DNS servers and found that things don’t really migrate and you have to go into each domain and change the SOA and stuff.

    Nothing broke but certainly didn’t work how I expected in a Windows system.

  • RandomHajileRandomHajile Not actually a Snatcher The New KremlinRegistered User regular
    Feldorn wrote: »
    Usually with this company it’s the two firewalls between us, but I checked our side as soon as I heard there was a problem. Typically the rules are the same, so then I actually looked at what their screenshots showed. Turns out they set up an entirely new DNS infrastructure a couple weeks ago and it doesn’t conditionally forward to ours like it should. Whoopsie!

    Recently promoted a couple new DCs that are also DNS servers and found that things don’t really migrate and you have to go into each domain and change the SOA and stuff.

    Nothing broke but certainly didn’t work how I expected in a Windows system.
    It sometimes works automatically, but I generally go back in and check/fix for all zones a bit later.

  • Dizzy DDizzy D NetherlandsRegistered User regular
    It depends on how things were configured, usually you want to mark the things you create in DNS to "Store in Active Directory" so the config is replicated to new DNS servers, but if that is not marked, then you need to do a lot manually.

    Steam/Origin: davydizzy
  • FeldornFeldorn Mediocre Registered User regular
    Dizzy D wrote: »
    It depends on how things were configured, usually you want to mark the things you create in DNS to "Store in Active Directory" so the config is replicated to new DNS servers, but if that is not marked, then you need to do a lot manually.

    Probably that. The AD was a lift and shift from Novell almost 20 years ago and it stayed in the same structure until I finished a redesign in 2014/15.

    Hell, scavenging wasn’t even configured when I started, was a right mess.

  • LD50LD50 Registered User regular
    Feldorn wrote: »
    Dizzy D wrote: »
    It depends on how things were configured, usually you want to mark the things you create in DNS to "Store in Active Directory" so the config is replicated to new DNS servers, but if that is not marked, then you need to do a lot manually.

    Probably that. The AD was a lift and shift from Novell almost 20 years ago and it stayed in the same structure until I finished a redesign in 2014/15.

    Hell, scavenging wasn’t even configured when I started, was a right mess.

    When I first started here, I was like "man, this AD is a bit of a dumpster" and was hoping there would be a big cleanup project to get it into ship shape. Then we started a project to build a new AD to migrate to to get off the old one and I was like "yeah, starting fresh sounds like a good idea". Then I saw what the 'new' AD looks like and I secretly hope that project never finishes.

  • wunderbarwunderbar What Have I Done? Registered User regular
    active directory is always the worst. I've never encountered one that wasn't set up in the 90's so there's just layers and layers of cruft and re-designing/building a new one from scratch is always going to be too much work so you just live with it.

    at my current place the domain isn't even a .local domain. For some reason whoever set this AD up, like I said probably in the 90's, used [domain].ads as the AD domain here.

    6 months in and it still gets me every so often.

    XBL: thewunderbar PSN: thewunderbar NNID: thewunderbar Steam: wunderbar87 Twitter: wunderbar
Sign In or Register to comment.