Walked in early this morning, all set to sit and slowly go over the morning monitoring events while sipping tea. What's that noise I hear before even entering my office? Is that... are those the server fans? I open the door and get blasted with heat, only to find that the HVAC system shut down at some point. A few of the LED panels on the front of the servers are blaring temperature warnings. System board temperatures are over 42C.
...god fucking damnit
I have a picture somewhere -- which I cannot seem to find -- of a thermometer/barometer that was in a server room at a school in which I was setting up some cameras. It was something like 95 degrees (F) and 80% relative humidity.
I'm not a Cisco guy, but I wrote down the part numbers of the switches and stuff they had in that closet and checked out the costs. They had something like $25,000 worth of equipment in what was basically a sauna. I reported it to my contact at the school district (a member of their IT team) and he told me he knew about it and had been trying to get the AC unit fixed for months.
My first IT job was with a small company that bought a building much too large for its needs because they got a good deal. The building's infrastructure was amazing and included battery backups powerful to run everything for several hours and a diesel generator that would take over when the batteries ran down. There were banks of these batteries and we only used one. Even that one put out a ton of heat so the electrical room had its own rooftop unit.
You can probably tell where this is going.
I came in one morning and there were alarms blaring from that end of the building. The rooftop unit had dies and the room was cooking the batteries. Nothing had gone down but the machines in the room were certainly not happy. I propped open the door and set up several fans to try and cool it off, then called someone to come out and take a look at the one battery that was running.
About an hour later a tiny giant of a man shows up, probably around five feet tall with forearms like thighs. He got right to work, said everything looked okay but he need to disassemble something underneath the running backup to be sure. Problem: his arms were too thick and too short to get to it. He looked at all 5' 11", 145 pounds of me and asked for a little help.
Being young and naive I said yes.
I got down on the floor and looked underneath the battery while he shined his flashlight on what needed to be removed. The battery was still running, giving off a very unnerving hum. Then the tech said in a very matter of fact way:
KakodaimonosCode fondlerHelping the 1% get richerRegistered Userregular
Yeah that's happened a couple of times to me. It's even more fun when the useless fuckers from facilities who didn't fix the AC in the first place keep coming by and turning off the fans and closing the doors because "Too much hot air in the hall causes problems with the building HVAC."
So we started shutting down servers. Starting with the exchange server that all the senior executives used and my boss went to the CEO to explain why no senior staff would be able to read email that day.
Oddly enough, the AC got fixed later that afternoon.
42C is nothing. When you walk into your server room and the ambient temperature is 37C and the servers are at 65C, then you can complain. :rotate:
I feel for you though. I had an AC system die over a night, and it took two days to fix. That was a very long two days.
That happened to us last year all while we're in the middle of office reconstructions and we (IT) are running around moving machines all over the place and working on power management. The HVAC company had to bring in one of those giant movable HVAC units from Home Depot that I had to monitor the water levels on for 4 days (including over the weekend). Any time something like this happens, I fear that it's the same road I'm about to travel down again and it makes me sad.
While I agree that being insensitive is an issue, so is being oversensitive.
Walked in early this morning, all set to sit and slowly go over the morning monitoring events while sipping tea. What's that noise I hear before even entering my office? Is that... are those the server fans? I open the door and get blasted with heat, only to find that the HVAC system shut down at some point. A few of the LED panels on the front of the servers are blaring temperature warnings. System board temperatures are over 42C.
...god fucking damnit
I feel your pain, we used to have that issue. We don't have a dedicated HVAC system for our server room so we resort to closing the vents in the fall/winter and opening them in the summer when the building AC is running while using two 12k BTU portable AC units to provide cooling year round. Every now and again in the summer months one of the AC units will trip itself off and the temps creep up pretty quick. Ended up installing one of the management cards in one of our APC UPS units with the temperature probe and use Nagios to send me alerts if it gets too high so I don't find myself coming in some Monday morning with a weekend worth of heat buildup.
And why not just add a cherry to this whole mess? SharePoint went off the god damn grid. I couldn't figure out what this update did when it failed in the middle and caused a cluster fuck. Turns out the fucking application pools in IIS totally shut off... every fucking one of them. That's all it was. How stupid is that? *slams head on desk* Fuck this morning.
While I agree that being insensitive is an issue, so is being oversensitive.
My first IT job was with a small company that bought a building much too large for its needs because they got a good deal. The building's infrastructure was amazing and included battery backups powerful to run everything for several hours and a diesel generator that would take over when the batteries ran down. There were banks of these batteries and we only used one. Even that one put out a ton of heat so the electrical room had its own rooftop unit.
You can probably tell where this is going.
I came in one morning and there were alarms blaring from that end of the building. The rooftop unit had dies and the room was cooking the batteries. Nothing had gone down but the machines in the room were certainly not happy. I propped open the door and set up several fans to try and cool it off, then called someone to come out and take a look at the one battery that was running.
About an hour later a tiny giant of a man shows up, probably around five feet tall with forearms like thighs. He got right to work, said everything looked okay but he need to disassemble something underneath the running backup to be sure. Problem: his arms were too thick and too short to get to it. He looked at all 5' 11", 145 pounds of me and asked for a little help.
Being young and naive I said yes.
I got down on the floor and looked underneath the battery while he shined his flashlight on what needed to be removed. The battery was still running, giving off a very unnerving hum. Then the tech said in a very matter of fact way:
Why do users think that putting the paper that their print job is asking for into a paper tray that is not set to use that type of paper is a good solution? Yeah, your job may get printed, but now you've gone and fucked up everyone else's jobs, and we all know who they are going to yell at when that happens... no, it won't be you; it will be me because I obviously don't know what the fuck I'm doing*, so thanks for causing me more stress than I'm already dealing with this morning.
*I had the unfortunate task years ago of taking a former employee's emails and sorting through what was work-related and what wasn't; she was supposed to before she left, but I guess that was her last "fuck you" to the agency; don't ask why it fell on me, but I've never been happy about it. I found numerous emails between her and other staff where all they did was bitch about how I don't even know how to keep a printer running, so how could they expect me to fix anything else on their computers.
Le_Goat on
While I agree that being insensitive is an issue, so is being oversensitive.
I find highlighting their stupidity works wonders in drawing attention to the real problem.
"Why did you think putting paper in the manual tray would only work for your print job and not every other print job that came before yours?", Bowen would proclaim loudly, "Do you think you're special or that the printer knows the difference somehow?"
not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
I find highlighting their stupidity works wonders in drawing attention to the real problem.
"Why did you think putting paper in the manual tray would only work for your print job and not every other print job that came before yours?", Bowen would proclaim loudly, "Do you think you're special or that the printer knows the difference somehow?"
The problem I have found with that is then users feel like they've been publicly punished. The next time they do something, they run away and don't tell us about it, which makes things even worse. When you hear "it's been like this for... oh... I think 2 maybe 3 months," it's just mindboggling.
While I agree that being insensitive is an issue, so is being oversensitive.
I find highlighting their stupidity works wonders in drawing attention to the real problem.
"Why did you think putting paper in the manual tray would only work for your print job and not every other print job that came before yours?", Bowen would proclaim loudly, "Do you think you're special or that the printer knows the difference somehow?"
The problem I have found with that is then users feel like they've been publicly punished. The next time they do something, they run away and don't tell us about it, which makes things even worse. When you hear "it's been like this for... oh... I think 2 maybe 3 months," it's just mindboggling.
Hey 2 or 3 months I'm not getting bugged or blamed!
Can even cover my ass "no I didn't know about the issue, no one submitted a ticket, here I can prove it."
not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
Why do users call me asking why they received a bounce message, when said bounce message is quite clear on which method of fuckuppery they decided to partake in when they attempted to send the email? It says right there that the user is unknown and you clearly typed it wrong... or perhaps it says you are attempting to FTP over SMTP, which it also says quite clear.
Why do users enter a URL into the fucking search field of their chosen search engine?
I could go on forever, those are just the two most recent user interactions.
A man flying in a hot air balloon suddenly realizes he’s lost. He reduces height and spots a man down below. He lowers the balloon further and shouts to get directions.
"Excuse me, can you tell me where I am?”
The man below says “Yes. You’re in a hot air balloon, hovering 30 feet above this field.”
“You must work in Information Technology,” says the balloonist.
“I do” replies the man. “How did you know?”
“Well,” says the balloonist, “everything you have told me is technically correct, but It’s of no use to anyone.”
The man below replies, “You must work in management.”
“I do,” replies the balloonist, “But how’d you know?”
“Well”, says the man, “you don’t know where you are or where you’re going, but you expect me to be able to help. You’re in the same position you were before we met, but now it’s my fault.”
Why do users call me asking why they received a bounce message, when said bounce message is quite clear on which method of fuckuppery they decided to partake in when they attempted to send the email? It says right there that the user is unknown and you clearly typed it wrong... or perhaps it says you are attempting to FTP over SMTP, which it also says quite clear.
Why do users enter a URL into the fucking search field of their chosen search engine?
I could go on forever, those are just the two most recent user interactions.
To be fair, standard email bounces replies look like reams of goobledygook. Not that the single actual piece of information isn't easily identifiable, but I teach first-year students who just freak the fuck out when they see a stack trace and have to be taught to just read up slowly.
Why do users call me asking why they received a bounce message, when said bounce message is quite clear on which method of fuckuppery they decided to partake in when they attempted to send the email? It says right there that the user is unknown and you clearly typed it wrong... or perhaps it says you are attempting to FTP over SMTP, which it also says quite clear.
Why do users enter a URL into the fucking search field of their chosen search engine?
I could go on forever, those are just the two most recent user interactions.
Why do users call me asking why they received a bounce message, when said bounce message is quite clear on which method of fuckuppery they decided to partake in when they attempted to send the email? It says right there that the user is unknown and you clearly typed it wrong... or perhaps it says you are attempting to FTP over SMTP, which it also says quite clear.
Why do users enter a URL into the fucking search field of their chosen search engine?
I could go on forever, those are just the two most recent user interactions.
Walked in early this morning, all set to sit and slowly go over the morning monitoring events while sipping tea. What's that noise I hear before even entering my office? Is that... are those the server fans? I open the door and get blasted with heat, only to find that the HVAC system shut down at some point. A few of the LED panels on the front of the servers are blaring temperature warnings. System board temperatures are over 42C.
...god fucking damnit
I have a picture somewhere -- which I cannot seem to find -- of a thermometer/barometer that was in a server room at a school in which I was setting up some cameras. It was something like 95 degrees (F) and 80% relative humidity.
I'm not a Cisco guy, but I wrote down the part numbers of the switches and stuff they had in that closet and checked out the costs. They had something like $25,000 worth of equipment in what was basically a sauna. I reported it to my contact at the school district (a member of their IT team) and he told me he knew about it and had been trying to get the AC unit fixed for months.
I must have completely missed this. The guy had been trying for months?! Dear god, man! I'm sitting here freaking out about the weekend coming up, to which I've authorized overnight shipping on the compressor (probably around $200) so that it's here before Friday night. And your guy had been trying for months? That's just... irresponsible
While I agree that being insensitive is an issue, so is being oversensitive.
Management, "It lasted this long, I wonder how long we can let it go and use the money elsewhere."
Most likely it's because the duder isn't really emphasizing how much money they'd lose if the equipment fails in the sauna. If he goes "we need this $200 piece to fix the AC to save $25,000 worth of hardware and $10,000 worth of labor" this would probably change his tune completely.
not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
+2
RandomHajileNot actually a SnatcherThe New KremlinRegistered Userregular
Not to mention potential data loss and whatnot. Sure, you have backups, but what about your tapes/hard drives that are now outside of their recommended operating temperature? Tapes especially, stretching out as they're being wound. The one time we ever had an AC failure in our old tiny server room, we lost a hard drive out of each of probably a quarter of our servers over the following three months or so.
Walked in early this morning, all set to sit and slowly go over the morning monitoring events while sipping tea. What's that noise I hear before even entering my office? Is that... are those the server fans? I open the door and get blasted with heat, only to find that the HVAC system shut down at some point. A few of the LED panels on the front of the servers are blaring temperature warnings. System board temperatures are over 42C.
...god fucking damnit
I have a picture somewhere -- which I cannot seem to find -- of a thermometer/barometer that was in a server room at a school in which I was setting up some cameras. It was something like 95 degrees (F) and 80% relative humidity.
I'm not a Cisco guy, but I wrote down the part numbers of the switches and stuff they had in that closet and checked out the costs. They had something like $25,000 worth of equipment in what was basically a sauna. I reported it to my contact at the school district (a member of their IT team) and he told me he knew about it and had been trying to get the AC unit fixed for months.
I must have completely missed this. The guy had been trying for months?! Dear god, man! I'm sitting here freaking out about the weekend coming up, to which I've authorized overnight shipping on the compressor (probably around $200) so that it's here before Friday night. And your guy had been trying for months? That's just... irresponsible
I worked in every school in the district for this project and the vast majority of them were warm, but nothing like the one mentioned above.
There were only three schools that were properly cooled. Two were the newest two schools in the district, and the third, oddly enough, was probably the oldest school -- or at least in the oldest building.
I've dealt with many different types of people in my line of work, but the arrogance of school employees is staggering.
I remember a story that one of my school district contacts told me about how half of the schools went a day and a half without phones or internet because someone in IT didn't want to get up and flip a breaker to restore power.
The local university bought land around a building that the school district owned. Inside the building was a data closet that housed the main fiber connection point for the north side schools in town. The university ended up buying the building as well, under the conditions that they allowed access to the school district to the building so maintenance could be performed if necessary. It was also stipulated that the university wouldn't perform any work on the building without contacting the school district first.
Well, the university decided to go ahead and do some outside work and cut power to the building -- without telling anyone.
After school upon school contacted the IT department to tell them they had no phones or internet, IT contacted the university who informed them that they cut power to the building, but power had since been restored.
I don't know all the specifics but I believe a circuit had tripped and that IT guy didn't think he should have to go over to flip it back to restore power to the closet. After all it was the university who caused the issue in the first place, and they should have gone and flipped the breaker back.
A day and a half with no internet or phones for half the school district because some lazy asshole had a superiority complex.
Posts
I have a picture somewhere -- which I cannot seem to find -- of a thermometer/barometer that was in a server room at a school in which I was setting up some cameras. It was something like 95 degrees (F) and 80% relative humidity.
I'm not a Cisco guy, but I wrote down the part numbers of the switches and stuff they had in that closet and checked out the costs. They had something like $25,000 worth of equipment in what was basically a sauna. I reported it to my contact at the school district (a member of their IT team) and he told me he knew about it and had been trying to get the AC unit fixed for months.
Turned out to some sort of bad patch dependency problem where a patch relied on an update they wasn't already installed.
Thankfully someone had already run into this issue before.
http://stevesps.blogspot.com/2014/09/method-upgrade-in-type.html
Applied the update in that blog post and it started working again.
Nintendo ID: Incindium
PSN: IncindiumX
I feel for you though. I had an AC system die over a night, and it took two days to fix. That was a very long two days.
You can probably tell where this is going.
I came in one morning and there were alarms blaring from that end of the building. The rooftop unit had dies and the room was cooking the batteries. Nothing had gone down but the machines in the room were certainly not happy. I propped open the door and set up several fans to try and cool it off, then called someone to come out and take a look at the one battery that was running.
About an hour later a tiny giant of a man shows up, probably around five feet tall with forearms like thighs. He got right to work, said everything looked okay but he need to disassemble something underneath the running backup to be sure. Problem: his arms were too thick and too short to get to it. He looked at all 5' 11", 145 pounds of me and asked for a little help.
Being young and naive I said yes.
I got down on the floor and looked underneath the battery while he shined his flashlight on what needed to be removed. The battery was still running, giving off a very unnerving hum. Then the tech said in a very matter of fact way:
'Don't touch this' points light, 'or this,' points light again.
'You'll die.'
So we started shutting down servers. Starting with the exchange server that all the senior executives used and my boss went to the CEO to explain why no senior staff would be able to read email that day.
Oddly enough, the AC got fixed later that afternoon.
I feel your pain, we used to have that issue. We don't have a dedicated HVAC system for our server room so we resort to closing the vents in the fall/winter and opening them in the summer when the building AC is running while using two 12k BTU portable AC units to provide cooling year round. Every now and again in the summer months one of the AC units will trip itself off and the temps creep up pretty quick. Ended up installing one of the management cards in one of our APC UPS units with the temperature probe and use Nagios to send me alerts if it gets too high so I don't find myself coming in some Monday morning with a weekend worth of heat buildup.
Shamelessly stolen from a H/A thread.
*I had the unfortunate task years ago of taking a former employee's emails and sorting through what was work-related and what wasn't; she was supposed to before she left, but I guess that was her last "fuck you" to the agency; don't ask why it fell on me, but I've never been happy about it. I found numerous emails between her and other staff where all they did was bitch about how I don't even know how to keep a printer running, so how could they expect me to fix anything else on their computers.
"Why did you think putting paper in the manual tray would only work for your print job and not every other print job that came before yours?", Bowen would proclaim loudly, "Do you think you're special or that the printer knows the difference somehow?"
It is 2015. They wouldn't be my vendor anymore
What is it, 2003?
Its running on 2003, yes.
Compile it with a 64 bit compiler.
... that might be all they even need to do.
Maybe they need to convert things like "int" to "int32_t" or something.
Well... not exactly what I meant, but that was pretty funny.
Its crippling.
Hey 2 or 3 months I'm not getting bugged or blamed!
Can even cover my ass "no I didn't know about the issue, no one submitted a ticket, here I can prove it."
Why do users call me asking why they received a bounce message, when said bounce message is quite clear on which method of fuckuppery they decided to partake in when they attempted to send the email? It says right there that the user is unknown and you clearly typed it wrong... or perhaps it says you are attempting to FTP over SMTP, which it also says quite clear.
Why do users enter a URL into the fucking search field of their chosen search engine?
I could go on forever, those are just the two most recent user interactions.
User loads up browser
User puts url in the search box
I facepalm.
XBL:Phenyhelm - 3DS:Phenyhelm
Some of our public facing websites.
"Excuse me, can you tell me where I am?”
The man below says “Yes. You’re in a hot air balloon, hovering 30 feet above this field.”
“You must work in Information Technology,” says the balloonist.
“I do” replies the man. “How did you know?”
“Well,” says the balloonist, “everything you have told me is technically correct, but It’s of no use to anyone.”
The man below replies, “You must work in management.”
“I do,” replies the balloonist, “But how’d you know?”
“Well”, says the man, “you don’t know where you are or where you’re going, but you expect me to be able to help. You’re in the same position you were before we met, but now it’s my fault.”
To be fair, standard email bounces replies look like reams of goobledygook. Not that the single actual piece of information isn't easily identifiable, but I teach first-year students who just freak the fuck out when they see a stack trace and have to be taught to just read up slowly.
People use browsers that don't have a unified search box and address bar?
I have seen a person type "http://www.google.com" into a google homepage search bar...
*Scoff*
Well I made google the homepage in chrome. My Boss will hit home to go to google so she can put a URL into the search field.
I have tried and have given up breaking her of this habit.
XBL:Phenyhelm - 3DS:Phenyhelm
Most likely it's because the duder isn't really emphasizing how much money they'd lose if the equipment fails in the sauna. If he goes "we need this $200 piece to fix the AC to save $25,000 worth of hardware and $10,000 worth of labor" this would probably change his tune completely.
This is a clickable link to my Steam Profile.
I worked in every school in the district for this project and the vast majority of them were warm, but nothing like the one mentioned above.
There were only three schools that were properly cooled. Two were the newest two schools in the district, and the third, oddly enough, was probably the oldest school -- or at least in the oldest building.
I've dealt with many different types of people in my line of work, but the arrogance of school employees is staggering.
I remember a story that one of my school district contacts told me about how half of the schools went a day and a half without phones or internet because someone in IT didn't want to get up and flip a breaker to restore power.
The local university bought land around a building that the school district owned. Inside the building was a data closet that housed the main fiber connection point for the north side schools in town. The university ended up buying the building as well, under the conditions that they allowed access to the school district to the building so maintenance could be performed if necessary. It was also stipulated that the university wouldn't perform any work on the building without contacting the school district first.
Well, the university decided to go ahead and do some outside work and cut power to the building -- without telling anyone.
After school upon school contacted the IT department to tell them they had no phones or internet, IT contacted the university who informed them that they cut power to the building, but power had since been restored.
I don't know all the specifics but I believe a circuit had tripped and that IT guy didn't think he should have to go over to flip it back to restore power to the closet. After all it was the university who caused the issue in the first place, and they should have gone and flipped the breaker back.
A day and a half with no internet or phones for half the school district because some lazy asshole had a superiority complex.