So uh, there is the very real possibility that I am going to be inheriting caring for the network at my job, as currently well, no one does and it’s basically on fire.
We are looking at a small office of about 30 people. Outside of finishing getting my Network+ and training in our specific hardware (Ubiquiti). What are some good places to look into the fundamentals of getting such a network into a good place?
Like, right now it’s a disaster, a soho modem plugged into our router. No vlan segmentation between our wireless and wired network, etc. I know enough to know that things are odd but not necessarily how important certain things are like cat5e vs cat6 cables or packetshaping to give VOIP priority (we are all software VOIP phone based).
Any suggestions are greatly appreciated.
If you've already got cat5, it should be fine. There's no real reason to rip it out if you aren't having issues. If you have to do new runs, cat6 is basically the same price so you might as well run that.
So on saturday evening 6pm local, I was alerted to it at 8pm, the SQL server that hosts our public website lost it's mind and became unresponsive, taking down our public websites.
Once I got alerted it took me an hour to fix. Had to reboot the server and then a bunch of other shit broke that was fascinating in a "wait how on earth was this working before?" kind of way that gook me about 45 minutes to resolve.
Monday morning I come in and look at the logs to see if I can find out why it lost it's mind.
I'm fairly confident that it died on a failed Windows Defender update. No joke. that's the last thing in the event viewer before I rebooted it.
So on saturday evening 6pm local, I was alerted to it at 8pm, the SQL server that hosts our public website lost it's mind and became unresponsive, taking down our public websites.
Once I got alerted it took me an hour to fix. Had to reboot the server and then a bunch of other shit broke that was fascinating in a "wait how on earth was this working before?" kind of way that gook me about 45 minutes to resolve.
Monday morning I come in and look at the logs to see if I can find out why it lost it's mind.
I'm fairly confident that it died on a failed Windows Defender update. No joke. that's the last thing in the event viewer before I rebooted it.
My suspicion is that the Windows Update told the VSS writers to take a break and then screwed up, and SQL is involved with VSS on the server because that’s how it gets into a backup-consistent state.
So uh, there is the very real possibility that I am going to be inheriting caring for the network at my job, as currently well, no one does and it’s basically on fire.
We are looking at a small office of about 30 people. Outside of finishing getting my Network+ and training in our specific hardware (Ubiquiti). What are some good places to look into the fundamentals of getting such a network into a good place?
Like, right now it’s a disaster, a soho modem plugged into our router. No vlan segmentation between our wireless and wired network, etc. I know enough to know that things are odd but not necessarily how important certain things are like cat5e vs cat6 cables or packetshaping to give VOIP priority (we are all software VOIP phone based).
I would start just by gathering info about the network. Put together:
- An inventory of servers, virtual machines, network devices, and printers. List static/reserved IP addresses, make and model numbers, and serial numbers. If you're feeling ambitious, also track down the warranty status and put in the end of warranty dates.
- A network diagram (or multiple diagrams for a very complex network)
- A list of WAN IP addresses and what servers/services are listening on them.
- a list of software licenses and who they're assigned to.
- important IT vendors and contractors. If you need support for your main business software, do you know who to call? How about your phone system?
- A list of open issues, recurrent problems, and other pain points. Talk to people in the business and find out what's troubling them about IT. Then triage it by severity and the number of people affected.
Don't worry about the gaps in your general IT knowledge yet. You'll learn; we all learn as we go along.
But general IT knowledge can be googled. The idiosyncrasies of your company cannot.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
I opened this thread solely because the title made me think "surely that cannot be a reference to the habit my end-users have of opening slack conversations with 'Hello' and then taking five minutes to ask the actual question, while I seethe at the 'end user is typing' message"?
Vendor, to me: "we need to whitelist your public IP address"
Me: "can you whitelist a range? We load balance across multiple public IPs"
Vendor's network technician: "sure!"
Me: "34.98.75.225 - 34.98.75.238"
Vendor's network technician: "oh, our firewall doesn't support that format, can you give me the subnet mask?"
I think there's something wrong with me, I just immediately knew that was a /28.
That'd give them 2 extra wouldn't it?
SECURITY RISK
They're listing the usable addresses, since two are "subnet" and "broadcast" by convention, and when using CIDR you'd be talking about it as the /28.
I was mostly doing that for the joke.
Also CIDR is awful, change my mind.
CIDR seems like the least worst solution
life's a game that you're bound to lose / like using a hammer to pound in screws
fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
that's right we're on a fucked up cruise / God is dead but at least we have booze
bad things happen, no one knows why / the sun burns out and everyone dies
Thanks for the responses so far everyone, especially that todo list @Feral that’s a very solid starting point.
Basically we are the customer care / billing / sales / account management / basic troubleshooting call center for a small dental software start up company. We have another office that does dev stuff but the rest is handled with us.
Our day to day is VOIP calls to clients, remotely connecting to their workstations to train or troubleshoot problems, and then running things in VMs to test on our end. (Our software is windows only but all of our employees are given Macs, this type of odd decision making permeates through out).
I’m probably the tech savviest person on the office, especially when it comes to networks. For the most part the sever rack has just quietly lived in a closet but when things go wrong and we have outages we call in like a family friend of the founders who comes in and fixed things. But frankly this just won’t cut it as we scale, and the individual in question is tired of getting sucked into our situation.
Basically our growth has outstripped our set up and we need to up our professionalism up. We have a laundry list of sins like no failover connection, unlocked sever closet, every user account having the same password and admin rights on their Macs, no one who regularly monitors the network or anyone on site with priviledged access to the sever, etc.
Currently I am in the process of convincing them that we need a person on onsite to tighten things up and be around when things break. We continue to grow (two new hires today, around 30 total employees in this office) and just winging it isn’t going to cut it any longer.
Thanks for the responses so far everyone, especially that todo list @Feral that’s a very solid starting point.
Basically we are the customer care / billing / sales / account management / basic troubleshooting call center for a small dental software start up company. We have another office that does dev stuff but the rest is handled with us.
Our day to day is VOIP calls to clients, remotely connecting to their workstations to train or troubleshoot problems, and then running things in VMs to test on our end. (Our software is windows only but all of our employees are given Macs, this type of odd decision making permeates through out).
I’m probably the tech savviest person on the office, especially when it comes to networks. For the most part the sever rack has just quietly lived in a closet but when things go wrong and we have outages we call in like a family friend of the founders who comes in and fixed things. But frankly this just won’t cut it as we scale, and the individual in question is tired of getting sucked into our situation.
Basically our growth has outstripped our set up and we need to up our professionalism up. We have a laundry list of sins like no failover connection, unlocked sever closet, every user account having the same password and admin rights on their Macs, no one who regularly monitors the network or anyone on site with priviledged access to the sever, etc.
Currently I am in the process of convincing them that we need a person on onsite to tighten things up and be around when things break. We continue to grow (two new hires today, around 30 total employees in this office) and just winging it isn’t going to cut it any longer.
1) Roughly how many users are on your network at one time?
2) Where is your VOIP server? Is it on-premises at the office, on-premises at a different location, or hosted by a vendor?
BTW, you've described a lot of things that you can be doing better but you haven't mentioned any problems with the user experience... yeah, an unlocked server closet is a bad practice and that needs to be fixed. But you can run for a long time with an unlocked server closet without users noticing anything wrong.
So, that leads me to question 3:
3) What would you say is the IT problem with the worst impact on your users?
Feral on
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
Thanks for the responses so far everyone, especially that todo list Feral that’s a very solid starting point.
Basically we are the customer care / billing / sales / account management / basic troubleshooting call center for a small dental software start up company. We have another office that does dev stuff but the rest is handled with us.
Our day to day is VOIP calls to clients, remotely connecting to their workstations to train or troubleshoot problems, and then running things in VMs to test on our end. (Our software is windows only but all of our employees are given Macs, this type of odd decision making permeates through out).
I’m probably the tech savviest person on the office, especially when it comes to networks. For the most part the sever rack has just quietly lived in a closet but when things go wrong and we have outages we call in like a family friend of the founders who comes in and fixed things. But frankly this just won’t cut it as we scale, and the individual in question is tired of getting sucked into our situation.
Basically our growth has outstripped our set up and we need to up our professionalism up. We have a laundry list of sins like no failover connection, unlocked sever closet, every user account having the same password and admin rights on their Macs, no one who regularly monitors the network or anyone on site with priviledged access to the sever, etc.
Currently I am in the process of convincing them that we need a person on onsite to tighten things up and be around when things break. We continue to grow (two new hires today, around 30 total employees in this office) and just winging it isn’t going to cut it any longer.
1) Roughly how many users are on your network at one time?
2) Where is your VOIP server? Is it on-premises at the office, on-premises at a different location, or hosted by a vendor?
BTW, you've described a lot of things that you can be doing better but you haven't mentioned any problems with the user experience... yeah, an unlocked server closet is a bad practice and that needs to be fixed. But you can run for a long time with an unlocked server closet without users noticing anything wrong.
So, that leads me to question 3:
3) What would you say is the IT problem with the worst impact on your users?
1. Pretty much everyone is on the network all day. All of our ticketing, scheduling, calling, etc is done through web services like Zen Desk, Acuity, etc
2. I don’t think we have our own VOIP server. We use Zen Desk talk for our general call tree. This is done via the web browser and headsets. We have Ring Central accounts for direct lines for account managers, this is an app on the Mac.
3. The biggest problem and impact on users currently is our network going down for hours at a time such as last Thursday from 6/7am to around 2pm. Network was in and out during this period, loading webpages was possible but difficult and things like VOIP calls or remote connections were impossible. A simple ping test to google’s DNS would alternate between most pings going through, to alternating pings failing and going through, to twenty+ in a row failing.
Unfortunately I don’t currently have rights or permissions to investigate the problem. The tech friend of the family that came out doesn’t have full admin rights to the sever (someone else does) and repeatedly stated that Ubiquiti hardware is not his speciality, but he attributed the problem to a general “packet storm” but couldn’t drill the issue down further than that.
This isn’t the first time the network has gone out in a similar way but the issue is relatively intermittent, it was preceded by several weeks of uptime before this most recent incident.
1. Pretty much everyone is on the network all day. All of our ticketing, scheduling, calling, etc is done through web services like Zen Desk, Acuity, etc
I was looking for a number. 50 people? 69? 100? 1337?
I don’t think we have our own VOIP server. We use Zen Desk talk for our general call tree. This is done via the web browser and headsets. We have Ring Central accounts for direct lines for account managers, this is an app on the Mac.
Okay, so all cloud-based. No big deal, it's just useful to know for troubleshooting.
3. The biggest problem and impact on users currently is our network going down for hours at a time such as last Thursday from 6/7am to around 2pm. Network was in and out during this period, loading webpages was possible but difficult and things like VOIP calls or remote connections were impossible. A simple ping test to google’s DNS would alternate between most pings going through, to alternating pings failing and going through, to twenty+ in a row failing.
Unfortunately I don’t currently have rights or permissions to investigate the problem.
Well, from this description it isn't clear if it's truly your network going down, or your Internet connection.
Try to use ping and traceroute to isolate the problem. You don't need special permissions to use those tools.
Do all of the following once while your network and Internet connection are working properly.
Traceroute to 8.8.8.8 and save the output to a text file.
Ping a few different internal addresses and see if the pings go through. (ping the IP address of your server, of a network printer, of another local workstation.)
Ping the internal IP address of your router (it should be listed as the default gateway in your workstation's IP address configuration) and see if that goes through.
Then, when the outage happens again, do all of those pings and traceroutes again and look at what's failing. That will help you narrow the problem down.
every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.
1. Pretty much everyone is on the network all day. All of our ticketing, scheduling, calling, etc is done through web services like Zen Desk, Acuity, etc
I was looking for a number. 50 people? 69? 100? 1337?
I don’t think we have our own VOIP server. We use Zen Desk talk for our general call tree. This is done via the web browser and headsets. We have Ring Central accounts for direct lines for account managers, this is an app on the Mac.
Okay, so all cloud-based. No big deal, it's just useful to know for troubleshooting.
3. The biggest problem and impact on users currently is our network going down for hours at a time such as last Thursday from 6/7am to around 2pm. Network was in and out during this period, loading webpages was possible but difficult and things like VOIP calls or remote connections were impossible. A simple ping test to google’s DNS would alternate between most pings going through, to alternating pings failing and going through, to twenty+ in a row failing.
Unfortunately I don’t currently have rights or permissions to investigate the problem.
Well, from this description it isn't clear if it's truly your network going down, or your Internet connection.
Try to use ping and traceroute to isolate the problem. You don't need special permissions to use those tools.
Do all of the following once while your network and Internet connection are working properly.
Traceroute to 8.8.8.8 and save the output to a text file.
Ping a few different internal addresses and see if the pings go through. (ping the IP address of your server, of a network printer, of another local workstation.)
Ping the internal IP address of your router (it should be listed as the default gateway in your workstation's IP address configuration) and see if that goes through.
Then, when the outage happens again, do all of those pings and traceroutes again and look at what's failing. That will help you narrow the problem down.
Oh sorry, size of the office was in the earlier post but it’s about 35 people.
And yeah, it’s hard to know if the issue is network based or ISP based at the moment. I was kind of humming merrily along until this most recent outage where I had a chance to talk with the tech who came in which is what made me realize that we have lots of optimizing to do.
I’ll get the trace route and ping stuff going when I get into work tomorrow.
Which breaks so much of IPv6 and is also such an ISP thing to do. "Why would we give our filthy customers a big block when we can give them a smaller block and charge them more? Standards? pffffffffffffbbbttt"
I'm so sick of troubleshooting wifi dropping signal randomly throughout the day from our two access points
How crazy am I for wanting to buy like 5 ubnt in-wall access points and slapping them in all our exam rooms?
Are your APs broadcasting 2.4 and 5ghz wifi? Is there anything in the office that wouldn't support 5ghz? I ask because I was having the exact same problem with one of my clients a few years ago. There was plenty of wifi coverage in the building but signal was getting cut off in the exam rooms. After months of troubleshooting I ended upgrading the wifi cards in their laptops and switching the main network over to 5ghz only. Not only did it fix the problem, the wifi was noticeably faster. I'm pretty sure some of the equipment they are using in the office put out a ton of 2.4ish ghz interference. Since it's so close to the resonant frequency of water, I'm betting it's something like the autoclave that was killing the wifi.
I'm so sick of troubleshooting wifi dropping signal randomly throughout the day from our two access points
How crazy am I for wanting to buy like 5 ubnt in-wall access points and slapping them in all our exam rooms?
Are your APs broadcasting 2.4 and 5ghz wifi? Is there anything in the office that wouldn't support 5ghz? I ask because I was having the exact same problem with one of my clients a few years ago. There was plenty of wifi coverage in the building but signal was getting cut off in the exam rooms. After months of troubleshooting I ended upgrading the wifi cards in their laptops and switching the main network over to 5ghz only. Not only did it fix the problem, the wifi was noticeably faster. I'm pretty sure some of the equipment they are using in the office put out a ton of 2.4ish ghz interference. Since it's so close to the resonant frequency of water, I'm betting it's something like the autoclave that was killing the wifi.
Or cordless phones or microwaves or cordless monitors or cordless mics or anything that transmits analog video or audio or or or. 2.4 is polluted AF.
That_GuyI don't wanna be that guyRegistered Userregular
It happened guys. I got a formal job offer from that dealership chain for $26/h. I told him I needed a day to think over the offer and give notice before formally accepting. Tomorrow I am going to sit down and talk with my current boss about getting a counter offer and/or giving notice. I would consider staying at my current job if they can meet the new offer but I highly doubt they'd give me a 25% raise like that.
It happened guys. I got a formal job offer from that dealership chain for $26/h. I told him I needed a day to think over the offer and give notice before formally accepting. Tomorrow I am going to sit down and talk with my current boss about getting a counter offer and/or giving notice. I would consider staying at my current job if they can meet the new offer but I highly doubt they'd give me a 25% raise like that.
Posts
If you've already got cat5, it should be fine. There's no real reason to rip it out if you aren't having issues. If you have to do new runs, cat6 is basically the same price so you might as well run that.
Once I got alerted it took me an hour to fix. Had to reboot the server and then a bunch of other shit broke that was fascinating in a "wait how on earth was this working before?" kind of way that gook me about 45 minutes to resolve.
Monday morning I come in and look at the logs to see if I can find out why it lost it's mind.
I'm fairly confident that it died on a failed Windows Defender update. No joke. that's the last thing in the event viewer before I rebooted it.
That'd give them 2 extra wouldn't it?
SECURITY RISK
This is a clickable link to my Steam Profile.
They're listing the usable addresses, since two are "subnet" and "broadcast" by convention, and when using CIDR you'd be talking about it as the /28.
@Inquisitor
I would start just by gathering info about the network. Put together:
- An inventory of servers, virtual machines, network devices, and printers. List static/reserved IP addresses, make and model numbers, and serial numbers. If you're feeling ambitious, also track down the warranty status and put in the end of warranty dates.
- A network diagram (or multiple diagrams for a very complex network)
- A list of WAN IP addresses and what servers/services are listening on them.
- a list of software licenses and who they're assigned to.
- important IT vendors and contractors. If you need support for your main business software, do you know who to call? How about your phone system?
- A list of open issues, recurrent problems, and other pain points. Talk to people in the business and find out what's troubling them about IT. Then triage it by severity and the number of people affected.
Don't worry about the gaps in your general IT knowledge yet. You'll learn; we all learn as we go along.
But general IT knowledge can be googled. The idiosyncrasies of your company cannot.
the "no true scotch man" fallacy.
I was mostly doing that for the joke.
Also CIDR is awful, change my mind.
CIDR seems like the least worst solution
fuck up once and you break your thumb / if you're happy at all then you're god damn dumb
that's right we're on a fucked up cruise / God is dead but at least we have booze
bad things happen, no one knows why / the sun burns out and everyone dies
I assume that after working for a rural WISP long enough, you just start to see the matrix.
the "no true scotch man" fallacy.
I've vaguely completed it but don't have enough access to move vaguely further than this.
Did I mention it's been pretty vague so far?
10% of IT is fixing problems
60% of IT is throttling end users to get them to tell you clearly what the fuck they want
(The remaining 30% is divided up between shitposting and desk whiskey.)
the "no true scotch man" fallacy.
Basically we are the customer care / billing / sales / account management / basic troubleshooting call center for a small dental software start up company. We have another office that does dev stuff but the rest is handled with us.
Our day to day is VOIP calls to clients, remotely connecting to their workstations to train or troubleshoot problems, and then running things in VMs to test on our end. (Our software is windows only but all of our employees are given Macs, this type of odd decision making permeates through out).
I’m probably the tech savviest person on the office, especially when it comes to networks. For the most part the sever rack has just quietly lived in a closet but when things go wrong and we have outages we call in like a family friend of the founders who comes in and fixed things. But frankly this just won’t cut it as we scale, and the individual in question is tired of getting sucked into our situation.
Basically our growth has outstripped our set up and we need to up our professionalism up. We have a laundry list of sins like no failover connection, unlocked sever closet, every user account having the same password and admin rights on their Macs, no one who regularly monitors the network or anyone on site with priviledged access to the sever, etc.
Currently I am in the process of convincing them that we need a person on onsite to tighten things up and be around when things break. We continue to grow (two new hires today, around 30 total employees in this office) and just winging it isn’t going to cut it any longer.
1) Roughly how many users are on your network at one time?
2) Where is your VOIP server? Is it on-premises at the office, on-premises at a different location, or hosted by a vendor?
BTW, you've described a lot of things that you can be doing better but you haven't mentioned any problems with the user experience... yeah, an unlocked server closet is a bad practice and that needs to be fixed. But you can run for a long time with an unlocked server closet without users noticing anything wrong.
So, that leads me to question 3:
3) What would you say is the IT problem with the worst impact on your users?
the "no true scotch man" fallacy.
1. Pretty much everyone is on the network all day. All of our ticketing, scheduling, calling, etc is done through web services like Zen Desk, Acuity, etc
2. I don’t think we have our own VOIP server. We use Zen Desk talk for our general call tree. This is done via the web browser and headsets. We have Ring Central accounts for direct lines for account managers, this is an app on the Mac.
3. The biggest problem and impact on users currently is our network going down for hours at a time such as last Thursday from 6/7am to around 2pm. Network was in and out during this period, loading webpages was possible but difficult and things like VOIP calls or remote connections were impossible. A simple ping test to google’s DNS would alternate between most pings going through, to alternating pings failing and going through, to twenty+ in a row failing.
Unfortunately I don’t currently have rights or permissions to investigate the problem. The tech friend of the family that came out doesn’t have full admin rights to the sever (someone else does) and repeatedly stated that Ubiquiti hardware is not his speciality, but he attributed the problem to a general “packet storm” but couldn’t drill the issue down further than that.
This isn’t the first time the network has gone out in a similar way but the issue is relatively intermittent, it was preceded by several weeks of uptime before this most recent incident.
Don't you just make everything a /64 for private stuff with v6?
I was looking for a number. 50 people? 69? 100? 1337?
Okay, so all cloud-based. No big deal, it's just useful to know for troubleshooting.
Well, from this description it isn't clear if it's truly your network going down, or your Internet connection.
Try to use ping and traceroute to isolate the problem. You don't need special permissions to use those tools.
Do all of the following once while your network and Internet connection are working properly.
Traceroute to 8.8.8.8 and save the output to a text file.
Ping a few different internal addresses and see if the pings go through. (ping the IP address of your server, of a network printer, of another local workstation.)
Ping the internal IP address of your router (it should be listed as the default gateway in your workstation's IP address configuration) and see if that goes through.
Then, when the outage happens again, do all of those pings and traceroutes again and look at what's failing. That will help you narrow the problem down.
the "no true scotch man" fallacy.
Oh sorry, size of the office was in the earlier post but it’s about 35 people.
And yeah, it’s hard to know if the issue is network based or ISP based at the moment. I was kind of humming merrily along until this most recent outage where I had a chance to talk with the tech who came in which is what made me realize that we have lots of optimizing to do.
I’ll get the trace route and ping stuff going when I get into work tomorrow.
You'd think so, the whole point was for ISPs to just give everyone a whole chunk like that for NATing all these IoT devices and what have you.
We only got offered (the equivalent of) a /27 last time we looked into it I think. Seemed to defeat the purpose of going after IPv6.
Which breaks so much of IPv6 and is also such an ISP thing to do. "Why would we give our filthy customers a big block when we can give them a smaller block and charge them more? Standards? pffffffffffffbbbttt"
I get what you're saying, but a minor point of order here: a major goal of IPv6 is to eliminate that kind of NATing.
NAT can still exist in a 100% IPv6 world but it's more of an edge case; it isn't for obfuscating your internal IP scheme at your perimeter.
the "no true scotch man" fallacy.
True
the "no true scotch man" fallacy.
How crazy am I for wanting to buy like 5 ubnt in-wall access points and slapping them in all our exam rooms?
Are your APs broadcasting 2.4 and 5ghz wifi? Is there anything in the office that wouldn't support 5ghz? I ask because I was having the exact same problem with one of my clients a few years ago. There was plenty of wifi coverage in the building but signal was getting cut off in the exam rooms. After months of troubleshooting I ended upgrading the wifi cards in their laptops and switching the main network over to 5ghz only. Not only did it fix the problem, the wifi was noticeably faster. I'm pretty sure some of the equipment they are using in the office put out a ton of 2.4ish ghz interference. Since it's so close to the resonant frequency of water, I'm betting it's something like the autoclave that was killing the wifi.
XBL:Phenyhelm - 3DS:Phenyhelm
Or cordless phones or microwaves or cordless monitors or cordless mics or anything that transmits analog video or audio or or or. 2.4 is polluted AF.
because of 4 microwaves in the lunchroom.
This is still related to the security appliance and not the wireless APs, right?
the "no true scotch man" fallacy.
Do you also get a free car?