The new forums will be named Coin Return (based on the most recent vote)! You can check on the status and timeline of the transition to the new forums here.
The Guiding Principles and New Rules document is now in effect.
Virtualizing servers with VMWare ESX - hardware recommendations, etc
Jimmy King
Registered User regular
Registered User regular
Hey guys, I'm being put in the lead for a project that is pretty new territory for me being the lead tech guy for the company here in the US. I'm a developer - primarily in linux environments, but we've got some windows servers here that need replaced as the hardware is old, out of warranty, and some of it such as the hard disks in our raid stuff can't even be purchased anymore. So, I call on you, sys admins of H&A, to point me in the right direction.
Here's the current hardware. In all honesty, this is actually overkill from a performance standpoint and we're preparing to lighten the load on these even more.
2 x Dual 3.2Ghz Pentium 4 with 2GB ram as web servers which are load balanced
1 x Dual 2.8Ghz Dual Core Xeon with 4Gb ram as web server not in the load balanced cluster and database
1 x Dual 2.8Ghz Dual Core Xeon with 4Gb ram as primary database server
I should also comment on those 2.8Ghz Xeon boxes. I am assuming they are dual core based on only seeing 2 cpus listed when I right click "My Computer" and go to properties, but having graphs for 4 cores when I view the performance tab of task manager. I find that odd, though, since the wikipedia entry for the Xeon doesn't show any dual core xeons being available until long after these servers were built and as far as I know, no cpu upgrades have been done to them - but there was time when these were being handled by people who no longer work here, so an upgrade may have been done then.
So, anyway, we need to virtualize these. The plan is to have 2 physical servers. Server A will run the 3 web servers and probably an additional very low load linux installation. Server B will run the database. The plan is to then also be able to run all of those with the chance of degraded performance off of a single phsyical server for failover. The database on that non-load balanced web server should be going away very soon and is also going to be looked at just running off the main db server.
Now, as I said, these are hugely overkill for what we're currently doing with them, so we don't necessarily need to be able to virtualize even this exact setup. This used to be used for our primary platform but is now just legacy stuff used for a few customers who don't do enough business to put the money into migrating to our new platform or cause us enough work to drop them and some very low load other things that just display splash pages which are in an iframe and the like. My company is also being very cheap. Spending 10s of thousands of dollars on new equipment is out of the question unless 100% absolutely necessary. My boss is really hoping we can do something more in the range of a couple dual core machines with 4gb ram and a ton of disk space in raid 1.
Can we get by with a couple machines as above? What hardware would you recommend for virtualizing this, again, the actual virtualized servers can probably be significantly lower powered than what we have given their current use.
Here's the current hardware. In all honesty, this is actually overkill from a performance standpoint and we're preparing to lighten the load on these even more.
2 x Dual 3.2Ghz Pentium 4 with 2GB ram as web servers which are load balanced
1 x Dual 2.8Ghz Dual Core Xeon with 4Gb ram as web server not in the load balanced cluster and database
1 x Dual 2.8Ghz Dual Core Xeon with 4Gb ram as primary database server
I should also comment on those 2.8Ghz Xeon boxes. I am assuming they are dual core based on only seeing 2 cpus listed when I right click "My Computer" and go to properties, but having graphs for 4 cores when I view the performance tab of task manager. I find that odd, though, since the wikipedia entry for the Xeon doesn't show any dual core xeons being available until long after these servers were built and as far as I know, no cpu upgrades have been done to them - but there was time when these were being handled by people who no longer work here, so an upgrade may have been done then.
So, anyway, we need to virtualize these. The plan is to have 2 physical servers. Server A will run the 3 web servers and probably an additional very low load linux installation. Server B will run the database. The plan is to then also be able to run all of those with the chance of degraded performance off of a single phsyical server for failover. The database on that non-load balanced web server should be going away very soon and is also going to be looked at just running off the main db server.
Now, as I said, these are hugely overkill for what we're currently doing with them, so we don't necessarily need to be able to virtualize even this exact setup. This used to be used for our primary platform but is now just legacy stuff used for a few customers who don't do enough business to put the money into migrating to our new platform or cause us enough work to drop them and some very low load other things that just display splash pages which are in an iframe and the like. My company is also being very cheap. Spending 10s of thousands of dollars on new equipment is out of the question unless 100% absolutely necessary. My boss is really hoping we can do something more in the range of a couple dual core machines with 4gb ram and a ton of disk space in raid 1.
Can we get by with a couple machines as above? What hardware would you recommend for virtualizing this, again, the actual virtualized servers can probably be significantly lower powered than what we have given their current use.
Jimmy King on
0
Posts
What you really need to be asking is why you are virtualizing? What is it you're hoping to gain?
My company virtualized heavily for a couple reasons. First is our power consumption. We spent something in the neighborhood of 1.3 million dollars a month on our datacenter power bill, and we were technically 20% over the rated capacity of our datacenter's power grid, running about 700 physical servers. We have virtualized over half of them since last summer and dropped our power consumption by almost 40%.
Secondly, we wanted to micromanage storage and server resources. When you have a print server or an app server that only really "needs" 15 gig of drive space to do it's business, you're wasting disk when you cant put harddrives in it that are smaller than 30some gig. Most of our ESX hosts have local storage that is doled out in small bits to servers as they need them. Some of our machines have fiber cards that allow them to connect to our bigass SAN storage and they use chunks of that to host the vmdk drive files.
Finally, we try to avoid virtualizing business critical servers due to performance degredation. I'm not a big fan of virtualizing database servers, as slow response times from SQL or whatever you run can put a big hit on a production web environment. If these servers don't have to be snappy and super fast, virtualizing is an option, and you don't have to throw a beast of a machine in as the host.
Most of our ESX hosts are IBM x3650s with ~24 gig of ram and a few dual or quad core processors, depending on when exactly they were purchased.
If you don't have that many servers to virtualize, you want to buy slightly lighter weight servers and spread your VMs across several boxes. If you buy one beast of a server and load up all hosts on it and it craps out, you lose your entire VM environment. Spreading your VMs on multiple boxes also spreads your risk factor. Buying more cheaper servers will do you better than buying fewer expensive ones, unless you have plans to virtualize a lot more boxes in the near future.
In your situation, you're looking at turning 4 physical servers into 2 ESX hosts, and I simply have to ask why. What are you gaining by only halving such a small physical environment? And what's the point of running 1 logical database server on 1 physical ESX host? The only thing you're accomplishing there is taking a performance hit.
A couple other points:
ESX is not the only virtualization tech out there. If you're not trying to fit into an existing ESX infrastructure, check out Microsoft's offering (it _may_ fit your needs, most likely not, but it more or less comes with the cost of the OS). Better yet, check out XenServer. New features coming out this year will bring it more in line with ESX features - though keep in mind that most of those ESX features aren't going to be super useful in such a small virtualized environment. And XenServer is much less expensive than ESX, which is also a good thing.
If you're looking to virtualize a lot of hosts, then you'll definitely want a big chunk of hardware like Erandus suggests. For 3 web servers? A little testing will help you find a sweet spot, but you can probably get away with pretty standard hardware. My company is actually beginning to deploy a bunch of SAN attached HP C-Class blades as ESX hosts. These things are not monsters by any stretch of the imagination.
Also - your Xeons most likely have hyperthreading enabled, which is a tech that Intel came up with that lets one CPU function almost like 2. Thus, each CPU shows up as 2 in task manager.
It sounds to me, however, like you'd benefit more from a little server consolidation. Move the sites on server #3 onto servers #1 & #2, move the DB to server #4, then drop server #3. Hardware and OS licensing costs for that server go away, and there's no need to purchase a virtualization technology.
I actually know one of the engineers on the MS virtualization team, and I have to admit, they did a good job. It's a slick product, but more customized for pure windows applications than anything. In this case, I'd likely avoid.
So, keep in mind I'm coming at this as a developer with only fairly basic sys admin experience focused on the software side - apache, bind, etc. So if I'm way off on something here, fill me in. The virtualization option was brought up by the company we use to manage our network, do hardware upgrades to the servers, etc. as an option because it would offer redundancy as mentioned below which is currently non-existent and could, depending on exactly what we need, be cheaper to purchase and set up and maintain than replacing each current individual server - especially if we were to add redundancy for those servers.
The real gains for us in virtualizing these would be in redundancy and a small amount of expandability. Right now there is no failover or redundancy for any of these servers other than the load balanced ones for a platform that has to be up 24x7. The db server goes and everything is done that runs on these servers. That non-load balanced web server goes and we lose an app that currently pulls in $3k to $5k daily. With the 2 physical servers we could run the db on one full time with it getting most of the resources (it sounds like this may still hurt performance more than I was thinking) and the web servers on their server and we gain the ability to fail over and bring up the web servers in VMs on the db machine should the web machine die or the other way around rather than just being screwed like we are right now.
I'm not sure if we can move the non-load balanced web server stuff over to the same physical servers that are load balanced or not. That was done separately so that the load balancer wouldn't mangle the source IP, which we need for some apps. We could just put another interface on one of the load balanced servers to deal with that, though. There are also a lot of .net environment requirements, which is the scary part. The plan was to use a tool to just copy the phsyical machines to virtual machines because, honestly, there's 0% chance of us getting any of this up and running from scratch. Anyone who had any chance at doing so left the company over a year ago - there are multiple .net versions running with certain apps needing certain versions, libraries used for asp.net sites and iis plugins all over the place that are no documented as to being used, etc.
There is also the potential for expansion that right now would require a new box installed and, at least right now, isn't worth installing a whole new server just for that. For an immediate need there, I'm primarily a Linux guy, and so it would be very helpful to me to have a machine running Linux on that network to have all of the tools I am used to available. I have also had several times where it would be great if I could throw together a web based utility for some of the non-technical people. I could do that in asp.net and run it on the windows servers, but it would take 5x as long as it would for me to get the same thing up and running on a Linux machine. For a near future use, we may also want to move some stuff that needs to run on Linux that is currently running from UK over here to the US for SEO reasons - again, a new server would be needed (well, likely it would run off of the same Linux server I would have available anyway). It doesn't need to be too ridiculously powerful, though, so being able to just virtualize that one one of the two phsyical servers would be great.
It would also allow us to have a lower performance testing environment running on a virtualized server, which we currently don't have, which is doubly bad since everyone who really knows/knew how that old platform works has left the company so on the rare occasion I have to make changes to it, it's really damned scary to be doing on the live servers.
So, I think there are some definite bonuses to going the virtualized route, even for this little deal, although possibly not enough to outweigh the negatives. I just don't know if we can do it and have the performance and options we want at the cheap ass cost my boss is hoping to do it for or what kind of hardware we should be really looking at to get my boss a more realistic price.
The other thing to think about, and it might suck up front because of the extra effort you'll have to go to, but when considering those other web apps that would be easier for you to build on Linux: Who else is around to support them? If you are a Linux guy surrounded by other Linux guys, great. If you're alone on Linux-guy island, keep in mind that every time there's a problem with those apps, it's your problem. From personal experience, it's never fun to be THE guy everyone looks to for a specific platform/application/whatever.
As to the rest, I'm now a Linux guys surrounded by no one technical with a couple other Linux guys across the ocean. See, we had our old platform which is what we're looking at upgrading hardware for. It was all Windows Server 2003 and ASP.Net based. There was a team of 5-7 windows devs and semi-admins (Much like my Linux experience, more or less) here in the US who managed this software, plus our 3rd party support for upgrades, larger network issues, etc. Then the company built a new Linux based platform intended to replace the old one and hired a bunch of us Linux guys. That platform is based in London and I was brought on to do US-centric development, manage integrations done by US customers, etc. and eventually manage a US installation of the platform (which is now not going to happen, which is a bit of a sore spot for me). The .Net platform was to be completely phased out, nearly all of the .Net guys were laid off, and the few that weren't quit not too long after. Only it turns out we've got a bunch of customers on the .Net platform who don't generate enough money using our platform to be worth migrating to the new one, but they are 0 work for us and no cost beyond keeping these servers up, and enough of them combined that it comes out to a decent amount of money (or so the business guys tell me, I'm actually looking into this), so our upper management and sales guys won't tell them to gtfo.
We've also got a few other things running on those servers worth a good bit of money, but at least most of the stuff I'm aware of could very easily be ported to a Linux solution.
So, now we've got these servers that are out of warranty with 30gb scsi hard drives, a few of which have failed over the last year or so, that we can't even purchase anymore that we have to keep up and running if we're not going to man up and dump the customers who won't move. We really can't keep using these servers if we're going to insist on keeping this stuff running, bur honestly, it doesn't generate enough money to spend very much on replacing it at all.
It's a fucking stupid ass, completely retarded situation, worthy of significant dick punching and face stabbing, but it's what I've got to work with at the moment.
What, roughly, would you recommend to virtualize our current hardware with minimal performance impact on the 2 machines as above? All on one machine with no failover? I want to see if we're talking a $10k server or $20k server here, or what. How much room would that give us to add some more similar powered virtual machines? Is there somewhere good I could be reading up on this myself rather than harassing you guys?
Going all new separate machines, we can probably do this for $5k-$7k just directly replacing the servers we currently have and then another $1000-$1500 or so for a low end server for the Linux stuff, one of which is definitely going to be moving from UK to here.