Webservers and networking issues...

SeñorAmor

Here at the office, we run our own webserver. It hosts our website as well as several internal web-based applications that I have created for our employees.

At my suggestion, my bosses purchased a second business-class internet connection on a different ISP than our primary net connection. My reasoning behind it was that if we put our two nameservers on different ISPs, both ISPs would have to experience problems for our website to be down. I checked into it, and it seems it's a pretty common set up.

Now I'm no networking guru, and an issue that I had not expected has arisen. It seems that you can only have one default gateway on a computer. I can have multiple network cards on multiple ISPs, but only one gateway. This is a problem if my gateway is associated with NIC 1, and traffic comes in NIC 2. It appears I cannot have two public IPs from separate ISPs on my webserver and have it function properly.

My solution to this was to set up 2 NAT firewalls and add a 2nd NIC to my webserver. My firewalls would now have the 2 external IPs (and 2 IPs internal to my LAN), and forward all web traffic to a NIC on my webserver (e.g. - firewall 1 would forward to NIC 1, and firewall 2 to NIC 2). Unfortunately, it would seem that I'm going to run into the exact same problem. I can't have multiple gateways. Data coming into NIC 2 would want to go back out NIC 1 (and subsequently firewall 1).

I am at a complete loss as to how to properly achieve this. Others do this (like big datacenters), and I cannot figure out how they do it. Does anyone have any suggestions? I'm tearing my hair out here.

Thanks in advance.

Kyzen

Would one of these help?

http://www.newegg.com/Product/ProductList.asp?Submit=ENE&N=2050400028+1138610150&Subcategory=28&description=&Ntk=&srchInDesc=

SeñorAmor

Kyzen wrote:

Would one of these help?

http://www.newegg.com/Product/ProductList.asp?Submit=ENE&N=2050400028+1138610150&Subcategory=28&description=&Ntk=&srchInDesc=

Um, I don't think so (but I could be wrong). I'm not looking to load balance. We're nowhere near saturating either of our two lines. I just want both nameservers to function properly, so that, depending on how your particular DNS resolves, our website still functions.

Jimmy King

It might be easier if we knew what OS and/or HTTPd you're running. Then someone maybe able to tell you how to configure what you need.

I know Apache can be configured to listen on multiple IPs and will respond on whichever IP it received the request on. I would have to assume IIS can do this as well, but I could be wrong and cannot tell you how to configure it if I am correct as my IIS experience is minimal. If your HTTPd only sends data out the default gateway no matter what IP or interface it receives the request on, I'd start looking into a new HTTPd.

With your 2 NAT solution why can't both NAT boxes forward to the same internal IP address? The response should go back out the same NAT as the request came in from. This still leaves you with a single point of failure internally but having just 1 server already causes that anyway.

edit: also, is the problem the web server or the dns? I ask this as one of your posts says you want the dns to work which is wholly a different thing from your httpd not listening on 2 different IPs/responding on the correct IP. I'm assuming you've got this setup as 2 primary dns servers where dns on isp 1 points the domain to the IP on isp 1 and dns on isp 2 points the domain to the IP on isp 2, correct?

SeñorAmor

Jimmy King wrote:

It might be easier if we knew what OS and/or HTTPd you're running. Then someone maybe able to tell you how to configure what you need.

I know Apache can be configured to listen on multiple IPs and will respond on whichever IP it received the request on. I would have to assume IIS can do this as well, but I could be wrong and cannot tell you how to configure it if I am correct as my IIS experience is minimal. If your HTTPd only sends data out the default gateway no matter what IP or interface it receives the request on, I'd start looking into a new HTTPd.

With your 2 NAT solution why can't both NAT boxes forward to the same internal IP address? The response should go back out the same NAT as the request came in from. This still leaves you with a single point of failure internally but having just 1 server already causes that anyway.

Gentoo Linux, running Apache.

My 2 NAT solution won't work because Apache sees an external IP address making a request for a website, and in returning data back to that IP, it uses the default gateway, which might not be the same way the data came in.

Jimmy King

SeñorAmor wrote:

Jimmy King wrote:

It might be easier if we knew what OS and/or HTTPd you're running. Then someone maybe able to tell you how to configure what you need.

I know Apache can be configured to listen on multiple IPs and will respond on whichever IP it received the request on. I would have to assume IIS can do this as well, but I could be wrong and cannot tell you how to configure it if I am correct as my IIS experience is minimal. If your HTTPd only sends data out the default gateway no matter what IP or interface it receives the request on, I'd start looking into a new HTTPd.

With your 2 NAT solution why can't both NAT boxes forward to the same internal IP address? The response should go back out the same NAT as the request came in from. This still leaves you with a single point of failure internally but having just 1 server already causes that anyway.

Gentoo Linux, running Apache.

My 2 NAT solution won't work because Apache sees an external IP address making a request for a website, and in returning data back to that IP, it uses the default gateway, which might not be the same way the data came in.

Oh, I get the problem now. I slightly misunderstood it before. I've gotta head to class but I'll do some thinking on this and see if I can't figure something out.

Jimmy King

I haven't had a chance to read over it in depth but take a look at http://linuxcult.blogspot.com/2006/05/how-can-i-have-two-default-routes.html

It looks like you may need to build your kernel with advanced routing and install the tools needed for that, which the article doesn't talk about and just assumes you've already got that handled.

Ruckus

I'm not sure how to do it in Linux, but in Windows networking we can add additionaly Default Gateways and simply assign them different metrics.

A default gateway entry is simply the IP address which the host will contact if the host cannot seem to directly contact an IP address.

If multiple routes (eg via multiple gateways) are available, the route with the lowest "cost" in metric value will be used.

If the "cheapest" route is not available, the next lowest metric value gateway is used.

Jimmy King

Ruckus wrote:

I'm not sure how to do it in Linux, but in Windows networking we can add additionaly Default Gateways and simply assign them different metrics.

A default gateway entry is simply the IP address which the host will contact if the host cannot seem to directly contact an IP address.

If multiple routes (eg via multiple gateways) are available, the route with the lowest "cost" in metric value will be used.

If the "cheapest" route is not available, the next lowest metric value gateway is used.

That would run into the same issue as my original solutions. He's not actually wanting the second ISP connection to only be there as a backup it's going to actually be active at all times and so the lower cost default gateway would still end up used.

The link I posted above gives instructions for setting up linux (once iproute2 is installed and the correct kernel options are configured) multiple routing tables so that each nic will have its own table with its own metric 1 default gateway. The interface that receives the request will be the one that responds to it and so will use its own routing table with the above config.

SeñorAmor

Jimmy King wrote:

Ruckus wrote:

I'm not sure how to do it in Linux, but in Windows networking we can add additionaly Default Gateways and simply assign them different metrics.

A default gateway entry is simply the IP address which the host will contact if the host cannot seem to directly contact an IP address.

If multiple routes (eg via multiple gateways) are available, the route with the lowest "cost" in metric value will be used.

If the "cheapest" route is not available, the next lowest metric value gateway is used.

That would run into the same issue as my original solutions. He's not actually wanting the second ISP connection to only be there as a backup it's going to actually be active at all times and so the lower cost default gateway would still end up used.

The link I posted above gives instructions for setting up linux (once iproute2 is installed and the correct kernel options are configured) multiple routing tables so that each nic will have its own table with its own metric 1 default gateway. The interface that receives the request will be the one that responds to it and so will use its own routing table with the above config.

Yup. Far as I can tell, you need to have some advanced routing features to do what I want. Guess now's as good a time as any to learn the intricacies of iptables.

Jimmy King

SeñorAmor wrote:

Jimmy King wrote:

Ruckus wrote:

I'm not sure how to do it in Linux, but in Windows networking we can add additionaly Default Gateways and simply assign them different metrics.

A default gateway entry is simply the IP address which the host will contact if the host cannot seem to directly contact an IP address.

If multiple routes (eg via multiple gateways) are available, the route with the lowest "cost" in metric value will be used.

If the "cheapest" route is not available, the next lowest metric value gateway is used.

That would run into the same issue as my original solutions. He's not actually wanting the second ISP connection to only be there as a backup it's going to actually be active at all times and so the lower cost default gateway would still end up used.

The link I posted above gives instructions for setting up linux (once iproute2 is installed and the correct kernel options are configured) multiple routing tables so that each nic will have its own table with its own metric 1 default gateway. The interface that receives the request will be the one that responds to it and so will use its own routing table with the above config.

Yup. Far as I can tell, you need to have some advanced routing features to do what I want. Guess now's as good a time as any to learn the intricacies of iptables.

I'm not sure iptables will do what you need.

iproute2 and the advanced routing features in the kernel are a whole different thing from iptables.

vonPoonBurGer

I'm assuming, since it's a webserver, that traffic originates with the client and not the server. If that's the case, it'd make a lot more sense to set up a DNS round robin arrangement, such that even numbered clients hit your webserver's IP address on the first ISP connection, and odd clients hit the IP of your second ISP connection.

Let's say you set up a DNS round robin arrangement, so that half the clients get 1.2.3.4 when they resolve your webserver's hostname, and the other half get 5.6.7.8. Your ISP(s) can probably help you with that, unless you're running your own DNS server. In order to ensure that your webserver replies using the same interface that the traffic arrived on, you'll want to do source-based routing as well. Recompile your kernel with advanced router and multiple routing tables support, then install iproute2. A tutorial on iproute2 is available here. Then all you have to do is set up source-based routing rules. Edit /etc/iproute/rt_tables and add two additional tables, let's say they're called isp0 and isp1. If eth0 is the interface for the first ISP, and eth1 the interface for the second ISP, you're source-based routing rules would like this:

ip rule add iif eth0 table isp0
ip rule add iif eth1 table isp1
ip route add default via <first ISP gateway> dev eth0
ip route add default via <second ISP gateway> dev eth1

Half the time, clients will get the IP address from your primary ISP. When they hit that address, your webserver will respond using the primary ISP interface, and thus the traffic will be coming from the IP that the client originally used. The rest of the time, clients will hit the IP from your secondary ISP, and when they do your webserver will reply using the secondary link. I use source-based routing as part of the customer connectivity solution where I work, it's pretty slick. Beats the living daylights out of a firewall sandwich.

Jimmy King wrote:

iproute2 and the advanced routing features in the kernel are a whole different thing from iptables.

Edit: The King is correct.

Jimmy King

Actually, if I'm understanding the config right, what he is doing makes sense.

I assume that DNS is hosted locally just like the web server and will be configured so that ns0.senoramor.com is on the link from ISP 1 and ns1.senoramor.com is on the link from ISP 2 with each of them configured as masters so that they can mantain separate/different zone information. The name server on ISP 1 will resolve www.senoramor.com to the IP assigned by ISP 1 and DNS hosted on ISP 2 will resolve www.senoramor.com to the IP assigned by ISP 2.

This way if the link from ISP 1 dies there is still a name server on ISP 2 and it will only be resolving the url to the IP which is still accessible.

If he round robins the name resolution then if one of the links dies every other attempt to resolve is going to go to the IP that is on the dead link and be inaccessible.

There's likely a better/more common way of doing this but I bet it costs a lot more and requires some fancy/expensive hardware.

SeñorAmor

Jimmy King wrote:

Actually, if I'm understanding the config right, what he is doing makes sense.

I assume that DNS is hosted locally just like the web server and will be configured so that ns0.senoramor.com is on the link from ISP 1 and ns1.senoramor.com is on the link from ISP 2 with each of them configured as masters so that they can mantain separate/different zone information. The name server on ISP 1 will resolve www.senoramor.com to the IP assigned by ISP 1 and DNS hosted on ISP 2 will resolve www.senoramor.com to the IP assigned by ISP 2.

This way if the link from ISP 1 dies there is still a name server on ISP 2 and it will only be resolving the url to the IP which is still accessible.

If he round robins the name resolution then if one of the links dies every other attempt to resolve is going to go to the IP that is on the dead link and be inaccessible.

There's likely a better/more common way of doing this but I bet it costs a lot more and requires some fancy/expensive hardware.

Yes. This is exactly what I want to do. Trying to configure rules and tables properly with iproute2 is extremely confusing.

Yar

By definition, you can't have more than one default route. The default route is where to route traffic when there's no explicit route for it.

DNS round robin ftw.

SeñorAmor

Yar wrote:

By definition, you can't have more than one default route. The default route is where to route traffic when there's no explicit route for it.

DNS round robin ftw.

Yes, thank you for your opinion on DNS round-robin. In case I hadn't made it clear, that's not what I'm interested in. I am interested in having two public nameservers, on different ISPs on the same computer; having data traverse the same path out as it took on the way in.

Yar

SeñorAmor wrote:

In case I hadn't made it clear, that's not what I'm interested in.

You hadn't.

SeñorAmor wrote:

I am interested in having two public nameservers, on different ISPs on the same computer; having data traverse the same path out as it took on the way in.

So you want one machine to have two IPs and two names on two nameservers? Two NICs. What is the problem, again?

EDIT: Also, load balancing isn't just about preserving bandwidth. It also ensures connectivity to at least one of two possible paths. Most modern firewalls will allow you to load-balance, as will a dedicated device.

blincoln

Your web server should not be aware that there are multiple internet connections. That functionality should be abstracted at the perimeter, like with a firewall or whatever.

If I were going to do something like this, I'd use Ruckus' suggestions about adding multiple gateways with different metrics *on the device at the perimeter*.

The DNS round robin is a good idea for load balancing in addition to that. There are better ways, but they are more complicated and/or cost money.

SeñorAmor

Yar wrote:

SeñorAmor wrote:

In case I hadn't made it clear, that's not what I'm interested in.

You hadn't.

Actually, I did. That was just my nice way of saying, "Learn to fucking read the thread, moron."

SeñorAmor wrote:

I am interested in having two public nameservers, on different ISPs on the same computer; having data traverse the same path out as it took on the way in.

So you want one machine to have two IPs and two names on two nameservers? Two NICs. What is the problem, again?

EDIT: Also, load balancing isn't just about preserving bandwidth. It also ensures connectivity to at least one of two possible paths. Most modern firewalls will allow you to load-balance, as will a dedicated device.

The problem is that standard routing won't always route data back out the way it came in. I need advanced routing for that, which is way over my head, but I'm willing to at least try to get it to work.

Also, I know about load balancing. Right now, it just isn't a concern of mine.

SeñorAmor

blincoln wrote:

Your web server should not be aware that there are multiple internet connections. That functionality should be abstracted at the perimeter, like with a firewall or whatever.

Perhaps, but either way I'm going to need multiple connections and advanced routing. Why not just do it on one box instead of 3?

blincoln

Doing it on the firewall means only doing it on one machine - the firewall. That gives you an internet connection that is load balanced and redundant not just for the web server, but for anything else on the internal network.

You also shouldn't need to worry about the data taking a different path back. That's not important, AFAIK.

Yar

SeñorAmor wrote:

Actually, I did. That was just my nice way of saying, "Learn to fucking read the thread, moron."

I read it over again one more time, just to make sure, and no, I don't see anywhere where you even came close to making that clear. "You hadn't" was just my nice way of saying, "don't get combatative with people who are trying to understand your situation and offer help, prick."

SeñorAmor wrote:

The problem is that standard routing won't always route data back out the way it came in. I need advanced routing for that, which is way over my head, but I'm willing to at least try to get it to work.

Also, I know about load balancing. Right now, it just isn't a concern of mine.

Well, ok, I'm not challenging what you do or don't know. But I still don't understand the routing issue that you are perceiving. Depending on the connection and routing protocol implementation, it will either route back out the interface it came in, or it will simply take whichever route is fastest. Either way, you should be ok even if one ISP is down. And as far as I can possibly understand the situation, a firewall load-balancing two public interfaces will solve the problem.

But I understand you want to do this by simply adding a NIC. I suggest reading up on Virtual Hosts in Apache. You can even do them IP-based, which I imagine is what you are interested in. It will be like having two Webservers at two IPs, except you can point them all to the same files.

It might help if you gave us an idea of the exact problems you're encountering while attempting to implement this.

Anyway, we use DNS failover here. It's a relatively cost effective solution. It pings an IP every minute or so, and if it ever fails to get a response for a period of time, it immediately changes the DNS record to point to a backup IP.