Problem
A high capacity load balancing solution is needed to address current and future needs to provide highly available and scalable services.Solution
Linux Virtual Server (LVS) provides the means of building scalable and high performing virtual cluster server. Heartbeat 2 can be used to feather increase the availability of the virtual services.Limitations
- Iptables redirection to avoid ARP problems with direct routing load balancing is not covered.
- Heartbeat 2 SSH STONITH is used without quorumd or pingd. Very limited "tiereaker" capability.
LVS hides real servers behind a virtual IP and load balances the incoming request across all cluster nodes based on a scheduling algorithm. It implements transport-layer load balancing inside the Linux kernel, also called Layer-4 switching.
There are 3 types of LVS load balancing:
- Network Address Translation (NAT)
Incoming requests arrive at the virtual IP and are forwarded to the real servers by changing the destination IP address. The real server send the response to the load balancer which in turn changes the destination IP address and forwards the response back to the client. As all traffic goes through the load balancer, it usually becomes a bottleneck of the cluster.
- IP tunneling
LVS sends requests to real servers through an IP tunnel (redirecting to different IP address) and the real server reply directly to client using their own routing tables. Cluster members can be in different subnets.
- Direct routing
Packets from end users are forwarded directly to the real server. The IP packet is not modified as the real servers are configured to accept traffic for shared cluster virtual IP address by using a virtual non-ARP alias interface. The response from the real server is send directly to the client. The real servers and load balancer have to be in the same physical net work segment.
As the load balancer is the only entry point for all incoming requests, it would present a single point of failure for the cluster. A backup load balancer is needed as well as a monitoring program that can fail the service along with the connection statuses.
In this example, Linux Drector Deamon (ldirectord) is used to monitor and administer real servers in the LVS cluster and heartbeat 2 is used as the fail-over monitor for the load balancers (ldirectord).
Note
linux-director or ldirector in this document are used to refer to the load balancing server.
Ldirectord monitors the healthy of servers by periodically requesting a known URL and checking that response contains an expected string. If a service fails on a server , then the server is taken out of the pool of real servers and will be resinserted once it comes back on line.
Goals
Linux-directors and real servers will have 1 real interface with their IP address and 1 virtual alias interface that will be configured with the shared Virtual IP (VIP) 192.168.0.20.
- A client will send a request for a web page from 192.168.0.200.
- Ldirector will check the IP and port number and if they are matched for a service, a real server is choosen from the cluster by a scheduling algorithm, and the connection is added into the hash table which records connections.
- The load balancer forwards the packet (VIP is unchanged) to the chosen real server.
- When the real server receives the forwarded packet, it finds that the packet is for the address on its loopback alias interface, it processes the request and returns the result directly to the client.
- Node level monitoring
If one of the nodes (ldirector1/ ldirector2) running cluster resources stops sending out heartbeat signals, declare it dead, reboot the node and fail over all resources to a different node. - Service level monitoring
If the VIP or ldirectord service fails, try to restart the service, if it fails, reboot the node and fail over all resources to a different node. - Service "stickiness"
If a dead or stand-by node becomes active again, keep the resources where they run now and don't fail-back.
Configuration: linux-director (load balancer)
Required softwareInstall heartbeat and ldirector by running:
sudo yum install heartbeat heartbeat-ldirectord perl-MailTools
IP forwarding
The Linux-Directors must be able to route traffic to the real servers. This is achieved by enabling the kernel IPv4 packet forwarding. Edit /etc/sysctl.conf and add net.ipv4.ip_forward = 1.
# /etc/sysctl.conf
net.ipv4.ip_forward = 1
For the changes to take effect, run:
sysctl -p
Ldirectord
Create the file /etc/ha.d/ldrirectord.cf and add:# /etc/ha.d/ldirectord.cf checktimeout=3 checkinterval=5 autoreload=yes logfile="/var/log/ldirectord.log" quiescent=yes virtual=192.168.0.200:80 fallback=127.0.0.1:80 real=192.168.0.110:80 gate real=192.168.0.120:80 gate service=http request="test.html" receive="Still alive" scheduler=wlc protocol=tcp checktype=negotiate
Important: The directives under "virtual=" have to start with a [Tab], not white space.
Explanation
virtual=192.168.0.200:80
Defines a virtual service by IP-address and port
real=192.168.0.110:80 gate
Defines a real service by IP-address and port. The second argument defines the forwarding method, which in this case (gate) translates to "Direct routing"
request="test.html"
Defines what file to request.
receive="Still alive"
Defines the expected response.
See "man ldirectord" for configuration directives not covered here.
What ldirector does:
Ldirectord will connect to each real server once every 5 seconds (checkinterval) and request 192.168.0.110:80/test.html (real/request). If it does not receive the expected string "Still alive" (receive) within 3 seconds of the last check (checktimeout), it will remove the server from the available pool. It will be added again once the check succeeds.
Because of the quiescent=yes setting, the real servers won't be removed from the LVS table. Rather. their weight is set to "0" so that new connections will be accepted. Already established connections will be persistent until they timeout.
Test
/etc/init.d/ldirectord start Starting ldirectord... success ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.200:http wlc persistent 600 -> 192.168.0.110:http Route 0 0 0 -> 192.168.0.120:http Route 0 0 0Note
The weight in this case is 0 because the real servers are not configured and ldirectord could not fetch the test.html page.
Disable ldirectord service
Make sure ldirectord is not running and won't start on boot. Only heartbeat 2 will be allowed start and stop the service.
/etc/init.d/ldirectord stop /sbin/chkconfig ldirectord off
Heartbeat 2
Heartbeat 2 runs on the two Linux-Directors (load balancers) and handles bringing up the interface for the virtual address. This is the a address to which end-users should connect. It will also monitor the ldirectord daemon.Main configuration file
Create a file /etc/ha.d/ha.cf and add:
# /etc/ha.d/ha.cf crm on udpport 694 bcast eth0 node ldirector1 ldirector2Note: The order of directives is significant.
Explaination
crm on
Use heartbeat version 2
udpport 694
Which port Heartbeat will use for its UDP communication
node ldirector1 ldirector2
Node names. Output of uname -n
bcast eth0
Use device eth0 to broadcast the heartbeat
Node authentication
The authkeys configuration file contains information for Heartbeat to use when authenticating cluster members.Create /etc/ha.d/authkeys and add:
# /etc/ha.d/authkeys auth 1 1 sha1 YourSecretKeyThis file cannot be readable or writer by anyone other than root
chmod 600 /etc/ha.d/authkeysName resolution
Add node names to /etc/hosts on boot linux-directors:
# /etc/hosts 192.168.0.10 ldirector1 192.168.0.20 ldirector2Time synchronization
Even though not required, it is very useful in every cluster environment where you want to compare log files from different nodes. The time server should be outside the cluster. See novell documentation on how to configure an NTP client through YaST2
Configuration: real servers
Virtual interface
Edit
/etc/sysconfig/network/ifcfg-lo and add:# /etc/sysconfig/network/ifcfg-lo IPADDR_0=192.168.0.200 # VIP NETMASK_0=255.255.255.255 NETWORK_0=192.168.0.0 BROADCAST_0=192.168.0.255 LABEL_0='0'
Restart the network:
/etc/init.d/network restart
The new
lo:0 virtual interface is now active:ip add sh lo
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet 192.168.0.200/32 brd 192.168.0.255 scope global lo:0
Restrict ARP advertisements
Clients will send all HTTP requests to the VIP 192.168.0.200. Before they can connect to the IP, an ARP request is made to match a MAC address to the requested IP address. Since the linux-directors and real servers both have an interface configured with the same virtual IP address, each one of them can randomly reply to an ARP request for 192.168.0.200. This would break the load balancing for the cluster. To solve this problem, ARP replies for the virtual interfaces have to be disabled.
Edit
/etc/sysctl.conf and add:# /etc/sysctl.conf net.ipv4.conf.all.ARP_ignore = 1 net.ipv4.conf.eth0.ARP_ignore = 1 net.ipv4.conf.all.ARP_announce = 2 net.ipv4.conf.eth0.ARP_announce = 2
Load the changes with:
sysctl -p
Explanation
net.ipv4.conf.all.ARP_ignore = 1Enable configuration of ARP_ignore option
net.ipv4.conf.eth0.ARP_ignore = 1Do not respond to ARP requests if the requested IP address is configured on the “lo” (loopback) device or any virtual eth0:X device.
net.ipv4.conf.all.ARP_announce = 2Enable configuration of ARP_announce option
net.ipv4.conf.eth0.ARP_announce = 2As the source IP address of ARP requests is entered into the ARP cache on the destination, it has the effect of announcing this address. This is undesirable for the lo or any other virtual interfaces from the real servers.
Using this setting, whenever the real server makes an ARP request, it tries to use the real IP as the source IP of the ARP request.
Default gateway
The real-servers need to be set up so that their default route is set to the gateway router’s address on the server network and not an address on one of the linux-directors. In this example, 192.168.0.254 is the default gateway.
echo "default 192.168.0.254" > /etc/sysconfig/network/routes; rcnetwork restart; # and check the routing table route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 192.168.0.254 0.0.0.0 UG 0 0 0 eth0
Web server
- Install Apache2 by running:
zypper install apache2
- Create a test.html page that ldirectord will periodically check to determine if the service is available:
echo "Still alive" > /srv/www/htdocs/test.html echo "Real server 1" > /srv/www/htdocs/index.html
Note:
The default SLES10 SP2 Apache2 DocumentRoot is used in this example.
The default SLES10 SP2 Apache2 DocumentRoot is used in this example.
Repeat the same on real-server2 but change the index.html to “Real server2” so it is visible which web server is serving the request.
Start HTTP service:
/etc/init.d/apache2 start
Note:
We only use a virtual HTTP service. It is possible to configure ldirectord to check any other services, such as , oracle listener, MySQL, SMTP, POP/IMAP, FTP, LDAP, NNTP and others.
We only use a virtual HTTP service. It is possible to configure ldirectord to check any other services, such as , oracle listener, MySQL, SMTP, POP/IMAP, FTP, LDAP, NNTP and others.
Ldirectord test
After setting up and starting the apache web server on both real-servers, check on the linux-director that is currently running the load_balancer resource group if both servers are available in the IPVS server pool:
ldirector2:~ # ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.200:80 wlc -> 192.168.0.110:80 Route 1 0 0 -> 192.168.0.120:80 Route 1 0 0
Now we see both servers with weight of 1. Connect with a browser to: 192.168.0.200. A page served by real-server1 or real-server2 should.
Nhận xét
Đăng nhận xét