LVS Cluster Configuration HOWTO
Wayne Sherrill (sherrill@redhat.com)
 
This document explains how to set up and administer a Linux Virtual Server (LVS) cluster that provides highly available Web and FTP services.
1. Introduction
A Linux Virtual Server (LVS) cluster is a collection of servers that have been specially configured to provide highly available Web and FTP services. The diagram below illustrates how an LVS cluster works. Service requests arriving at an LVS cluster are addressed to a virtual server: a publicly advertised, fully-qualified domain name that is associated with a floating IP address, and which can be migrated to a different host.
Figure 1. LVS Cluster Interactions
  
                             _/\__/\_ 
                            |        | 
                           / Internet \ 
                           \_  _  _  _/ 
                             \/ \/ \/
                                 |
                                 |
                       Virtual Server IP/FQDN
             |------------------------------------------|
             |eth0                                      |eth0
       ------|-----                              -------|-----
       | Primary  |                              |   Backup  |
       |  Node    |                              |    Node   |    LVS
       |          |                              |           |  routers
       ------|-----                              -------|-----
             |eth1                                      |eth1
             |-------|------------|-----------------|---|
                     |            |                 | 
                     |            |                 |
                     |            |                 |
                |----|----|  |----|----|       |----|----|
                | Web/FTP |  | Web/FTP |       | Web/FTP |        real
                |  Node#1 |  |  Node#2 |  ...  |  Node#n |       servers
                |_________|  |_________|       |_________|

An LVS cluster consists of one or two router nodes (top of figure) and a variable number of Web/FTP servers (bottom). We will refer to the LVS router nodes as LVS routers, and to the pool of Web/FTP servers as real servers. The real servers are connected via a private network. The LVS routers are connected to this private network and also to the public network. The adapters connecting the LVS routers to the public and private networks (eth0 and eth1 in the figure) may be any device but must be the same on each router. 
Only one LVS router is active at a time. The role of the active router is to redirect service requests from the virtual server address to the real servers. The redirection is based on one of the four supported load-balancing algorithms (described in Table 1). The active router dynamically monitors the health of the real servers, and the workload on each, via one of three supported methods (described in Table 2). If a real server becomes disabled, the active router stops sending jobs to the server until it returns to normal operation. 
Periodically, the LVS routers exchange "I'm alive" heartbeat messages over their public connection. Should the backup node fail to receive a heartbeat message within an expected interval, it initiates failover in order to assume the role of the active router. During failover, the backup router takes over the failed router's floating IP addresses (as indicated in the cluster configuration file), and uses the technique known as ARP spoofing: starts announcing itself as the destination for IP packets addressed to the failed node. When the failed node returns to service, it assumes the hot backup role. 
Currently, the LVS cluster supports one routing method, Network Address Translation (NAT). (In the future, tunneling and direct routing will be added.) The diagram below illustrates how a NAT virtual server works.
Figure 2. An LVS Cluster Implemented with NAT Routing

                             _/\__/\_ 
                            |        | 
                           / Internet \ 
                           \_  _  _  _/ 
                             \/ \/ \/
                                 |
                                 |
             |------------------------------------------| public
                                 |                        network
                            eth0 | virtual server IP address
                       --------------------         
                       |   active router  |
                       |                  |
                       --------------------
                            eth1 | NAT router IP address
                                 |
             |-------|-------------------------|--------| private
                     |                         |          network
                     |                         |
                     |                         |
                |----|----------|  |-----------|---|
                | real server#1 |  | real server#2 |
                |               |  |               |  ...
                |_______________|  |_______________|

Client requests for service arrive at a virtual server IP address. This is a publicly-advertised address that an administrator at the site will have associated with a fully-qualified domain name (for example, lvs.ajax.com). The illustration shows only one virtual server address, but there may be many. A unique virtual server address is a triplet comprising a protocol (TCP or UDP), IP address, and port number. 
The IP components of virtual server addresses are floating addresses. They may be aliased to the device (for example, eth0:1) that connects the LVS routers to the public network, or each could be associated with a separate device. The NAT router IP address, also a floating IP address, is the default route used by each real server on the private network to communicate with the active router. As with virtual server addresses, the NAT router IP address may be aliased to the device (for example, eth1:1) connecting the virtual server to the network of real servers, or it could be associated with a separate device. 
Virtual server and NAT addresses are enabled only on the active router. Thus, should the active router fail, the backup router enables the virtual server and NAT addresses during take-over of the floating IP addresses. In the topology shown in Figure 2, virtual server addresses are enabled on device eth0 and the NAT router address on eth1. 
The IPVS table in the kernel maps requests from the virtual server address to a real server on the private network. For example, a TCP request addressed to port 80 on virtual server 1.2.3.1 might be routed to port 80 on real server 192.168.1.2. The actual mapping in the IPVS table of jobs to real servers is based on which load-balancing algorithm is in use. Table 1 describes the supported load-balancing methods. 
Table 1. Load-balancing Methods
Name	Description	
Round robin	Distribute jobs equally among the real servers.	
Least-connections	Distribute more jobs to real servers with fewer active connections. (The IPVS table stores active connections.)	
Weighted round robin	Distribute more jobs to servers with greater capacity. Capacity is indicated by the user-assigned weight, which is adjusted upward or downward by dynamic load information.	
Weighted least-connections	Distribute more jobs to servers with fewer active connections relative to their capacity. Capacity is indicated by the user-assigned weight, which is adjusted upward or downward by dynamic load information. 	

As a real server processes a request, it returns packets to the active router, where the address of the real server in the packets is replaced by the virtual server address. In this manner, the private network of real servers is masqueraded from requesting clients. 
2. Components of an LVS Cluster
The components of an LVS cluster are described below.
2.1. pulse
This is the controlling process that starts the other daemons as needed. It is started on the LVS routers by the /etc/rc.d/init.d/pulse script, normally at boot time . Through pulse, which implements a simple heartbeat, the inactive LVS router determines the health of the active router and whether to initiate failover. 
2.2. lvs
The lvs daemon runs on the LVS routers. It reads the configuration file and calls ipvsadm to build and maintain the IPVS routing table. 
2.3. nanny
The nanny monitoring daemon runs on the active LVS router. Through this daemon, the active router determines the health of each real server and gets its workload. A separate process runs for each real server used by each virtual server. 
2.4. /etc/lvs.cf
This is the LVS cluster configuration file. Directly or indirectly, all daemons get their configuration information from this file.
2.5. piranha
The GUI tool for monitoring, configuring, and administering an LVS cluster. Normally this is the tool you will use to maintain /etc/lvs.cf, restart the running daemons, and monitor an LVS cluster. 
2.6. ipvsadm
This tool updates the IPVS routing table in the kernel. The lvs daemon sets up and administers an LVS cluster by calling ipvsadm to add, change or delete entries in the IPVS routing table.
3. Background of the LVS Cluster
The Red Hat LVS cluster is based either directly on contributions from the Linux community, or on components that were inspired or enriched by Linux community projects. 
The primary source of the LVS cluster is Wensong Zhang's Linux Virtual Server (LVS) kernel routing algorithm (see http://www.linuxvirtualserver.org). The capabilities of the LVS project that the Red Hat LVS cluster currently supports are: 
•	Building virtual servers: floating IP addresses where requests for service arrive from the public internet. 
•	Routing service requests from virtual servers to a pool of real servers. 
•	Load-balancing (see Table 1). 
•	Packet-forwarding Network Address Translation (NAT). 
The LVS innovations supported by the Red Hat LVS cluster are based on a number of technologies including Network Address Translation (NAT), IP Masquerading, and Port Forwarding. For a good general discussion and index of the relevant HOWTOs on these and related topics, see http://www.linas.org/linux/load.html. 
4. Hardware/Network Requirements
An LVS cluster consists of one or two LVS routers and a collection of real servers providing Web and FTP services. The connection and hardware requirements are described below. 
4.1. LVS Routers
A Linux server that will be the primary LVS router is required. This machine needs two network adapters, one connecting it to the public network and the other to a private network of real servers. 
If you want failover capability, a second Linux server that will be the backup LVS router is required. This machine also needs two network adapters connecting it to the public network and to the private network of real servers. The adapter devices must match on the two LVS routers. Thus, if devices eth0 and eth1 on the primary LVS router connect it to public and private network, respectively, then these same devices on the backup LVS router must connect it to the public and private network. Note that the backup LVS router is a purely hot standby machine. 
4.2. Real Servers
The private network to which the LVS routers are connected includes a variable number of Web/FTP server hosts. Jobs addressed to virtual servers are redirected to these real servers, which may be any kind of computing platform running any operating system or Web server. 
During configuration, you assign a weight to each real server. This is an integer reflecting each server's processing capacity (based on memory, processor speed, number of processors, etc.) relative to that of the others. It is the ratio (2/1, 20/10, 200/100) that is significant. For example, a weight of 2000 assigned to a server indicates that it has twice the computing power of a server assigned a weight of 1000. The assigned weight, which may be dynamically adjusted based on load information, is used by two available job scheduling algorithms (described in Table 1). You should be prepared to assign accurate weights. 
5. LVS Router Prerequisites
The LVS routers require Red Hat Linux 6.1 or greater. Also, packet forwarding, packet defragmenting, and IP masquerading must be enabled on the LVS routers. 
To enable packet forwarding and defragmenting, make sure the following two lines are present in /etc/sysconf/network: 
    FORWARD_IPV4=yes
    DEFRAG_IPV4=yes
These lines cause /etc/rc.d/rc.sysinit to execute these two commands whenever the routers boot: 
    echo 1 > /proc/sys/net/ipv4/ip_forward
    echo 1 > /proc/sys/net/ipv4/ip_always_defrag
To enable IP masquerading, issue this command: 
    ipchains -A forward -j MASQ -s n.n.n.n/type -d 0.0.0.0/0
where:
•	 n.n.n.n is the address of the private subnet to which the real servers are connected. 
•	type is 8, 16, 24, or 32 indicating the address type and mask: 
     netmask         | type | Subnet
     ~~~~~~~~~~~~~~~~|~~~~~~|~~~~~~~~~~~~~~~
     255.0.0.0       | 8    | Class A
     255.255.0.0     | 16   | Class B
     255.255.255.0   | 24   | Class C
     255.255.255.255 | 32   | Point-to-point
You will probably want to put the ipchains command in an init script (e.g., /etc/rc.d/rc.local), so that masquerading is configured on the LVS routers at system startup. 
ipchains is the tool used to create and manage firewalling rules set in the kernel's TCP stack. Masquerading is a small subset of these rules that allows machines making use of private IP networks to communicate with the outside world. Using ipchains can have an impact on system security. If you have security concerns, read the ipchains HOWTO (http://www.linuxdoc.org/HOWTO/IPCHAINS-HOWTO.html).
6. Cluster Node Interconnection Prerequisites
During configuration, you select the tool family (rsh or ssh) that will be used to synchronize the /etc/lvs.cf configuration files on the LVS routers. The selected tool must be enabled on the LVS routers, such that root on each router can log in to the other router without administrator intervention. 
Also during configuration, you select the tool (uptime, ruptime, or rup) that the active router will use to monitor the workload on the real servers. Enable the selected tool on the real servers. If this cannot be done (for example, one of your real servers is a Windows/NT Web server), the cluster will still provide highly available services. However, the Weighted round robin and Weighted least-connections algorithms (described in Table 1 will be affected. Namely, since load information will not be available, the user-assigned weights will be applied statically rather than dynamically adjusted based on server workload. 
Table 2 describes in general terms what you do to enable these tools on the source and destination hosts. For more detailed information, see the man pages. Note that, with rsh and ssh, root must be able to log in over the network. To enable remote root login to a Red Hat Linux system, remove the following line from the file /etc/pam.d/login: 

auth      required     /lib/security/pam_security.so

 This is a security hole, albeit small. Make sure you have the LVS nodes properly firewalled so that logins are allowed only from trusted sources.
Table 2. Enabling Synchronization and Monitoring Tools
Tool	Do This	
rsh	Create a .rhosts file with permission 600 in the root on the destination host naming the source host and user (for example, foo.host1.com root).	
ssh	Obtain/install the tool, which for legal reasons is not released with international Linux distributions. On source and destination hosts, disable remote login via all other methods, set up RSA-based authentication using .ssh/authorized_keys, and start sshd. 	
uptime	On each real server, enable either rsh or ssh as described above.	
ruptime	Set up each LVS router and real server to start rwhod whenever it boots. 	
rup	Set up each real server to start rpc.rstatd whenever it boots. 	

7. Installing the Software
The LVS cluster software is released in three RPM packages: 

piranha (programs)
piranha-gui (graphical configuration tool)
piranha-docs (documentation)

You can install these packages from the 6.1 CD-ROM with this command:
   #rpm -Uvh /mnt/cdrom/RedHat/RPMS/piranha*.rpm
To obtain possible updates of these (or other) packages for your hardware platform, visit http://www.redhat.com/corp/support/errata/rh61-errata-updates.html on the Red Hat website.
8. Configuring an LVS Cluster
You set up and maintain an LVS cluster from the LVS routers, by editing the configuration file and starting or restarting the pulse daemon. Specifically, the steps are: 
	1.	 On the primary router, edit the configuration file  /etc/lvs.cf.  
	2.	 Copy the edited configuration file to the backup router.  
	3.	 Start (or restart) the pulse  daemon first on the primary router, then on the backup router.  
You can perform these steps from the shell, by editing the configuration file with the editor of your choice. The shell commands for starting, restarting, and stopping the pulse daemon are: 
    /etc/rc.d/init.d/pulse start
    /etc/rc.d/init.d/pulse restart
    /etc/rc.d/init.d/pulse stop
The pulse daemon starts or restarts the other LVS cluster daemons as needed, which obtain their configuration information, directly or indirectly, from the current configuration file. 
If you stop pulse (in order to shut down the cluster), stop it on the backup router first. This will prevent the backup router from possibly taking over the active role.
Alternatively, you can use piranha to set up, monitor, and administer your LVS cluster. The entry fields on its windows set or change lines in /etc/lvs.cf, and there are buttons to start, stop, and restart the cluster. Use of piranha requires that the X Window System be configured on the LVS routers. An advantage of using piranha is that it synchronizes the configuration file on the primary and backup routers, and you perform all administrative tasks from the active router. 
The next section describes the LVS cluster configuration file. Read this section if you want to edit this file manually. If you chose to use piranha, read Section 8.2.
8.1. Editing the Configuration File
The /etc/lvs.cf file has three sections. The global section, described in Table 3, sets up the LVS routers and specifies networking and heartbeat parameters. There is one set of parameters for the cluster. The per-virtual-server section, described in Table 4, defines virtual server addresses, sets up the associations between virtual servers and real servers, and specifies job-scheduling parameters. There is a separate set of parameters for each defined virtual server. The per-real-server section, described in Table 5, defines the real servers routed to from each virtual server. There is one set of these parameters for each virtual server. 
Table 3. Setting Global Parameters
Parameter	Description	
primary =	Enter the IP address of the adapter connecting the primary LVS router to the public network.	
backup =	Enter the IP address of the adapter connecting the backup backup LVS router to the public network.	
heartbeat_port = 	Enter the port number used for the heartbeat on the primary and backup LVS routers.	
keepalive =	Enter the number of seconds between heartbeats.	
deadtime =	Enter the number of seconds to wait before declaring a non-responding router dead and initiating failover.	
rsh_command = [rsh|ssh]	Enter the command family to use for synchronizing the configuration files on the primary and backup routers. Important: as described in Table 2, you must enable the selected command on the primary and backup routers.	
network = [nat|direct|tunnel]	Currently, only Network Address Translation (nat) is supported. 	
nat_router = 	Enter the floating IP address and device of the NAT router. This IP address must be the default route used by each real server to communicate with the active LVS router. The IP address is aliased to the device (for example, eth1:1) connecting the LVS routers to the private network of real servers. The device must be the same (i.e., eth1) on both LVS routers. 	

Table 4. Setting Per-Virtual-Server Parameters
Parameter	Description	
name	Enter a unique identifier for the virtual server.	
address =	Enter the virtual server's IP address: a floating IP address that has been associated with a fully-qualified domain name. 	
active = [0|1]	Enable (1) or disable (0) this address.	
load_monitor = [uptime|ruptime|rup]	Select the tool (default uptime) that will be used by the active router to monitor the workload of real servers. Important: as described in Table 2, unless you enable the selected command on the real servers, the scheduling algorithms that use dynamic load information will apply the assigned weight statically rather than adjust the weight from load information. If you select the default (uptime), the tool you specified for "rsh_command" is used to log in to the real servers. This tool must be enabled on the real servers.	
timeout =	Enter the number of seconds (default 10) that must lapse before a real server determined to be dead is removed from the routing table.	
reentry =	Enter the number of seconds (default 180) that a restored real server must remain alive before being re-added to the routing table.	
port = [http/80|ftp/21]	Enter the listening port on this virtual server: http for 80 (the default) or ftp for 21. Or you can enter the numbers.	
scheduler = [wlc|lc|wrr|rr]	Select the scheduling algorithm (default wlc) for distributing jobs from this virtual server to the real servers. The choices are described in Table 1. 	

Table 5. Setting Per-Real-Server Parameters
Parameter	Description	
name	Enter a unique name for the real server.	
address = 	Enter the IP address of the real server on the private network. 	
active = [0|1]	Enable (1) or disable (0) the real server.	
weight =	Enter an integer (default is 1) specifying this server's processing capacity relative to that of other real servers. For example, a server assigned 2000 has twice the capacity of a server assigned 1000. The weighted scheduling algorithms adjust this number dynamically based on workload.	

8.2. Using the Piranha Configuration Tool
To start piranha, become root and type piranha &. When you do, its main window opens. The window has four tabs:
Controls/Monitoring -- The initial tab. Used to start/stop/restart the cluster daemons and monitor the runtime status. See Section 8.2.1. 
Global Settings -- Use to set the IP address of the primary LVS router and NAT router. See Section 8.2.2. 
Redundancy -- Use to set the address of the backup LVS router and set the heartbeating parameters. See Section 8.2.3.
Virtual Servers -- Use to set service addresses and set up routing between service addresses and real Web/FTP server hosts. See Section 8.2.4.
Piranha windows have one or more of these buttons along the bottom:
 OK -- Apply changes and exit from the window.
 Apply -- Apply changes without exiting the window.
 Close -- Exit from the window without applying changes.
8.2.1. Controls/Monitoring Tab
Field/Button	Description	
Start/Stop	When cluster daemons are not running, this button is labeled Start: click it to start the cluster. When the daemons are running, the button is labeled Stop: click it to stop the cluster daemons.	
Add pulse daemon to this runlevel	Select to start pulse at system boot at this runlevel.	
Update information now	Click to display current information in the kernel routing table. 	
Auto-update	Select to display kernel table routing information automatically at the specified interval. 	

8.2.2. Global Settings Tab
Field/Button	Description	
Primary LVS server IP	Contains the public IP address of the primary LVS router. 	
NAT Router IP	Contains the floating IP address associated with the network adapter connecting the virtual server with the network of real servers. This address is the gateway used by the real servers to communicate with the virtual server. If this node fails, this address migrates to the backup LVS node. 	
NAT Router Device	Names the device with which the NAT router IP address is associated. This is done via address aliasing. For example, the NAT router IP may be aliased to physical device eth1 as eth1:1. 	
Sync tool	Select the tool (rsh or ssh) for synchronizing files on the primary and backup LVS routers. The tool you select must be enabled, such that the LVS routers can log in to one another without needing to enter a password. For general procedures, see Table 2. 	

8.2.3. Redundancy Tab
Field/Button	Description	
Enable redundant server	Select to enable failover.	
Redundant LVS server IP	Contains the public IP address of the backup LVS node.	
Heartbeat interval (seconds)	Contains the number of seconds between heartbeats: the interval at which the backup LVS node checks to see if the primary LVS node is alive. 	
Assume dead after (seconds)	If this number of seconds lapses without a response from the primary LVS node, the backup router initiates failover. During failover, the backup router claims all virtual server IPs being processed by the primary LVS router, and broadcasts gratuitous ARPs advertising its MAC address as the destination of packets addressed to the primary router.	
Heartbeat port	Enter the port used for the heartbeat on the primary and backup LVS routers.	

8.2.4. Virtual Servers
This screen displays a row of information for each currently defined virtual server. Click a row to select it. The buttons on the right side of the screen apply to the currently selected virtual server. Click Delete to remove the selected virtual server.
Field/Button	Description	
Status	Displays Active or Down. Click Disable to take down a selected active virtual server; click Activate to enable a selected down virtual server. After changing the status, you must restart the pulse daemon to make the change take affect. To do this, go to the Controls/Monitoring tab, click the Stop button (its name changes to Start), then click Start. 	
Name	Contains a unique identifier for the virtual server.	
Port	Contains the number of the listen port for incoming requests for service. 	
Protocol	Currently, only TCP is supported. 	

8.2.4.1. Add/Edit a Virtual Server
Click the Add button to create a new, undefined virtual server. Click the Edit button (or double-click the row) to define or modify a virtual server. Click the Real Servers tab to view or modify the real servers associated with the selected virtual server. 
Field/Button	Description	
Name	Enter a unique identifier for this virtual server.	
Application	Click to select HTTP or FTP.	
Port	Enter the number of the listen port for incoming requests for service. 	
Address	Enter the floating IP address where requests for service arrive. This address will have been associated with the fully-qualified domain name to which requests for service are addressed. 	
Device	Name the adapter device connecting the LVS routers to the public network, with which the floating IP address is associciated via IP aliasing (For example, eth0:1). If the active router fails, the virtual server address and port are failed-over to this device on the backup router, which becomes the active router. 	
Re-entry Time	Enter the number of seconds (default is 180) that a failed real server associated with this virtual server must remain alive before it will be re-added to the cluster. 	
Service timeout	Enter the number of seconds (default is 10) within which a real server must respond to a redirected request. If a host takes longer than this, it is declared dead and removed from the cluster.	
 	Every two seconds the nanny process on the active router sends a heartbeat (essentially a ping) to each real server. If this succeeds, then a service connect (essentially a telnet conection) is sent out on the specified service port. If nanny gets back any response from a real server, it and the service are assumed to be alive. If "Service timeout" seconds lapse without a response, then the real server is presumed to be dead and is dropped from the routing table. But nanny continues to monitor the server/service by sending out heartbeats every two seconds. If, after a successful ping, a dropped server remains alive for "Re-entry Time" number of seconds, the server is re-added to the kernel routing table. 	
Load monitoring tool	Select the tool to use (uptime, rup, or ruptime) for determining the workload on the real servers. The general procedures for enabling these tools are described in the table in Table 2. 	
Scheduling	Select the algorithm for routing requests from this virtual server to the real servers. The choices are described in Table 1. 	

8.2.5. Real Servers
This screen displays a row of information for each currently defined real server. Click a row to select it. The buttons on the right side of the screen apply to the currently selected row. Click Delete to remove the selected real server.
Field/Button	Description	
Status	Displays Active or Down. Click Disable to take down a selected active virtual server; click Activate to enable a selected down virtual server. After changing the status, you must restart the pulse daemon to make the change take affect. To do this, go to the Controls/Monitoring tab, click the Stop button (its name changes to Start), then click Start. 	
Name	Displays the server's name in the cluster.	
Address	Displays the server's IP address. 	

8.2.5.1. Add/Edit a Real Server
Click the Add button to create a new, undefined real server. Click the Edit button (or double-click the row) to define or change the selected real server.
Field/Button	Description	
Name	Enter a descriptive name. 	
Address	Enter the IP address of the real server on the private network. 	
Weight	Assign an integer to indicate this host's processing capacity relative to that of other hosts in the pool. 	

9. Example -- Setting Up a Five-node Cluster
This section describes step by step how to create a cluster of two LVS routers and three Web/FTP servers. First, collect information and set up the five systems as explained in the next section. Then, implement the example either from a shell (explained in Section 9.2) or by starting the GUI configuration tool (explained in Section 9.3). 
Figure 3 shows the network that will exist after you've set up the LVS routers and real servers. All network addresses shown are for purposes of illustration only. 
Figure 3. Layout of the Example Network
|-------|------------------------------------------|---------| Public network
        |eth0=1.2.3.2                              |eth0=1.2.3.3
        |eth0:1=1.2.3.1 (vs1)                      |
  ------|-----                              -------|-----
  |  active  |                              |   backup  |
  |  router  |                              |   router  |  
  |          |                              |           | 
  ------|-----                              -------|-----
        |eth1=192.168.1.1                          |eth1=192.168.1.2
        |eth1:1=192.168.1.254 (NAT router)         |
|-------|-|------------------|-----------------|---|----------| Private network
          |eth0=192.168.1.3  |eth0=192.168.1.4 |eth0=192.168.1.5 
          |                  |                 |
          |---------|        |---------|       |---------|
          |   rs1   |        |   rs2   |       |   rs3   |  
          |_________|        |_________|       |_________|

9.1. Preliminary Setup
	1.	>From the webmaster, obtain a virtual server IP address. In our example this will be 1.2.3.1. Requests for service at the LVS cluster will be addressed to a fully-qualified domain name associated with this address.  
	2.	Locate 5 servers and designate their roles: 1 primary LVS router, 1 backup LVS router, 3 real servers. The LVS routers must be Linux boxes running Red Hat 6.1 or later. The real servers may be any platform running any operating system and Web server.
Steps 3-7 set up the LVS routers.
	3.	On each LVS router, install two ethernet adapter cards, eth0 and eth1. Create a public IP interface on eth0 and a private IP interface on eth1. The public interface device (eth0) is the heartbeat device. The virtual server address is aliased to this device.
 	Primary node	Backup node	
eth0	1.2.3.2	1.2.3.3	
eth1	192.168.1.1	192.168.1.2	

	4.	Designate an IP address (192.168.1.254) for the router device (eth1) connecting the active LVS router to the private network. This floating IP address will be aliased to the router device as eth1:1, and will be the gateway to the private network and the default route used by each real server to communicate with the active router.
	5.	On each LVS router:
	a.	Enable packet forwarding. To do this at system boot, make sure the file /etc/sysconf/network contains the line FORWARD_IPV4=yes. To enable packet forwarding without rebooting, as root issue this command: 

    echo "1" > /proc/sys/net/ipv4/ip_forward
	b.	Enable packet defragmenting. To do this at system boot, make sure the file /etc/sysconf/network contains the line DEFRAG_IPV4=yes. To enable packet defragmenting without rebooting, as root issue this command: 

    echo "1" > /proc/sys/net/ipv4/ip_always_defrag
	c.	Masquerade the private network. Issue this command and put it in /etc/rc.d/rc.local: 

    ipchains -A forward -j MASQ -s 192.168.1.0/24 -d 0.0.0.0

	6.	Decide whether to use the rsh or ssh families for synchronizing LVS cluster files. Verify that your choice is installed such that the LVS routers can log in to one another as root without administrator intervention (see Table 2). In this example, we will choose rsh. 
	7.	On each LVS router, verify that the LVS cluster software is installed (see Section 7.
Steps 8-11 set up real servers.
	8.	On each real server, install an ethernet network card, eth0, create an IP address on the same private subnet as in Step 3, and assign a weight to each server indicating its processing capacity relative to that of the others. In this example, rs1 has twice the capacity (two processors) of rs2 and rs3.
 	rs1	rs2	rs3	
eth0	192.168.1.3	192.168.1.4	192.168.1.5	
weight	2000	1000	1000	

	9.	On each real server, verify that the address named in Step 4 (192.168.1.254) is its default route for communicating with the active LVS router.
	10.	Decide which program (uptime, ruptime, rup) will be used by the active router to monitor the workload on the real servers. If you choose uptime, each LVS router must be able to connect with each real server without administrator intervention, using the tool family you selected in Step 6. See Table 2 for general enablement instructions. If the selected tool cannot be enabled (for example, one of the real servers is an NT box), the scheduling algorithms that use dynamic load information will still work but the user-assigned weights will be statically applied rather than dynamically adjusted based on load.
	11.	Verify that each real server runs an installed and configured httpd server. Note that the real servers must listen on the same port (80 in the example) as the corresponding virtual server. 
	12.	Verify (for example, using telnet or  ping) that each real server can reach hosts on the  public LAN. If a real server on the private network cannot  reach a host on your LAN, this probably indicates a communication  failure between the server and the active router. See Steps 8 and 9.
	13.	Determine the runtime parameters. For some of these, you may   need to experiment over time to obtain optimal values. In this  example, we will use the values listed.
Value	Parameter Description	
1050	Number of the heartbeat listening port on the primary and backup routers.	
2	Number of seconds between heartbeats.	
10	Number of seconds to wait for a non-responding router to respond before initiating failover.	
10	Number of seconds to wait for a non-responding real server to respond before removing it from the routing table.	
180	When a real server that has been removed from the routing table starts responding again, wait this number of seconds before re-adding the server to the routing table.	
wlc	Use the Weighted least-connections load-balancing algorithm (assign more jobs to servers that are least busy relative to their load-adjusted weight). See Table 1 for a description of the choices.	
http	Application. The alternative is ftp.	
80	Virtual server port number. The listening port selected for the virtual server is used on the real servers as well.	

Now we are ready to implement the example. You can do this from a shell as explained in the next section. Or you can use the GUI configuration tool as explained in Section 9.3.
9.2. Implementing the Example from the Shell
	1.	With your favorite editor, open /etc/lvs.cf  and set the values shown below. The number on the right is a link  to the step in Section 9.1 that discusses  this setting.  
# Global section
primary = 1.2.3.2                    3
backup =  1.2.3.3                    3
keepalive = 2                        13
deadtime = 10                        13
heartbeat_port = 1050                13
rsh_command = rsh                    6
network = nat
nat_router = 192.168.1.254 eth1:1    4
# Per-virtual-server section
virtual server vs1 {                 8      
 address = 1.2.3.1                   1
 active = 1
 load_monitor = ruptime              10
 timeout = 10                        13
 reentry = 180                       13
 port = 80                           13
 scheduler = wlc                     13
# Per-real-server section
 server rs1 {                        8
  address = 192.168.1.3              8
  active = 1
  weight = 2000                      8
 }
 server rs2 {                        8
  address = 192.168.1.4              8
  active = 1
  weight = 1000                      8
 }
 server rs3 {                        8
  address = 192.168.1.5              8
  active = 1
  weight = 1000                      8
 }
}

	2.	 Copy the edited configuration file to the backup router.  
	3.	 On the primary router, start the pulse  daemon with this command:
              /etc/rc.d/init.d/pulse start                
          
	4.	 Start pulse on the backup router.  
9.3. Implementing the Example with Piranha
	1.	On the primary LVS router, as root, type piranha & to start the GUI configuration tool. 
	2.	Click the Global Settings tab and enter the values shown:
Field	Enter:	See Step:	
Primary LVS server IP	1.2.3.2	3	
NAT Router IP	192.168.1.254	4	
NAT Router Device	eth1:1	4	
Sync tool	rsh	6	

	3.	Click the Redundancy tab and enter the values shown:
Field	Enter:	See Step:	
Enable Redundant server	(select)	 	
Redundant LVS server IP	1.2.3.3	3	
Heartbeat interval	2	13	
Assume dead after	10	13	
Heartbeat port	1050	13	

	4.	Click the Virtual Servers tab. On the tab, click Add, then Edit, and enter the values shown:
Field	Enter:	See Step:	
Name	vs1	 	
Application	http	13	
Port	80	13	
Address	1.2.3.1	1	
Device	eth0:1	3	
Re-entry time	180	13	
Service timeout	10	13	
Load monitoring tool	ruptime	10	
Scheduling	weighted least-connections	13	

	5.	Click the Real Servers tab. On the tab, click Add, then Edit, and enter the values shown:
Field	Enter:	See Step:	
Name	rs1	8	
Address	192.168.1.3	8	
Weight	2000	8	

	6.	Repeat Step 5 for rs2 (192.168.1.4) and rs3 (192.168.1.5). For them, enter a weight of 1000, which will indicate that rs1 has twice the computing capacity of rs2 and rs3.
	7.	Click Close to return to the Real Servers tab. There, select each real server and click Activate to activate it.
	8.	Click Close to return to the Virtual Servers tab. There, select the vs1 virtual server and click Activate to activate it.
	9.	Click the Controls/Monitoring tab and:
	a.	Click the Start button (name changes to Stop).
	b.	If you check "Add pulse daemon to this runlevel" the cluster will be started at boot time and, on startup of piranha, the button above will usually be labeled Stop.
	10.	Log in to the backup router and start pulse  with this command:  

              /etc/rc.d/init.d/pulse start
          
If you enter the values exactly as shown above, piranha generates at the end of the configuration file the structures shown below for this virtual server and its real servers.
network = nat
nat_router = 192.168.1.254 eth1:1

virtual vs1 {
        address = 1.2.3.1 eth0:1
        active = 1
        scheduler = wlc
        load_monitor = ruptime
        timeout = 5

        server rs1 {
                address = 192.168.1.3
                active = 1
                weight = 2000
        }
        server rs2 {
                address = 192.168.1.4
                active = 1
                weight = 1000
        }
        server rs3 {
                address = 192.168.1.5
                active = 1
                weight = 1000
        }
}