The following is an outline for a HOWTO I'm writing. It discusses how to create single system image (SSI) clusters using UML (User-Mode Linux). This is an interesting concept: UML allows the user to create one or more virtual machines on a single host machine, whereas SSI clustering allows the user to create a single virtual machine on one or more host machines. Combining the two is useful in the following ways: - letting new users easily experiment with a virtual cluster before investing the effort into setting up a hardware-based one - providing a friendlier environment for debugging SSI kernel code and testing new features For those unfamiliar with the Open SSI Clustering project, it was seeded with code from Compaq's NonStop Clusters for UnixWare. It also leverages other open-source technologies, such as OpenGFS and the Mosix load leveler. The goal is a full single system image clustering environment that provides availability, scalability, and manageability at the same time. Note that some of the download links in the outline are not yet available. Although the SSI code works just fine on physical hardware, there are still some UML issues to be resolved. Also, I have yet to write the virtual cluster management commands (ssi-start, ssi-add, ssi-ls, etc.). Feedback on my outline is appreciated. -- Brian Watson | "Now I don't know, but I been told it's Linux Kernel Developer | hard to run with the weight of gold, Open SSI Clustering Project | Other hand I heard it said, it's Compaq Computer Corp | just as hard with the weight of lead." Los Angeles, CA | -Robert Hunter, 1970 mailto:Brian.J.Watson@compaq.com http://opensource.compaq.com/ Introduction Why create SSI clusters using UML? Overview of SSI clustering CI GFS LVS MosixLL SSI Overview of UML Intended Audience System Requirements Copyright License Getting Started Root image (over 100 MB) - download: bash$ wget http://prdownloads.sf.net/ssic-linux/\ root_fs.rh72.ssi.bz2 - extract a working copy: bash$ bunzip2 -c root_fs.rh72.ssi.bz2 >root_fs UML utilities - download: bash$ wget http://prdownloads.sf.net/user-mode-linux/\ uml_utilities_20020212.tar.bz2 - extract the tarball - build and install: bash$ cd tools bash$ su bash# make install bash# bash$ cd .. SSI over UML - download: bash$ wget http://prdownloads.sf.net/ssic-linux/\ ssi-uml-0.6.1.tar.bz2 - extract the tarball - install: bash$ cd ssi-uml-0.6.1 bash$ su bash# make install bash# bash$ cd .. Booting the cluster - the following command starts a four-node cluster: bash$ ssi-start 4 Cluster 10001 has been started with 4 nodes - the number of nodes must be greater than zero and less than sixteen - if the number of nodes is not specified, it defaults to three - each node's console will appear in a new xterm (X Windows must be running) - the script looks for the shared root image at ./root_fs by default; this may be overridden with the [-r ] argument to ssi-start - a 64 MB swap space will be automatically created for each node; the file naming convention is swap_ - the size of the swap space may be overridden with the [-s ] argument to ssi-start; swap_size is in megabytes Booting an individual node - use the ssi-ls command to get the ID of your cluster: bash$ ssi-ls The following clusters have been found: ID Nodes Date started 10001 1 2 3 Mar 06 2002, 06:03:48 PM PST - the following command starts nodes 4 and 5 in cluster 10001: bash$ ssi-add 10001 4 5 Crashing an individual node - the following command crashes node 1 in cluster 10001, forcing all centralized services to restart on node 2: bash$ ssi-rm 10001 1 Playing Around Process migration Distributed process relationships and access Clusterwide fifos Clusterwide devices HA process monitoring and restart What to read next - continue to the next section if you'd like to build your own kernel; this is useful if you'd like to: * customize the kernel configuration * upgrade to the latest SSI source from the CVS repository * debug SSI source or add new features - skip ahead to the section on hardware-based clusters if you'd like to take advantage of the availability and scalability it can offer - otherwise, skip ahead to the section on contributing to see how you can help the SSI clustering project Building a Kernel and Ramdisk Base kernel - download - extract Official SSI release - download - extract - apply patch CVS checkout of SSI - checkout CI sandbox - checkout SSI sandbox - apply UML patch from either sandbox - copy CI files into kernel source - copy SSI files into kernel source Configure - configure kernel: bash$ cd linux bash$ make menu_config ARCH=um - enable the appropriate options - disable conflicting options Build kernel - build the kernel and modules: bash$ make linux modules ARCH=um - mount the shared root bash$ su bash# - install modules into root image bash# make modules_install ARCH=um \ INSTALL_MOD_PATH=/mnt Build ramdisk - use cluster_mkinitrd: bash# cluster_mkinitrd -u /mnt initrd.img 2.4.16-um2 - clean up: bash# chown initrd.img bash# umount /mnt bash# Start cluster - pass new kernel and ramdisk into startup script: ssi-start -k linux/linux -i initrd.img - the -k argument takes the path of the SSI over UML kernel - the -i argument takes the path of its associated ramdisk What to read next Building a Root Base root image (over 100 MB) - download ext2 image: bash$ wget http://prdownloads.sf.net/user-mode-linux/\ root_fs.rh72.pristine.bz2 - extract a working copy: bash$ bunzip2 -c root_fs.rh72.pristine.bz2 \ >root_fs.rh72.pristine - make a blank GFS image: bash$ dd if=/dev/zero of=root_cidev bs=1024 \ count=4096 bash$ bash$ dd if=/dev/zero of=root_fs.rh72.ssi bs=1024 \ count=560000 bash$ - mount both images bash$ su bash# mkdir /mnt.ext2 /mnt.gfs bash# mount root_fs.rh72.pristine /mnt.ext2 -o loop,ro bash# - copy base root into GFS image bash# cp -ax /mnt.ext2 /mnt.gfs bash# Building and installing tools - download and extract: bash$ wget http://prdownloads.sf.net/ssic-linux/\ cluster-tools-0.6.1.tar.bz2 bash$ tar jxf cluster-tools-0.6.1.tar.bz2 - or CVS checkout - build and install: bash$ cd cluster-tools bash$ su bash# make install_ssi_redhat UML_ROOT=/mnt.gfs Configuration - edit /etc/fstab - disable certain startup scripts What to read next Moving to a Hardware-Based Cluster Advantages - availability: the cluster can survive a hardware failure - scalability: each node has real CPUs that truly work in parallel, like a large-scale SMP box; the MosixLL can ensure that the work is fairly well distributed Requirements - one or more nodes (two is the minimum for an interesting cluster) - shared storage hardware (shared SCSI for a two-node, Fibre Channel for more nodes) - private TCP/IP interconnect (100 Mbps ethernet is fine) - an extra machine for the GFS lock server (will eventually be eliminated when GFS is integrated with a distributed lock manager) Resources - SSI website http://ssic-linux.sf.net/ - SSI mailing list ssic-linux-devel@lists.sf.net Contributing Testing Documentation Debugging Adding new features Further Information Clustering SSI CI UML