The DiRT Linux Setup (FAQ)

Overview

The DiRT Linux Setup is a part of the 10Gig network [PDF]. The following diagram illustrates the setup.

As shown, the network has the following capabilities:

• 7+7 end hosts running linux 2.6, >512MB Memory, >800Mhz Processor
• 2 routers running linux 2.6, 2GB Memory, 3.6GHz Xeons
• 1 control machine (lorenzo) running linux 2.6, 2GB Memory, 3GHz Xeon
• All network interconnects are Gigabit, 10Gig link between routers
• The routers have large amounts of fast local storage for dumps

The Setup

The linux distribution used is Gentoo. There is no fixed release installed; only a limited but sufficient number of packages are installed (and updated). If you require a piece of software on the testbed and its not installed by default please send mail to the current maintainer of the setup.

The filesystem tree on the end systems and routers is an approx. replica of the tree on lorenzo. So if some software is available/works on lorenzo it should work on all machines.

User home directories are resident on lorenzo and are mounted as NFS on the other tested machines. The NFS traffic is restricted to the 138 and 139 subnetworks.

User account management is done using /etc/passwd, /etc/shadow and /etc/groups. To have your account created on the setup please send mail to the current maintainer of the setup. Also, setup ssh keys for password-less login from lorenzo into the rest of the testbed. Since the filesystem trees on the testbed machines and lorenzo are not synced so regularly, password changes on lorenzo might temporarily have no effect on the rest of the testbed. Password-less login using ssh is hence highly recommended.

Access to the lab is governed by the 10G network reservation system. Users of the network are encouraged to notify other users using mail if they are dropping a previously reserved slot or even if they need a slot urgently.

Handy Utilities

As any other testbed, there are scripts to automate some common tasks. /root/all_hosts.sh is a file which contains shell(bash) variables containing the names of all the machines on the testbed. It is recommended that you source (bash-speak for include) this script into your scripts which have to work with all the machines. This helps in adding and removing machines from the network without breaking all the scripts. Please note that the routers are not yet added to the all_hosts.sh script. However, this will be a good thing to do. I will update this section once I am finished with modifying my scripts.

There is a handy script /root/sw/bin/doall.sh which executes the command given as an argument on all the machines in the testbed. It runs the command one by one on all the machines and prints the output of a machine followed by its name. Try a doall.sh uname -rv to see it in action. This script is simple and hence a good starting point to build an automated tool.

Maintenance and Installation

Rome was not built in a day and the same holds for the 10G linux setup :). Actually I say this because the installation procedure for new machines is somewhat unconventional. However, there are good reasons to follow this as it makes the maintenance of the setup a lot easier.

The Gentoo installation on lorenzo is a master replica of all the other testbed machines. The files present on all the testbed machines match very closely with the files on lorenzo. The exceptions include /var, /tmp and configuration files which are different for different hosts. The per host configuration files are kept in separate directories in /root/machines/"machine name". These directories mimic a partial root filesystem and are intended to be an overlay for the lorenzo installation to convert it into the installation on that particular machine. There are also separate directories for the common overlay for "left" side machines and "right" side machines (/root/machines/{left|right}_common). There is a /root/machines/all_common for configuration files which are common for all testbed hosts but are different from lorenzo. The /root/machines/"machine name" files override /root/machines/{left|right}_common files which override /root/machines/all_common which in turn overrides the original lorenzo installation.

The idea of the above is to ease maintenance. Whenever a new piece of software is added to lorenzo or a configuration change is made, it is made on just lorenzo or /root/machines on lorenzo. Then rsync is used to sync the entire tree of files from lorenzo and /root/machines with the rest of the testbed. A script /root/machines/update-all.sh is used to automatically achieve the same. Let me chalk out the steps required for some simple maintenance tasks on the testbed.

• Add/update software: You add/update the software on lorenzo using conventional gentoo package management. When you are done just do a cd /root/machines; ./update-all.sh. Simple!
• Change configuration: First categorize the configuration change into one of the following categories:
  • Affects all machines including lorenzo: lorenzo installation
  • Affects all machines excluding lorenzo: /root/machines/all_common
  • Affects all left side machines: /root/machines/left_common
  • Affects all right side machines: /root/machines/right_common
  • Affects a specific machine X: /root/machines/X
  Add/change the configuration file in the directory pointed out by the above categorization. Make sure that you place new config files mimicing the directory structure on the target machines. You might need to create some or all of the config file's parent directories. When you are done, just do a cd /root/machines; ./update-all.sh. If you need to reboot the testbed use doall.sh reboot.

Installation on a new machine involves the following steps:

1. Connect the machine to the testbed. Make sure the power cabling, KVM and network connections are okay.
2. Boot the Gentoo live CD on the machine. Make sure that the machine boots properly and there are no hardware issues.
3. When logged in as root do a dmesg to check out the hardware on the system. Make sure to check out the following hardware: Disk subsystem(SCSI, IDE, Fusion MPT), Network Interface Cards. Check if drivers for this hardware is present in the kernel running on lorenzo. You might want to take a look at the kernel configuration of the lorenzo kernel for this. If you feel that the kernel will need reconfiguration, shoot a mail to the group so that everybody using custom kernels on the testbed is able to run their kernel on your newly installed hardware. If you are lucky and all drivers are already in the lorenzo kernel then proceed to the next step. Otherwise, please touch base with one of the maintainers of the testbed so that a new kernel can be recompiled for the testbed.
4. In the root directory, create a directory called f/ (or anything else you like; I will call it f/ henceforth)
5. Mount the linux partition for / on f/ and the linux partition for /boot on f/boot. (Please take a look at the Partition Setup section below to find out the conventional wisdom on partitioning hard drives for the linux 10G setup.)
6. Setup networking manually on the system using ifconfig. Use ifconfig interface ip-addr netmask 255.255.255.0. Try pinging lorenzo (152.2.13[8,9].134) to check for connectivity.
7. Startup sshd using /etc/init.d/sshd start. This will generate host keys on the live CD session.
8. Set a root password using the passwd command.
9. On lorenzo add a hosts entry for the newly added machine in /etc/hosts. Then try sshing into the new machine using its name. Proceed to the next step only if you are successfully able to ssh as root on the new machine.
10. scp the /usr/bin/rsync binary to the new machine using scp /usr/bin/rsync "new machine":. I am assuming you left the rsync binary in the /root directory (along side f/) on the new machine.
11. Make a new directory with the name of the new machine in /root/machines. Copy over the configuration files from another machine's directory to the new directory and make appropriate changes. Basically, depending on the partitioning, you need a new grub.conf and fstab, new network settings in /etc/conf.d/net, new /etc/conf.d/hostname etc.
12. Give the following commands to sync an install image on the new new machine. This follows the update-all.sh script rather closely.
    cd /root/machines
    rsync -az --rsync-path=/root/rsync --exclude-from=excludes-install / "new machine":/root/f/ (Make sure you give this command exactly as given. Missing a / might result in something unexpected. This might take some time. You may check progress by looking into /root/f on the new machine to see what has been synced so far.)
    rsync -az --rsync-path=/root/rsync all_common/ "new machine":/root/f/ (Make sure you give this command exactly as given. Missing a / might result in something unexpected.)
    rsync -az --rsync-path=/root/rsync left_common/ "new machine":/root/f/ (Use right_common/ if the new machine resides on the right side of the network instead of the left side. Make sure you give this command exactly as given. Missing a / might result in something unexpected.)
    rsync -az --rsync-path=/root/rsync "new machine"/ "new machine":/root/f/ (Make sure you give this command exactly as given. Missing a / might result in something unexpected.)
13. Check /root/f on the new machine to see if the files seem to be intact.
14. Chroot into this environment using chroot /root/f /bin/bash. This will now allow you to run the software installed in f/ as if it was installed normally.
15. Run grub and setup the boot loader. (Typical root (hd'x','y'); setup (hd0) stuff).
16. Exit the chrooted shell and unmount f/boot/ and f/.
17. Reboot... you new system should be up!

Partition Setup

Handling Multiboot

Typical kernel configuration

Running experiments using tmix

Point of contact