CSE 306: Lab 4: Kernel Rootkit

Due on Friday, May 10, 2013, 11:59 PM
Note: You may use your remaining late hours on this lab.

Introduction

This lab will introduce you to Linux kernel programming and OS security issues, by implmenting a simple rootkit. This lab is designed to be a bit more open-ended.

The course staff have provided you with a simple kernel rootkit as a starting point. Your task will be to take several steps to hide the rootkit from the administrator or user.

Getting Started

We will provide you with some initial source code to start from. To fetch that source, use Git to commit your Lab 3 source, fetch the latest version of the course repository, and then create a local branch called lab4 based on our lab4 branch, origin/lab4:

The git checkout -b command shown above actually does two things: it first creates a local branch lab4 that is based on the origin/lab4 branch provided by the course staff, and second, it changes the contents of your lab directory to reflect the files stored on the lab4 branch. Git allows switching between existing branches using git checkout branch-name, though you should commit any outstanding changes on one branch before switching to a different one.

You will now need to merge the changes you made in your lab3 branch into the lab4 branch, with the git merge lab3 command.

In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g. if you modified some of the code that is changed in the third lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a.

Sharing code with a partner

Important: In this lab, you may work in teams of up to 4 students. We recommend creating larger teams so that you can help each other find your way around the OS kernel. This lab is challenging, and you will benefit from working on larger teams.

Unless we hear otherwise from you, we will assume you are working with the same partner as lab 3. You are welcome to change partners if you like; if you do, please email the course staff immediately to change permissions on your repositories.

We will set up group permission to one team member's git repository on scm. Suppose Partner A is the one handing in the code. Partner A should follow the instructions above to merge the lab4 code. After Partner A has pushed this change to scm, Partner B should simply clone Partner A's repository and use it. For example:

Note that it may take a few days about letting the course staff know your partner selection for the tech staff to apply these permission changes. Again, you are not required to use git to coordinate changes, only to hand in the assignment, but we recommend you learn to use git. You may use any means you like to share code with your partner.

Hand-In Procedure

When you are ready to hand in your lab code and write-up, create a file called slack.txt noting how many late hours you have used both for this assignment and in total. (This is to help us agree on the number that you have used.) This file should contain a single line formatted as follows (where n is the number of late hours):

Then run make handin in the labs directory. If you submit multiple times, we will take the latest submission and count late hours accordingly.

In this and all other labs, you may complete challenge problems for extra credit. If you do this, please create a file called challenge.txt, which includes a short (e.g., one or two paragraph) description of what you did to solve your chosen challenge problem and how to test it. If you implement more than one challenge problem, you must describe each one.

This lab does not include any questions for you to answer, but you should document your design in the README file.

For Your Safety

Modifying the OS kernel on your system can lose all data on the system! If you introduce a null pointer in a regular program, it crashes and loses all of its data; the same is true of an OS kernel. If you introduce a bug in the OS, it will crash. When an OS crashes, it can corrupt the file system and lose all of your data (but we hope it won't). Thus, it is essential that you do two things to protect yourself.

Snapshot your VM before you start. This can be done through the vSphere client---there is a button to take a snapshot and roll back a snapshot of the VM. Note that this will not save your changes, but will allow you to recreate a corrupted VM on your own (rather than waiting for the system administrator do to this).

Push your code to another machine before testing. Before you install and test kernel code, be sure to use git to commit and push your code to another machine (e.g., scm). That way, if the file system is corrupted, you don't lose your work. If you don't want to inflict untested code on your teammates, create a branch in git.

Rootkits

A kernel rootkit is a particular type of malware that hides its presence from the user and system administrator, by modifying the OS kernel. A rootkit generally conceals another type of malware, such as a "back door" that allows an attacker to come back to the system later.

For example, if an attacker breaks into a system once, the attacker may set up an ssh service that accepts an attacker-provided key, running as root and listening on an unusual port, that expects a different password than the system administrator knows about.

The problem is that, if the administrator does a ps and sees that an unexpected ssh daemon is running, this alerts the administrator that something is wrong. Thus, the attacker will also install a rootkit, which hides the presence of the ssh daemon.

A rootkit is a kernel module---a library dynamically loaded into the kernel. Modules are a common kernel facility for writing device drivers and other kernel extensions. Modules run as part of the OS kernel and have unfettered access to any OS kernel data structure or device---a module is trusted to behave itself. Unlike an application, which is in a protected address space, a buggy or malicious kernel module can corrupt the system arbitrarily.

Rootkits make small changes to OS kernel data structures to hide the presence of malicious code. In our ssh example, a rootkit might hide the ssh process in the output of the ps command. A rootkit might also hide its binary in the file system, the open socket from netstat, or even hide its CPU usage.

Note that rootkits are generally used after an initial attack. In other words, if an attacker compromises a web server or other application with an external network connection, the attacker installs a rootkit to make it easier to come back later. Rather than re-attack the initial entry point, which could be patched later, the attacker simply uses a secret service to log in.

In this lab, you will learn your way around the Linux kernel by making a few small, but tricky changes to hide such a ssh service.

Building the skeleton code

In the lab4 directory, type make to build the rootkit.ko kernel module. You can load the kernel module by typing sudo insmod ./rootkit.ko; the module can be unloaded by typing sudo rmmod rootkit. You can view all loaded modules by reading the output of the lsmod command. You can confirm that the rootkit was loaded by checking the output of dmesg:

Our rootkit will also set up an ssh daemon which listens on port 19999. Note: an external firewall blocks this port, so ssh-ing to this port from outside another VM will not work.

To set up the "secret" ssh, first generate a set of ssh keys you want to use. This can be done by typing ssh-keygen -t rsa. This will create two files in your .ssh directory: id_rsa (your private key) and id_rsa.pub (your public key). Issue the following commands:

This will set up a fake directory for authorized keys. The directory must be owned by root with permissions 700. Once this is all set up, launch the ssh daemon by running sudo ./start-daemon.sh. You should see this daemon running in ps, and you can test it using this command:

This command should drop you to a root shell on the local machine, without asking for a password.

Understanding the Skeleton Code

All kernel modules provide an init and exit method, which are called when the module is loaded and unloaded, respectively. This is currently all the rootkit includes; many modules will create devices or register other callbacks.

The rootkit also provides examples of how to find kernel functions. Some functions are exported explicitly as symbols: a module can simply call these.

In other cases, functions are only meant to be called within the kernel. In these situations, we use kallsyms_lookup_name to find the address of the function, and cast it to the appropriate symbol. For instance, the provided code has an example of how to find the unmap_page_range, kernel-private function.

Note: You probably will not need the particular functions we provide---don't feel like you are doing anything wrong if you don't use them. These are provided only as examples of how to find functions you may need.

Helpful Resources

The best resource to finding kernel function is the Linux Cross-Reference (LXR), located at http://lxr.linux.no/. This site includes a number of useful features that can help you find your way through the source code.

These books (available through the campus Safari Online subscription), also are a helpful reference in understanding Linux kernel code:

Debugging the kernel

Attaching a debugger to a running kernel is tricky, especially if you try to run the debugger on the same machine! But it is possible. We will give you a few tips that can help you when printk and intuition aren't enough.

Installing vmlinux

The first thing you will need is an uncompressed vmlinux file (your VM is actually booting a vmlinuz file, which is a compressed kernel image). Issue the following commands to get an uncompressed kernel image with debugging symbols:

Once this is finished, you should see a file in /usr/lib/debug/boot such as vmlinux-3.2.0-40-generic-pae, in addition to the similarly named vmlinuz files.

Attaching to the running kernel.

You can inspect variable values on your running kernel using the command below (substituting the version of the running kernel as appropriate). However, this approach will not let you set breakpoints, only inspect values)

Running your kernel in a system emulator

You can also run your kernel/rootkit inside of qemu (or another emulator), and attach gdb to the emulator. Qemu emulates x86 hardware, and can run a complete OS kernel. Qemu can also export the gdb protocol over a network socket, allowing more access to the running kernel.

Download a disk image here, which is a file that qemu will treat like a disk. The disk image contains a simple Ubuntu file system (very similar to your VM in many respects). You can mount the disk image and add files to it using commands as below:

At this point, a window should pop up. If a window cannot, or you get X errors, log into your VM again, being sure to provide the '-X' option to ssh.

The window will say that qemu is stopped. It is waiting for gdb to attach to it and continue. Start gdb in another window as below:

From here, you should be able to set breakpoints, inspect memory, break execution, etc. You can type c to continue execution.

Core Assignment

Making the rootkit persistent.

If you just install the rootkit by hand, it will not be reloaded after the system reboots. Thus, a sensible first step is to create and install an init script, which is run during boot, to relaunch the rootkit.

There are a number of helpful references on the internet about how to write an init script, including this one, or yoou can "pattern match" the existing ssh init script.

Exercise 1. (10 points) Create an init script in /etc/init.d and appropriate /etc/rc*.d directories that launches your rootkit upon reboot.
Be sure to test that the daemon and kernel module are loaded after rebooting your VM (using sudo shudown -r now.
Turn in a copy of your init script in the file rootkit-init.

Hide the ssh server

The next step is to take our "extra" ssh process out of the output of the ps command, so an administrator wouldn't notice this is running.

Note: For the remaining exercises, feel free to be creative and implement each task how you think best. The handout will have suggestions, but there may be multiple approaches that will work. Be sure to document your approach in the README file.

One approach is to remove the process's directory from the /proc directory, or hide it when a process calls readdir or getdents on the /proc directory. You can find code for the proc file system in the fs/proc directory of the kernel source.

Note: Avoid making overly disruptive changes, as you want the process to continue running properly.

Hint: You may benefit from starting with the strace tool, which dumps the list of all system calls issued by a command. For instance, you can type strace ps -eaf and it will dump a lot of output to the console listing all sysetm calls and arguments. This can give you a fair bit of insight into how these utilities work. From there, you might have some better intuitions about which system call or to focus on adding a filter to.

Hint: You may find it helpful to pass things like the process ID to hide as a module parameter. You can learn more about module parameters in The Linux Kernel Module Programming Guide, specifically Section 2.6.

Exercise 2. (25 points) Hide the ssh process from the output of ps -eaf.
Be sure to double check that it is really running by ssh-ing to it.

Hide the module itself

If you want to hide your rootkit, the next task is to prevent the administrator from seeing strange entries in lsmod.

Exercise 3. (25 points) Hide the rootkit module from the output of lsmod.
At this point, you may want to also remove any "hello world" debug messages.

Hide the open socket

The sudo netstat -nap command shows the administrator all open sockets on the system. The administrator shouldn't be able to see that there is something listening on port 19999 (although she may find it if she randomly tried to ssh to this port). Again, this list is exported through various files in the /proc/net directory.

Exercise 4. (20 points) Hide the open socket on port 19999 from the output of sudo netstat -nap.

Hide the module files

Of course, the final task is to hide all of the relevant module files. We don't want the administrator finding the rootkit.ko file, the alternate ssh config, or the bogus authorized_keys files lying around. So create a file system filter that hides them.

Hint: Each file's inode object points to a set of routines, defined in include/linux/fs.h (see the file_operations structure). This includes a readdir operation. One approach is to look up the parent directory's inode using filename_lookup, and redirect it to your own file_operations, which filters out names you don't want others to see.

Exercise 5. (20 points) Hide all rootkit-related files after the rootkit is loaded. Commands including ls should not report that the file exists.
It is ok if an attempt to create a file with the same name fails.

Challenge! Think of other ways an administrator might notice the presence of your rootkit, and mask these. Be sure to report any tricks you use in challenge.txt.