COMP 530: Lab 2: Simple Shell

You may do the lab alone, or in a group. You should complete this lab with the same team as Lab 1, except with instructor permission. You will extend the code you wrote for Lab 1 to complete a working shell. If you work in a group, please submit one assignment to Gradescope, and list PIDs of all group members in the code comments.

Introduction

To become familiar with low-level Unix/POSIX system calls related to process and job control, file access, IPC (pipes and redirection). You will write a mini-shell with basic operations (a small subset of Bash's functionality). Expected length of this C program is 1000-2000 lines of code (not very long, but the code will be challenging, so start early).

Getting the starter code

The starter code for this assignment is on the lab2 branch of your thsh repository from lab 1. You will need to merge your solution for lab1 into the lab2 starter code.

The git checkout -b command shown above actually does two things: it first creates a local branch lab2 that is based on the origin/lab2 branch provided by the course staff, and second, it changes the contents of your lab directory to reflect the files stored on the lab2 branch. Git allows switching between existing branches using git checkout branch-name, though you should commit any outstanding changes on one branch before switching to a different one.

You will now need to merge the changes you made in your lab1 branch into the lab2 branch, as follows: git merge lab1.

In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g. if you modified some of the code that is changed in the second lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a.

Helpful References

There are no required readings for this lab, but a few references explain how shells work in some detail. These references may provide substantial insight into how to complete this assignment. Do NOT copy and paste code from these sources into your assignment.

Core assignment

You will fill in the C program, called "thsh.c" (for Tar Heels SHell) that performs a subset of commands you're familiar with from other shells like GNU's Bash. You're welcome to study the code for bash, but the code you submit should be your own!

Assuming you completed lab 1 correctly, the starter code for thsh should let you type commands, and they are initially just repeated back to the console:

Note that commands like ls are (usually) just programs. There are a few built-in commands, discussed below. In general, though, the shell's job is to launch programs and coordinate their input and output.

Important: You do not need to reimplement any binaries that already exist, such as ls. You simply need to launch these programs appropriately and coordinate their execution.

Helpful and allowed interfaces

You are welcome to use any standard C version, including C99 or C11, as well as K&R, ANSI, or ISO C.

The code from lab 1 should be sufficient to parse any expected input for thsh. In this lab, you will need to take the parsed command and then use fork(2), clone(2), and/or and exec(2) (or flavors of exec, such as exece, execle, etc.). Programs you run should output to stdout and stderr (errors); programs you run should take input from stdin. You will have to study the wait(2) system call and its variants, so your shell can return the proper status codes. You may use any system call (section 2 of the man pages) or library call (section 3 of the man pages) for this assignment, other than system(3).

Hint: Note that, by convention, the name of the binary is the first argument to a program. Carefully check in the manual of the exec() variant you are using whether you should put the binary name in the argument list or not.

In general, your selection of libraries is unrestricted, with one important exception: you should avoid the use of system(), which is really just a wrapper for another shell. Speaking more broadly, it is not acceptable to simply write a wrapper for another shell---you should implement your own shell for this assignment.

Finding programs

Shells provide a nicer command-line environment by automatically searching common locations for commands. For instance, a user may type ls, and the shell will automatically figure out that the binary is actually located at /bin/ls. On Linux, the paths to automatically search is stored in the environment variable PATH.

In Lab 1, you implemented code to parse the PATH and construct a table of prefixes to search for commands. In this lab, you will complete the run_command function in jobs.c, which actually finds and executes a command. You will also need to add code to thsh.c to call this function.

Hint: You can use the stat() system call to check whether a file exists, rather than relying on the more expensive exec() system call to fail.

You may not use execlp(), execvp(), execvpe(), or any other *p* variant that searches the PATH for you. Also, be sure to correctly handle the case where the shell is launched with different values of PATH (i.e., do not hard-code a given PATH value or array of values).

You will also need to add code to thsh.c that checks whether a command is a built-in. (Hint: see handle_builtin from Lab 1.). You should check for a builtin before trying to find a file on the file system.

Exercise 1. (6 points) Implement simple command support in your shell. Upon reading a line, launch the appropriate binary, or detect when the command is a special "built-in" command, such as exit. For now, exit and cd are the only built-in command you need to worry about, but we will add more in the following exercises.

The shell should print output from commands as output arrives, rather than buffering all output until the command completes. Similarly, if the user is typing input that should go to the running command via stdin, your shell should send these characters as soon as possible, rather than waiting until the user types a newline. If you use the right exec variant, inheritance should handle most of this for you.

You do not need to clear characters from the screen if the user presses backspace (and this doesn't "just work" on your system). Simply rewrite the command on a new line without the missing character.

Be sure to use the PATH environment variable to search for commands. Be sure you handle the case where a command cannot be found.

When you are finished, your shell should be able to execute simple commands like ls and then exit.

Note that for the purposes of testing/grading, your shell should only report an error if there was a failure to create or run the child process. If the child runs and returns a non-zero exit code, shells (including thsh) do not treat this as an error. Rather, a fully functional shell would store this in a special variable called $?; for this lab you may ignore exit_codes from the child.

Changing Directories

Another built-in command you should support is 'cd' to change directory using the chdir(2) system call.

Exercise 2. (3 points.) Add support for changing the working directory, (i.e., cd). Verify that pwd works properly. At this point, you should extend the starter code in handle_cd in builtin.c.

Note that the working directory can affect the interpretation of environment variables, as '.', the current working directory, is a valid entry in PATH.

Note that cd - should change to the last directory the user was in, and cd with no argument should go to a user's home directory (also stored in an environment variable).

Similarly, the built-in command should handle the targets cd . and cd .. properly. (Note that every directory includes these file names if you type ls -a, so this should not require special handling.)

Now that we can change directories, let's add some style to our shell. Any self-respecting shell has a fancier command prompt, which includes the working directory.

Exercise 3. (2 points.) Add the current working directory to your shell prompt. Rather than simply printing thsh> , instead print the current working directory in brackets, like this:

[/tmp] thsh> ls
# shows files in /tmp

Hint: We recommend extending the print_prompt function in builtin.c.

Challenge! (1 bonus point) Complete support for a "NULL" entry in the PATH environment variable, i.e., two colons with nothing between them (there is a related challenge in Lab 1 to handle the parsing for this case). For example: PATH=/bin:/sbin::/usr/bin. In this example, the third entry should be converted to the current working directory --- "./" would suffice. Demonstrate that this setting can find a binary file in the current working directory. When you change directories, the search should be changed to look in the new directory.

Dealing with Zombies!

Whenever you fork a process (i.e., create a new process), the forked process runs in parallel with the forking process. If these processes are not synchronized properly, or do not terminate properly, you run the risk of creating a "zombie" process. A zombie process is a process that does not execute (is "dead") but does not terminate and go away ("die") , and as such continues to consume resources within the operating system such as process descriptors. If zombie processes accumulate, it is possible to slow down, hang, or crash the operating system.

Since this assignment will be the first time many of you have created processes, there is an excellent chance you will create one or more zombie processes because of bugs in your program. If this happens, your system can become sluggish and/or hang until you either reboot or open the task manager (or equivalent) to terminate them.

That you will have bugs related to the use of fork is to be expected. However, to mitigate the effect of these bugs on the performance of the server, you need to take steps to limit the number of processes you can create.

Unfortunately, the command to limit process creation depend on what your shell is. The default shell on your container is bash. But you may be using another shell, such as tcsh or zsh. You can figure out what shell you are running with this command:

Limiting processes in bash

The first command will show the limits of various resources you can consume. The second command limits the number of processes you can create to 10. The third command will again show your limits and allow you to confirm that you've correctly limited the number of processes you can create.

In order to execute these commands every time you create a shell intance, you can edit the file ".bashrc" (with a leading period) in your home directory, and add the line "ulimit -Su 10" to the file as the last line in the file. This will always set the process limit and then you don't manually have to do it every time you log in.

Limiting processes in tsch

In order to execute these commands every time you create a shell intance, you can edit the file ".cshrc" (with a leading period) in your home directory, and add the line "limit maxproc 10" to the file as the last line in the file. This will always set the process limit and then you don't manually have to do it every time you log in.

What to do if you run out of processes?

If you limit the maximum number of processes, then, if you have a bug in your program and are creating zombie processes, you'll eventually get an error message when you try to run your program (the message indicating the maximum number of processes has been exceeded). This error message will be the only indication you get that you have a bug in your program. Should this happen, you won't be able to continue testing your program until you kill off your zombie processes. To kill zombie processes, first, use the "ps" command to see the identities of the processes you've created:

where you replace YOUR-ONYEN with your ONYEN. For example, if this were my login session I'd type:

You can then use the "kill" command to kill any found zombie processes by using the process number (the PID) which is shown under the second column of output by the ps command. To kill a process use the command:

Generally, until you are certain your program is working, execute the ps command prior to logging out so that you can see if you are leaving behind any zombie processes and kill them before you log out.

Debugging

One feature which will help with development of your shell is to add debugging messages, which can be enabled when you start your shell.

Exercise 4. (3 points.) Add debugging messages to your shell

If you start thsh with -d, it should display debugging info on stderr:

every command executed should say "RUNNING: [cmd]", where cmd is replaced with the text of the command. For instance, if you run ls -l, thsh should print "RUNNING: [ls]"
When command ends you should say something following the pattern "ENDED: [ls] (ret=0)" and show the return value from run_command
add anything else to the debugging output (be creative)

Redirection Support

One of the most powerful features of a Unix-like shell is the ability to compose a series of simple applications into a more complex workflow. The key feature that enables this composition is output redirection.

Redirection is accomplished by three special characters '<', '>', and '|'. You will need to add logic to your parsing code which identifies these characters and uses them to identify shell-level directives, rather than simply passing them to exec().

The first two characters can direct input from a file into a program, and and output from a program, respectively.

In the example above, the standard output of ls -l is directed to a file, named newfile. If this file didn't exist previously, the shell created it. Note that the ls program does not know it is writing to a file, and is not passed the string '>newfile' as an argument. Similarly, the contents of newfile are passed to the cat program as its standard input.

You'll have to learn how to manipulate file descriptors carefully using system calls such as open, close, read/write, dup/dup2, and more.

In this example, my shell creates three child processes. The first reads the contents of my home directory and outputs them to the grep program, which searches for the string '.txt'. The output of grep, i.e., all files with the .txt extension, is then sent to the wc program, which counts how many lines of input it is given (i.e., the number of .txt files in my home directory.

Exercise 5. (7 points.) Add support for all three forms of redirection described above.
Be sure to run several test cases for piping applications together, and ensure that termination is handled cleanly.

Challenge! (3 bonus points) Add support for assigning inputs to arbitrary file handles other than stdin and stdout. The syntax for specifying the file handle is to put an integer in front of the redirection operator to indicate another handle, such as stderr. For example:

[/home/porter] thsh> somecommand 2>err.log

This example runs "somecommand" and redirects its stderr to "err.log". Note that you should handle multi-digit integers.

Scripting Support

Most shells can be run interactively as well as non-interactively. In non-interactive mode, you can put the shell commands in a plain file---essentially creating a program of shell commands (called a shell script). For example, if I put this in a file called "foo.sh":

Then I can use this file (or program) to have the shell run these commands sequentially as follows:

In other words, thsh will identify the string 'foo.sh' on its own command line and then interpret these commands as a batch. In a batch, the first line runs to completion, then the second, and so forth. These commands do not need to run in parallel, except for pipes on the same line (described below).

One can also make the shell script, executable, and then run it directly like any other program. For that, I need the file to start with a special character sequence called a 'shebang' followed by the path of the shell

Note that thsh must be in your PATH for the shebang above to work, otherwise, you should use an absolute path, like /home/porter/thsh/thsh.

When in scripting mode, unlike interactive mode (the default you have implemented thus far), thsh does not need to print the prompt or commands within the script - only the output from the commands in the script.

Exercise 6. (3 points.) Add support for thsh to run non-interactively: this boils down to basically supporting an optional input file argument. If 'testscript' is a shell script, the following examples should work, where '$' indicates your default shell (e.g., bash).

$ ./thsh testscript

[/home/porter] thsh> chmod u+x testscript
[/home/porter] thsh> ./testscript

You will also have to support a comment character '#' so if you see a line starting with '#' in the script, you should ignore it. The comment command could also be run non-interactively and "do nothing".

[/home/porter] thsh> #this is some text
[/home/porter] thsh>

Challenge: Job Control Support

Another useful feature of a shell is the ability to pause and resume execution of a job. Here, we define a job as a single command, which can be either a single process, or multiple processes in a pipeline. For instance, ps -eaf | grep foo would be a single job.

In the case of a long-running program, it is helpful to be able to place it in the "background"---allowing the user to issue more commands interactively while the long-running program continues execution.

Your shell should identify the special character '&', which means that a program should be executed in the background, returning a shell prompt immediately. The built-in command jobs should list all background running jobs, their name, PID, job number, etc. just like bash with their status (running or suspended). It should also print the exit status code of background jobs that just ended.

In addition to jobs, we will need to add a few more built-in commands to make job control useful. The command fg 3 should make job number in your list to go to the foreground (and resumed execution if it is not running/stopped). The command bg 2 should cause suspended program 2 to run in the background.

Finally, we need to be able to forcibly pause or terminate a program. If you type Ctrl+C: the foreground program(s) should be killed. If you hit Ctrl+Z: the foreground program(s) should be suspended and added to the list of jobs (i.e., you send it a SIGTSTP signal to suspend it; fg sends it a SIGCONT to resume running).

Fun

Exercise 7. (1 point.) Create a built-in command, or a separate program, called goheels that draws a Tar Heel on the console using ASCII art. You are welcome to use an ASCII art generator, or draw your own by hand

Contests

In order to encourage creativity and a bit of friendly competition, the instructor and TAs will judge a few contests. The prizes will be bonus points. Only teams that complete all exercises will be eligible to win.

Winners will be announced in class after the grading of lab 2 is complete. More than the points, of course, is the pride of winning.

Late submissions that are handed in after the judging has started (probably a few days to a week after the deadline), will not be included.

Challenge! (3 bonus points.) Add support for tracking the history of a user, including saving the history to a file. Support the up and down keys to cycle through history, and add a built-in command history that dumps the entire history to the console. Also, add a built-in command, clear to reset the history.

For this challenge problem, it is fine to set a compile-time history length, such as 50 lines.

In order for my history to survive after the shell exits, most shells will write a file in the user's home directory, such as /home/porter/.thsh_history. For full credit, your shell should persistently store the user's command history. You can use environment variables to figure out where the user's home directory is.

Style and More

It should not be possible for a user of your shell to ever make your shell crash or hang. If your shell has some limitation (e.g., the command line is limited to being x characters or less), you must detect when the limitation is reached and take an appropriate action (e.g., output a meaningful error message).

Along the lines of the previous point, any error conditions generated by fork or exec (or any other system calls you make) should be processed by your program and should result in the generation and output of an appropriate error message.

To be sure your code is very clean, it must compile with make without any errors or warnings!

If the various sources you use require common definitions, then do not duplicate the definitions. Make use of C's code-sharing facilities.

Challenge! (5 bonus points) Support time counting. If you start thsh with -t, it should count how long each program ran and print stats when the program ends. The output should be something like:

$ thsh -t
thsh> du -sh /usr
4.3MB /usr
TIMES: real=23.7s user=12.1s sys=7.0s

Be sure to note this in challenge.txt if you do this.

Challenge! (5 bonus points) Support file "globbing" for extensions, such as

thsh> ls *.jpg

The above should print all the file names that end with ".jpg". Only support *.[EXTENSION]. That is, you'll need to check to see if an argument starts with an '*', then use readdir(2) and getdents(3) as needed to read all files from the current directory, match them -- using strstr(3) -- and add them to list of args you pass to exec(2). In other words, your shell will be exec-ing a command that'll be as if you typed the full names of all the files on the command line one by one.
Be sure to note this in challenge.txt if you do this.

Challenge! (5 bonus points) Add support for "tab completion" in your shell. If a user types a prefix of a command and then hits the "Tab" key twice, the shell should show all possible commands that match the prefix. If only one command is possible, the shell should automatically fill in the rest of the command. If all possible commands share subsequent letters, automatically fill in letters until the commands diverge.
Hint: Consider using a trie data structure to organize the available commands.

Hand-In Procedure

You will be handing in the code via gradescope. You should have been added to the class; if not, please contact the instructors as soon as possible. If you work in a team, you should only submit one copy of your code in gradescope; you can add your teammates to the handin. We recommend handing in directly from your github repository to the assignment. Gradescope will run the autograding program, giving you immediate feedback on the assignment. You may hand in more than once and we will take the most recent, applying lateness penalties as appropriate (out-of-band).

In the event of any discrepancies with the autograder environment, please contact course staff as soon as possible.

Generally, unless the homework assignment specifies otherwise, you should compile your program using the provided Makefile (e.g., by just typing make on the console). Do not add any special command line arguments ("flags") or compiler options to the Makefile.

Note: We do not have an automated way to calculate late penalties. These will be applied manually at the end of the semester.

The program should be neatly formatted (i.e., easy to read) and well-documented. The Style Guide gives additional guidance on lab code style.

If you complete any challenge problems, please describe the solution and how to demonstrate it in challenge.txt. Note that we have a separate submission option in gradescope for submitting challenge problems, in order to accommodate later submissions without charging late hours. Even if you are submitting on time, please submit your challenge problems a second time through the appropriate challenge assignment --- we will only manually grade assignments handed in through the challenge option.

Acknowledgments

The current lab includes helpful contributions from Kevin Jeffay, Jacob Fisher, and Yicheng Wang.