Principles of Operating Systems

CMSC 421 — Fall 2020

Project 0

Due by 11:59 PM EST on Sunday, February 16th

Changelog

February 1, 2020: Fixed copy/paste error in version number.

January 31, 2020 @ 7:50 PM: Fixed typo in step 4F of building the kernel. Thanks to Sir Percival on discord for pointing it out.

January 31, 2020: Initial Version

Introduction/Objectives

All of the programming projects in this course will require making changes to the Linux kernel. In addition, all projects will be performed inside of a Virtual Machine for various reasons (uniformity of results, for instance). In order to do the later projects in this course, you must become familiar with the prospects of compiling the kernel inside the VM and creating updates to submit. Every project this semester will involve these steps, and this project is no exception. Project 0 is a very simple project that is designed to familiarize you with the procedures of installing and updating the kernel inside a VM.

In this project, you will build a custom version of the Linux kernel with a modified version string as well as an additional system call. You will also utilize the Git version control system for managing the source code of the project.

This project is divided into several parts that will help you perform the following objectives:

Software and Hardware Requirements

Linux on VirtualBox

In order to aid development of the projects in this course, we will be running the projects under virtualization. Virtualization allows us to run a Virtual Machine on an actual physical machine. In this way, we can run a second "guest" operating system inside the regular "host" operating system. To do the projects in this class, we assume you will have access to a relatively modern PC that can run the VirtualBox VM. VirtualBox requires an x86 CPU with a decent amount of RAM. In addition, it would be advantageous if the CPU on your host machine supported the x86 Virtualization extensions (VT-x (for Intel processors) or AMD-V (for AMD processors)). For more information about hardware and software requirements for VirtualBox, please consult the VirtualBox website. All projects are expected to run in 64-bit mode on VirtualBox.

VirtualBox is available on the machines in the ITE 240 lab running Linux.

For the purposes of projects in this course, we will be using the amd64 (64-bit) version of the Debian 10.2 Linux distribution. In addition, the custom kernels that will be built in this course will be based on the Linux Kernel version 5.5.

Hard Drives

You must have an external USB hard drive or flash drive if you wish to work in the lab. Even if you do not intend to work in the lab, it would be very useful to have an external storage device for storing backups of your VM image. You will need to have more than 32GiB of free space on the device to store one copy of your VM image on it. Also, You will need to format the drive as some type of filesystem that supports files of size > 4GiB. Many flash drives come pre-formatted as FAT32, which does not support files with a size ≥ 4GiB. FAT32 will not work for this reason! You should format your drive as either NTFS (if you will be using Windows with the drive as well) or a Linux filesystem such as ext3 or ext4.

Even if you do not wish to work in the lab, you will still want to make regular backups of your VM images. It is recommended to do so on an external hard drive or flash drive, since your VM images will be relatively large in size and it is always a good idea to store your backups in a different place than where you are storing the data being backed up.


Installing Linux

Perform the following tasks to create the environment that will be used to complete the projects in this course:

  1. Install VirtualBox 6.1 on your computer. If you are installing on a Linux host OS, please install VirtualBox by way of your package manager.
  2. Create a Virtual Machine and install Debian into it (see below for some tips while building the VM).

When creating your VM, you should select that you will be running the 64-bit version of Debian Linux in the new VM wizard. Also, when creating your VM, you should increase the disk size from the default to at least 32 GiB. This should give you enough space to build the kernel properly, assuming you don't install a bunch of software within the VM. You should also be sure to give your VM enough RAM to work with while not hampering the rest of your PC. 1GiB of RAM should be plenty, but be sure to not give your VM more than 1/4th the total amount of RAM you have installed in your computer. You may also find it useful to increase the number of cores that are devoted to your VM in the virtual machine configuration. If you do so, you generally should not increase it to any more than one less than the total number of physical cores you have in your PC (Do not double the number of cores if you are using a HyperThreaded system!). To do this, after you create the VM, select it in the main screen of VirtualBox, hit the Settings button, then in that window select System, go to the Processor tab and increase the slider labeled Processor(s).

You should get the Debian network install CD image from this link to ensure that you will be installing the correct version of Debian. If the previous link does not work, you should try this link instead (if the first link is working, this one will probably not work).

When installing Debian within the VM, you can use either the normal install or graphical install options -- either is fine. You must be sure to use the 64-bit installation, however (which will be the case if you downloaded the iso from the link in the last paragraph). When you get to the portion about partitioning, you should select the guided partition option and the option to create one large partition on the disk (having a small swap partition is fine as well). Finally, when installing packages later in the installation, be sure to select at least one graphical environment from the list if you want a GUI. I personally suggest using Cinnamon or MATE, as they are both relatively lightwieght and will ensure you have ample space to work with later on your virtual hard drive. In that same step, you should also unselect the print server option and select the SSH server option. Finally, late in the installation, the installer may ask you where to install the boot loader. Be sure to select to install it on the VM's hard drive (/dev/sda).

For advanced users only: It is possible to do all of the assignments for this class without installing a GUI in your VM, however some of the setup can be a little bit wonky (like getting your SSH key added to your GitHub account). If you wish to do this, then during the installation unselect the "Debian desktop environment" during setup and do not install any of the UI options after it (still make sure to enable the "SSH server" option, though). Later, in the instructions when installing packages, leave out the qt5-default, qtcreator, and gedit packages. Finally, when building the kernel later on, use make menuconfig instead of make xconfig. As mentioned, when doing the SSH key addition to your GitHub account, you will need to find another way to import it, as you will not be able to copy/paste in the terminal. I find it easiest to use scp/sftp to copy it off of the VM to a place easier to access. If you are not comfortable with the Linux terminal, I highly suggest that you just install a GUI environment and ignore this paragraph entirely.

Creating an SSH key

In order to work on repositories hosted on GitHub, you will need to have an ssh key set up on your VM and registered on your GitHub account. To do so, run the following command in a terminal in your VM (replacing the yourumbcemail part as appropriate): 

ssh-keygen -t rsa -b 4096 -C "yourumbcemail@umbc.edu"

While running this command, you can accept the defaults for just about everything. However, for security, you should provide a passphrase for the key when it asks. This passphrase can be just about anything, but you must make sure to not forget it. You will need your passphrase any time you access resources in your GitHub account via the git command line tool. You may opt not to use a passphrase for your key as well, but as mentioned above a passphrase is helpful for security. Do not change the filename where the key is stored from the default location!

Now that you've created your key, you must associate that key with your GitHub account. To do so, open up the web browser (it may be either Iceweasel or Firefox, depending on what options you installed at setup time) inside your VM, and go to github.com. Upon logging in with your GitHub credentials, visit the SSH settings section of your GitHub user settings. Use the "New SSH key" option, give the key a title ("421 VM Key" works nicely, if you can't think of anything better), and then run the following command in a terminal and carefully copy/paste the output into the Key box on the page, and hit the Add SSH key button. 

cat ~/.ssh/id_rsa.pub

If you run into issues using your SSH key after adding it to your GitHub account properly, you may need to add it to the SSH agent. Follow the instructions on this GitHub documentation page to do so.


Obtaining root Privileges

Many of the commands that you will be running on the Linux installation within the VM will require root privileges. There are a variety of methods to elevate your user privileges on Linux. From a regular terminal, you can use su to do so: 

su -
(enter your root password when prompted)
(perform any commands to execute as root)
exit

Once you have installed the initial set of packages listed in the instructions below, you will also be able to use the sudo program to run higher privileged commands like this (once again, assuming the command to be run is command): 

sudo command
(enter your regular user password when prompted)

Additionally, sudo can also be used in the same manner as the su - command, by using the -s flag, like this: 

sudo -s
(enter your regular user password when prompted)
(run any commands you should be running as root)
exit

It is very important that you pay careful attention to what steps should be run as root. In these instructions, we specifically mention which steps should be run as root. Anything else should not be run in a root shell, and should be run as your normal user account. If you fail to do so, you will have problems with your build.


Building a Custom Linux Kernel

The next part of this project is to obtain the Linux kernel sources and build a customized kernel. The following set of steps will guide you through the process. You should make sure to read all of these instructions before you start working on any of them!

  1. Update all packages on your installation.
    In a terminal, as root, run the following commands, substituting the username you log into the VM with in the second-to-last command as appropriate: 
    apt-get update
apt-get upgrade
apt-get dist-upgrade
apt-get install sudo
usermod --append --groups sudo YourVMUsername
reboot
    Wait for the VM to reboot itself. After the VM reboots, you should be able to use sudo for running commands as root. You will need to type sudo in front of each command if you are running multiple commands at a time as root.
  2. Install the required software packages used for kernel development.
    1. In a terminal, as root, install the required packages with the following command: 
      apt-get install build-essential qt5-default qtcreator valgrind nano patch diffutils curl fakeroot git pkg-config gedit libssl-dev libncurses5-dev libelf-dev bison flex bc dkms rsync
  3. Obtain the Linux Kernel sources and unpack them into the appropriate directory.
    1. In a terminal, as root, run the following commands: 
      cd /usr/src
chmod 777 .
    2. In a terminal, run the following commands as your normal user (not as root). Replace yourusername with your github username, yourumbcemail with your UMBC email account, and make sure to put your full name where suggested. 
      cd /usr/src
git config --global user.email yourumbcemail@umbc.edu
git config --global user.name "Your Full Name"
git clone git@github.com:umbc-cmsc421-sp2020/project0-yourusername vanilla-project0
cp -ra vanilla-project0 project0
    3. For the rest of this project, the sources in the directory /usr/src/project0 will be referred to as the working copy of the kernel. The sources in the directory /usr/src/vanilla-project0 will be referred to as the vanilla copy of the kernel. Never make any modifications to the vanilla copy of the kernel! They are kept here in case you mess something up!.
    4. If the git clone command returns an error message about the repository not existing or you not having permission to access it, then make sure that the repository has been created (that is to say that you clicked on the link on the project assignment on Piazza and that your repository has correctly populated), that you have correctly created and added your SSH key to GitHub, and that you are not trying to run the command as root.
  4. Build and install your customized kernel.
    1. Read the Compiling a Kernel chapter of the Debian Administrator's Handbook. The instructions on this page are a customized/condensed version of part of what is covered on that page. There may be interesting tidbits in that chapter that aren't covered herein.
    2. You may wish to read the Kernel Rebuild Guide if you have never built your own Linux kernel before to get more context on the subject. We will be using Linux 5.5 for this project, which is handled very similarly to Linux 2.6.x. You should disregard the building instructions in the Kernel Rebuild Guide, as you will be following the instructions in this project description instead, which is based in part on the instructions in the Debian Administrator's Handbook!
    3. You may wish to review Linux Kernel in a Nutshell as well.
    4. Give your working copy of the kernel a unique version string of the form 5.5.0-cmsc421project0-USERNAME . Where USERNAME is your UMBC username (your UMBC email username -- NOT your GitHub username, if they are different). This is to ensure that our customized kernel does not interfere with the modules or other files of any kernels installed by the Debian package manager. If you have followed all of the instructions up to this point, you should have no trouble doing this. You must change at least one file in the kernel source code to do this for this project! You may not use the LOCALVERSION environment variable or configuration parameter to complete this assignment.
      As an example, if I were doing the project, this would be what should be printed out by uname -a (the hostname on my VM is ryuujou-vm):
      Linux ryuujou-vm 5.5.0-cmsc421project0-lsebald1+ #1 SMP Fri Jan 23 13:37:01 EST 2020 x86_64 GNU/Linux
      Your version string must be all lowercase, otherwise you will not be able to build the kernel properly for Debian. If you have compiled the kernel multiple times, the #1 may change in the command (and the date will, of course, update).
    5. Add a "Hello World" system call to the kernel, performing the system call implementation and kernel modification steps in this tutorial. You should be sure to run the test program and verify that the system call actually was able to run successfully only after you build, install, and reboot into your new kernel.
    6. Configure and compile the custom kernel and its modules. This step will likely take quite a while. Run the following commands in a terminal (make the change mentioned below during the make xconfig step and be sure to save the configuration file to its default filename): 
      cd /usr/src/project0
make mrproper
make xconfig
make localmodconfig
make deb-pkg
      During the make xconfig step, you should go into the Library routines section of the configuration and make the "CRC32c CRC algorithm" or "CRC32c (Castagnoli, et all) Cyclic Redundancy-Check" option be built-in to the kernel instead of being built as a module (make it so it has a checkmark, not a dot in the box next to it). Once you've done that, hit the floppy disk icon to save the configuration. If the make localmodconfig step prompts you at any point, you can just hit Enter to accept the defaults. If you have increased the number of cores given to your VM earlier in the setup, you may wish to give the -j flag to the make deb-pkg step to allow it to use multiple cores. If you've dedicated 2 cores to your VM, you'd run that command as make -j2 deb-pkg instead.
    7. Now that the kernel has been built, you must install the kernel image. To do so, run the following command, as root from the /usr/src directory (substituting your UMBC username as appropriate, of course): 
      dpkg -i linux-image-5.5.0-cmsc421project0-USERNAME+_5.5.0-cmsc421project0-USERNAME+-1_amd64.deb
      If this exact command (after substituting in your username in both places) does not work, you are either not in the correct directory, do not have root privileges, had a compiler error while building the kernel, or have not changed the version string correctly. Please re-check your work and ensure that you have made the required changes to the kernel source code. If you have built the kernel multiple times, the +-1 at the end may increment. If this is the case, you should always be sure to install the newest kernel image (it should have the highest number). In certain situations, you may find that you need a kernel-headers package to be installed as well. The build commands included earlier should produce one for you. Most of the time this will not be necessary at all and you can safely ignore the kernel headers package. Also, please note that the packages that include -dbg in the filename are not likely to be needed generally, either.
    8. Your working copy of the kernel should now be built and installed. In addition, the install process should have updated your bootloader configuration to add the new entry to it (and made it the default kernel option too).
    9. Reboot your VM, and select your customized kernel from the GRUB boot menu
    10. Make sure your custom kernel boots properly in the VM. Run the following command to check the customized kernel: uname -a
    11. Verify that the output from the uname command shows the correct version string, as specified earlier in the project.
    12. If your the output from the last command does not match the version string format provided, try again. Note that this will possibly change the #1 in the version string (right before the date that the kernel was compiled on). This is not important and will not affect your grade in any way. Generally, when rebuilding the kernel after the initial build for each project, you can just run make bindeb-pkg (with a -j option if you have set up your VM with multiple cores) — not running any of the other make steps up above (like make xconfig) and then install it with dpkg as root). You should not have to re-run make mrproper after you have run it once for this assignment until you are cleaning up afterwards, unless you are told otherwise by an instructor or TA!
    13. Verify that the Hello World system call can be called correctly from userspace by performing the steps in the "Testing Your New System Call" section of the Hello World system call tutorial. If this does not work properly, ensure that you have followed all steps in the tutorial and the compilation steps of this assignment. Also, be sure that you didn't make any mistakes in creating the test program.

Submission Instructions

You have already initialized and created your git repository early in this project (when you ran the git clone command earlier). Now it is only a matter of updating the copy of the git repository on your local VM and pushing the update up to the GitHub server.

First, ensure that your changes are all reflected in the local git repository. Run the following command in a terminal in your /usr/src/project0 directory and ensure that any files you have modified within the kernel sources are included in the section labeled "Changes not staged for commit": git status.

Next, you should ensure that the changes you made are accurately reflected in the modified data. To do so, run the command git diff and make sure that every change you made to the kernel source code is shown. You should be able to scroll back and forth through the diff output with the arrow keys on your keyboard, if needed.

Now that you have ensured that all the changes you made are reflected in the git repository's status, you must add the changes to your local repository and commit the changes. Do so with the following commands (replace FilesYouModifiedOrAdded with the filename of any files/directories you modified or created including the hello/ directory and the hello.c and Makefile inside it — the git status command should help you here): 

git status
git add FilesYouModifiedOrAdded
git commit
(Give a *good* commit message here describing your changes and exit the editor)

You do not need to add the test program described in the hello world system call tutorial to your repository. If you followed the directions in that tutorial, the test program should be in your home directory and not in the kernel source tree, thus it should not show up in the git status command above.

Now that you have committed your changes to your local git repository, you may wish to ensure that the commit has happened properly. To do so, run the git log command and make sure that your commit shows up at the top of the list.

Now that your commit is in your local repository, it is time to push the changes up to your GitHub account. To do so, run the git push origin master command. You should now be able to open your account on the GitHub website and see that your changes are pushed to the repository.

Please fill out this form to inform the TAs of your GitHub username. If you do not, we may not be able to grade your assignments, as we will not know which assignment belongs to whom.

Congratulations! You have completed Project 0!


Cleaning up...

After submitting your project, you should clean up your VM to ensure that you do not run out of space later on. To do so, run the following commands in a terminal. 

cd /usr/src/project0
make mrproper
cd /usr/src
rm -rf vanilla-project0

What to do if you want to lose points on this project:

Please do not make us take off points for any of these things!

Last modified Saturday, 01-Feb-2020 15:17:07 EST