CMSC 421, Principles of Operating Systems.

Computer Science & Electrical Engineering Department

University of Maryland Baltimore County

Project 0.

100 points. [Keycode: PRJ0].

Due by 11:59pm on February 8, 2009.

Introduction

This semester, you will make changes to the the linux kernel. To do that, you must be familiar with compiling the kernel, creating updates reflecting your changes, and submitting them for grading. You will repeat these operations many times for every subsequent project this semester. Project 0 is a very simple project designed to familiarize you with various pieces of linux and to instruct you in these procedures.

Project 0 consists of three parts with the following objectives:

• Becoming familiar with basic tools in building a custom Linux Kernel.
• Installing Linux and building a custom kernel.
• Creating kernel updates.
• Creating a linux kernel patch file.
• How to properly format and submit your linux kernel project submissions.

Hardware and Software

Students are expected to use virtualization to complete their projects. Virtualization allows us to run a virtual machine ("the guest") on a real physical machine ("the host"). For the purposes of the class projects, we assume that the host machine is a PC or Mac running a version of VMWare compatible with VMWare Player. For details on H/W and S/W requirements refer to the VMware home page.

There are many linux distributions and kernel releases. This course expects students to use the Ubuntu 8.04.2 i386 and linux kernel revision 2.6.28.2.

You should buy 1 or more external USB drives to work in the lab and for backups. The drive should have at least 10GB of disk space. Flash drives or USB-powered hard disks are the most convenient option. Make sure it is USB 2.0. The cheapest option is to buy a hard drive enclosure and put a hard drive in it.

MAKE REGULAR BACKUPS. Every semester a small percentage of students experience drive failure. You will not receive an extension if your hard drive fails or the data on it becomes corrupt. At a minimum, Every time you finish making changes for the day you should backup to your computer or another disk drive.

N.B. Format your drive NTFS or ext2. FAT32 has a file size limitation that you may encounter.

Download Locations

• VMWare
• Ubuntu 8.04.2 release
• Linux Kernel
• 421VM.zip
• ignorelist

N.B. The files you need and their signatures will be placed on blackboard under "Downloads" for your convenience.

Also, at the time of this writing there is no free version of VMware for Mac users. You can install windows, use the lab, or use another virtualization solution (VirtualBox, Parallels, etc). Be aware that using another virtualization software package should be fine for changing the kernel, but it will have an effect on what you turn in for this project and it is not supported by the TAs. Always test your projects in the lab before submission.

Part 1: Installation

1. Download the Ubuntu ISO, and 421VM.zip.
2. Unzip 421VM.zip in a folder called "421VM". Hereafter we refer to the virtual machine in the 421VM directory as 421VM.
3. Place the Ubuntu ISO in 421VM.
4. Open the configuration file, 421VM.vmx, and change it so that the virtual cdrom points to the ubuntu ISO file.*
5. Boot the 421VM by using VMWare to open the 421VM.vmx file in your host machine. (You may receive a prompt asking if you have moved or copied the virtual image. If so, say that you have "copied" the image.)
6. Install Ubuntu Linux.*
7. Ensure that you have network access from inside the VM. One common problem is that vmware switches the vNIC to "vlance" when it should be set to "e1000" in the configuration file. Installations of AOL and many personal firewall products will also often break network connectivity. The VMware forums are the best resource to resolve network connectivity problems. The TAs will help you, but be prepared to intelligently answer the question "What have you already tried?".*
8. Install all system updates from the internet using the package manager.*
9. Shut down the system, and remove your changes to the 421VM.vmx file, switching back to "cdrom-raw".
10. Open VMWare and Start 421VM. At this point you should have a properly working installation of Ubuntu linux.

Part 2: Building a custom kernel

You should access these instructions from inside your VM. The next objective is to get comfortable building and installing a custom linux kernel.

         sudo sh
         ... (commands to execute as root)
         exit
         
         --OR--
         
         sudo -s
         ... (commands to execute as root)
         exit
         
         --OR--
         
         sudo (commands to execute as root)
         

1. Install software packages used for Kernel development.*
   
   1. Open a terminal window.
   2. Enable all additional package repositories by uncommenting the deb lines in the /etc/apt/sources.list file as root (you can use nano, vi, or gedit).
   3. Upgrade and Install some needed packages using apt-get as root. NOTE: these steps can take some time (15 minutes or more).

      	 sudo apt-get update
               sudo apt-get upgrade
               sudo apt-get install build-essential bin86 kernel-package libqt3-headers libqt3-mt-dev
               

2. Download the linux kernel sources to /usr/src/
   1. Commands to do this are:

      	    cd /usr/src/
                  sudo chmod 775 .
                  sudo adduser `whoami` src
                  sudo wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.28.2.tar.bz2
                  md5sum linux-2.6.28.2.tar.bz2
                  

   2. Verify the md5sum or sha1sum with the one posted on kernel.org or blackboard.
   3. At this time, you should log out and log back in (to refresh permissions).
   4. After logging back in, unzip the kernel and make two copies called "pristine" and "project0":

      	    tar -xjvf linux-2.6.28.2.tar.bz2
                  cp -ra linux-2.6.28.2 pristine-linux
                  cp -ra linux-2.6.28.2 project0
                  

   5. We will refer to the sources in the directory "/usr/src/project0" as $prj. We will refer to the sources in the directory "/usr/src/pristine-linux" as the $pristine. Never make any changes in the pristine kernel sources. Also, when you see $val or <val>, these indicate values that you should change.
   6. Create a symbolic link to $prj:

                  ln -s project0 linux
                  

3. Build and install your first custom kernel.
   1. It is highly recommended to read the Kernel Rebuild Guide very carefully before you proceed.
      You may also want to review LKN (Linux Kernel in a Nutshell).
   2. You will make your changes to $prj.
   3. Name the version of your custom kernel as linux-2.6.28.2-GLUSERNAME-FIRSTNAME-cs421project0, where GLUSERNAME is your UMBC GL username, and FIRSTNAME is your first name.
      Look at "Assign Unique Name" in the Kernel Rebuild Guide.
   4. Configure and compile the custom kernel and modules. This step may take quite some time. (NOTE: "V=1" is a verbosity setting for make)

                  cd /usr/src/linux
                  make V=1 mrproper 
                  cp /boot/config-<some revision #>-generic /usr/src/linux/.config
                  make V=1 xconfig
                  make V=1 bzImage
                  make V=1 modules
                  ...
                  

      The linux directory in the Kernel Rebuild Guide corresponds to the experimental kernel build directory.
   5. Install the new kernel modules as root

                  cd /usr/src/linux
                  sudo make V=1 modules_install
                  

   6. Prepare an initial RAMdisk for booting the new kernel

                  sudo update-initramfs -c -k 2.6.28.2-GLUSERNAME-FIRSTNAME-cs421project0
                  

   7. Install the new kernel as root

                  sudo -s
                  cp /usr/src/linux/arch/i386/boot/bzImage /boot/bzImage-2.6.28.2-GLUSERNAME-FIRSTNAME-cs421project0
                  cp /usr/src/linux/System.map /boot/System.map-2.6.28.2-GLUSERNAME-FIRSTNAME-cs421project0
                  ln -s /boot/System.map-2.6.28.2-GLUSERNAME-FIRSTNAME-cs421project0 /boot/System.map
                  

   8. Update GRUB so that the 421VM can boot from the custom kernel by appending, as root, the following lines at the end of the /boot/grub/menu.lst file.

                  title Project 0 421 Kernel (2.6.28.2)
                  root (hd0,0)
                  kernel /boot/bzImage-2.6.28.2-GLUSERNAME-FIRSTNAME-cs421project0 root=UUID=(copy from another entry) ro quiet splash
                  initrd /boot/initrd.img-2.6.28.2-GLUSERNAME-FIRSTNAME-cs421project0
                  quiet
                  

   9. Make sure that 421VM boots properly with your new kernel.
   10. Boot the 421VM with your kernel. Run uname and capture its output.

                   cd /usr/src/
                   mkdir project0-submission
                   cd project0-submission
                   script
                   uname -a
                   exit
                   cat typescript
                   

   11. Check to make sure your output shows GLUSERNAME-FIRSTNAME-cs421project0 in the uname output.

Part 3: Creating and applying Kernel updates/patches

The second objective is to learn using diff to generate kernel source patches (updates) and applying the patches to the original kernel. Since many modifications to the kernel are small (compared to the total kernel source size), such updates are usually distributed as "patches". If you make minor modifications to the kernel (e.g. for CMSC421 projects) you will want to create a patch. Remember that you will always submit a "patchfile" for every Kernel project and we will apply your patch to our copy of the pristine kernel sources when grading your project. A bad patch may result in 0 points!

A good tutorial on how to create patchfiles using diff can be found at http://www.gnu.org/software/diffutils/diffutils.html. The typical call is "diff [<options>] <from-file> <to-file>"

You should have already created the pristine kernel sources directory $pristine as described earlier. You should have already built your first custom kernel as specified in Part 1, by making appropriate changes to the $prj.

1. Download the course kernel file ignorelist. This will permit you to create patches without first running "make mrproper" in the kernel sources directory.

            cd /usr/src/
            wget /project0/ignorelist.txt
         

2. Create a patchfile for your kernel using the diff command.

         cd /usr/src/
         diff -rcP -X ignorelist.txt pristine-linux linux > project0-submission/patchprj0.diff
         

   The file patchfile patchprj0.diff contains all the information the patch command needs to transform the pristine kernel sources to your experimental kernel sources.

   IMPORTANT: If the ignorelist does not work then you should execute "make mrproper" in $prj directory before creating the patchfile. If you forget to do this, you may get a very large patchfile.

3. Verify your patch using the patch command.
   You should always validate your patchfiles before submitting them. Make a new full copy of your pristine kernel sources, apply your patch to it, and compare with $prj

         rm -rf verify-linux
         cp -ra pristine-linux verify-linux
         cd verify-linux
         patch -p1 < ../project0-submission/patchprj0.diff
         cd ..
         diff -r -X ignorelist.txt linux verify-linux
         

   The sources in the verify-linux directory should now be transformed into your experimental sources. The diff command above should not show any differences. For project0, if you see several "Only in project0/...", this is also OK.
4. Examine your patchfile, note the format and that your changes are included in the file.
5. Note that all patches you submit must work with the "-p1" option. You can guarantee this by always using diff as shown above. Also, do not edit patch files once you have created them! This will break the patch command.

Part 4: Project Report and Readme

Your project report and readme are due for each project you submit. If you do not submit a project report and a readme, your project will not be graded.

Your README file is located in the base directory of your submission archive and is named "README". It is a text file that explains how to compile and run your project, and gives a brief explanation of problems.

The basic Readme report should follow the following format:

		Firstname Lastname
		glusername
		Date
		Project Keycode
		
		==Manifest==
		
		There are X applications included with the project:
		
		Part 1: part1/
				part1/file1.c
				part2/file2.sh
				...
		Part 2: part2/
				part2/somefile.txt
		...
		
		==Compiling==
		
		Part 1: simply run "make"
		Part 2: Is a kernel module, you can make it by doing XYZ, here's a stepped list:
		... 
		
		==Running==
		
		Part 1: Requires 3 options, which must each be within the following range...
		...
		
		==Bugs==
		
		Part 1: 
			-Should accept a range of 0-10, but it's actually 0-9.
			-Always segfaults at the end, don't know why.
		Part 2:
		...

You may, alternatively, place a README with a manifest in the base directory and give each part it's own readme file following the above format. You are also welcome to format your README differently provided you still provide the required information.

The Project Report is a brief document named "report.pdf" in the base directory of your submission archive. It explains your rationale and approach to the project and must answer all project questions. Reports should be formatted as follows:

1. Introduction: A title, Name/section/glusername/keycode at the top, and a description of how you broke down the rest of the document along with a quick acknowledgement of how far you got.
2. Design: What was your approach on this project? This is usually broken down by each part of the project and combined with the problems section.
3. Problems: What problems did you come across, and what did you try to do to fix them.
4. Answers: responses to project questions, if any.
5. Feedback: Your thoughts on the project, if any. The TAs will read these and provide summaries to your instructors.

We expect your reports to be brief and to the point. They should not be filled with prose. Use lists, diagrams, and written pictures where possible. The report file is used to determine how much partial credit you will receive for an unworking portion of your project. The best strategy is to keep a notebook or write an outline for the report when you get started, and to update it as you go through the project.

The report file must be a PDF. You can create PDFs in a myriad of ways, and if you need to, feel free to ask on the blackboard discussion board.

Part 5: Project Questions

1. Using lshw, lspci, lsusb, the proc filesystem, and other commands, find all the hardware devices connected to your VM and list their names.
2. Knowing what you found out above, go back and reconfigure your kernel so that only the actual devices on the system are compiled into the kernel. Recompile and verify that your program works. Name the new .config file "smallconfig" and include it in your submission. If you submit the smallest config in your section, you will receive extra credit.
3. How much disk space is available on your system?
4. What are all the mount points on your system?
5. Some of the mounts are not physical devices, what are they? Explain how each one works in detail.
6. Explain what the UUID on your root path means in the menu.lst file. If you could not use UUID, what would the root parameter be? (Hint: in order to get the smallest config possible, you will need to do this).
7. Using the proc filesystem, how much ram do you have?
8. Using the proc filesystem, what is the speed of your processor?

What to submit

1. patchprj0.diff
2. the /usr/src/linux/.config file
3. the menu.lst file from /boot/grub
4. sample output from the "uname -a" command run in your system booted with your custom kernel.
5. All other files needed to answer the questions from part 5.

Any of the following will cause significant point loss. Any item below with a "*" will give that submission a 0.

1. Missing patchprj0.diff.*
2. Submissions in excess of 10MB.*
3. Missing any of the required files above.
4. A patch file not named "patchprj0.diff" (case sensitive).
5. Submissions that do not extract with the command "tar -xjvf filename.tar.bz2".
6. Submissions that create a directory and put all their files in it (e.g. patchprj0.diff is not in the base directory).

         cd /usr/src/
         mkdir project0-submission
         cd project0-submission
         ... copy files to appropriate locations ...
         cd /usr/src/project0-submission
         tar -cjvf project0.tar.bz2 *
         scp project0.tar.bz2 yourusername@gl.umbc.edu:~
         ...
      

Read the Submit FAQ:

 
      submit cs421 PRJ0 <files>
      submitls cs421 PRJ0
      

Hints

1. You may find the /boot/config-<revision>-generic configuration file useful when building custom kernels for the course.