CSCI 4113 - Unix System Administration - Spring 2005

Time: Tue, Thu - 5:00 - 6:15 PM
Location: HALE 270
Instructor: Tor Mohling
Office: ECOT 727
Email: tor@bio3d.colorado.edu
Office Hours:
  Tue/Thu 3-4:40pm in the WOZ - eccs 123
   or by appointment

Grading Information

Grades will be determined with the following percentages for each type of work:

The Group Project

Look at http://bio3d.colorado.edu/tor/saclass/group.html for information about the group project.

Syllabus

The following is a high-level synopsis of topics that will be covered during each period of time (dividing the semester into thirds).

Specific dates are given for homework and exams. Homeworks are due one week after they are handed out.

weeks 1 - 5
  Introduction
    Basic admin commands
    Shell scripting
  OS installation
    Package management
    User management
  System startup/shutdown
    PC Bios
    Sparc Openboot
    Serial Consoles
    Bootloading
    Kernels
    Init
  Basic daemons: syslog cron
  Filesystems
weeks 6 - 10
  TCP/IP networking
  Host network setup
  Routing, NAT, bridging
  Inetd/xinetd
  RPC, NFS and AMD/automount
  Firewalls
  Cryptography, OpenSSL, OpenSSH
<spring break>

weeks 11 - 15
  DNS
  Sendmail
  Backups
  Maintaining multiple machines
  FTP
  Apache
  CGIwrap


Campus Policy Regarding Sexual Harassment

The campus has requested that all faculty include the following in our course materials:

The University of Colorado Policy on Sexual Harassment applies to all students, staff and faculty. Sexual harassment is unwelcome sexual attention. It can involve intimidation, threats, coercion, or promises or create an environment that is hostile or offensive. Harassment may occur between members of the same or opposite gender and between any combination of members in the campus community: students, faculty, staff, and administrators. Harassment can occur anywhere on campus, including the classroom, the workplace, or a residence hall.

Any student, staff or faculty member who believes s/he has been sexually harassed should contact

  the Office of Sexual Harassment (OSH) at 303-492-2127
  or the Office of Judicial Affairs at 303-492-5550.

Information about the OSH and the campus resources available to assist individuals who believe they have been sexually harassed can be obtained at: http://www.colorado.edu/sexualharassment/


Other Information about the Class

Class Notes
Many of the slides used in the course have annotated versions that can be found Here: http://bio3d.colorado.edu/tor/sadocs/. Links to specific documents will be given below in the Class Diary for the days when they are relevent.

Operating Systems
The focus of the class will be on Fedora Linux and OpenBSD. However, information for other operating systems (particularly Solaris and FreeBSD) will be mentioned when relevant.
OpenBSD - http://www.openbsd.org/
FreeBSD - http://www.freebsd.org/
NetBSD - http://www.netbsd.org/
Fedora Linux - http://fedora.redhat.com/
Debian Linux - http://www.debian.org/
Redhat Linux - http://www.redhat.com/
SuSe Linux - http://www.novell.com/linux/suse/


Class Diary

Notes about what actually happened in each day's class will be added to this document here.

T Jan 11 - Class One
Introduction to the class.

Unfortunately, I was told I would have a smart classroom that would allow me to display from my laptop, but instead we have an old-school, chalkboard-only room. On thursday I will bring a VGA projector. After that we will be meeting in a new classroom. Before you get too excited, however, it has a down side: the new room will be in another building across campus. I will announce the exact room location in thursday's class. A big plus about the new classroom is that it will allow everyone into the class from the waiting list! (The room seats 200.)

Despite not having the ability to display them in class, we discussed the information from slides that can be found here: http://bio3d.colorado.edu/tor/saclass/intro.html

R Jan 13 - Class Two
We spent the second day wading through a swamp of introductory concepts. Three lists were written on the chalkboard which sort of guided the discussion (Main topics, Shell concepts, and Commands).

Main topics

Logging in
Logging in is straightforward enough, except when it doesn't work! Problems with the graphical Window Manager that one uses is a frequent cause of login failure (when you know that the login and password are correct). For old-school window managers like FVWM one have a few dotfiles, the most important of which is .xsession (or .xinitrc, I symlink one to the other). When there are errors with the window manager starting, an attempt is made to record these errors in the user's .xsession-errors file. Gnome/KDE logon screens have an options button that allows you to select a failsafe login mode. This will bypass the window-manager and just give a shell window from which you can diagnose the problem. If you use XDM and have no options button then you need to use the special ctl-alt-f1 after you type your password (instead of the enter key) to enter failsafe-mode.

I also talked about using ctl-alt-f1 to go to a virtual console if you are using a PC with BSD or Linux on it in order to avoid problems with the Window Manager or just to use a serial console instead of a graphical one.

Shell
The shell is your main interaction with a unix system and it is important to know it (them) as well as possible. As an administrator it is also important to have a solid understanding of both csh-like shells (e.g. tcsh) and bourne-like shells (e.g. bash). Please read through the manpages for both bash and tcsh. I understand that these are very looong documents, but I have found that reading them carefully once (every few years ;) allows me to subsequently find what exact details I need much more quickly.

Whenever you type a command into a unix shell, it is (obviously) the shell that takes care of locating and running that command. What all is involved? How is it that environment variables are propagated to those commands? What happens if you put:

  echo "hello world"

into your .cshrc (or .bash_rc) file? Why?

Regarding shell redirection, the question was raised: how does one redirect STDERR to a file but still view STDOUT on the screen? In bash this is easy:

  $ cat /etc/passwd /etc/nofile 2>errors

but in tcsh.. I'm not really sure. Here is a way to put each in a seperate file:

  % (cat /etc/passwd /etc/nofile >goodoutput) >&errors

Documentation
Did I mention manpages?
  man man

Other useful local documentation can usually be found in places like /usr/share/doc or /usr/local/share/doc.

The Internet is the best resource though. Most distros have a doc URL, here are a few:

http://www.openbsd.org/cgi-bin/man.cgi
http://www.openbsd.org/faq/index.html
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/index.html
http://www.tldp.org/
http://www.redhat.com/docs/

And finally, my favorite is google. I use it all the time every day to answer various system-administration problems.

VI
VI is good to know 'cause it is ubiquitous like no other editor I know of. You can rely on it being there. Knowing the : (colon) commands is especially good when you are stuck with a broken terminal that won't do curses (cf ncurses(3)). There are many good VI references on the 'net. I recommend using google with search keywords like:
  vi tutorial
  vi reference card

These days there are a number of different flavors of VI. My personal favorite is vim.

Commands
The list of commands is given with the expectation that you read the manpages for each.

Shell concepts

Shell variables
Environment variables
Subsitution
Variable
History
Command
Redirection
Builtins

Commands

man
ls
ps
find
tcsh
bash
screen

Next class I want to introduce some more commands and then tie it together with shell scripting.

Your homework assignment is:

  Read the manpages for the above list of commands
  Read chapters one and four in the Purple or Green book

[ I understand that the bookstore is out of these at the moment; please read the chapters as soon as you can; try the library, borrow from another student... ]

T Jan 18 - Class Three
The first thing we did in class was to talk about the group project and the need to form up into groups of five people. There is a mandatory workshop that will be held on a bunch of different occasions. See http://bio3d.colorado.edu/tor/saclass/group.html#installation_workshop for a list of available time slots. There will be at most 10 people per time slot.

The WOZ lab is now in ECCS 123. If you have the old 'card access' cards, I will be enabling them for the room. There is a pile of other peoples' stuff in this room, so it is important to attend the workshop simply to find out where we will be setting up and what hardware you get to choose from.

Then I talked about various commands related to working with groups, but also useful things in their own right.

RCS
RCS (Revision Control System) is a totally useful tool for maintaining version-history for any type of file. Originally designed for tracking software versions, it is widely used to maintain system config files too. See rcs(1), ci(1), co(1), rcsdiff(1) and rlog(1). To get started, you need to have an RCS subdirectory in the directory that contains files you wish to track. For example, I create an RCS directory in /etc on my server and then track important system files using rcs.
  coatlicue % cd /etc
  coatlicue % ls -lad RCS
  drwxr-xr-x  2 root  wheel  512 Aug  3 11:13 RCS/
  coatlicue % rlog pf.conf | head -16
  RCS file: RCS/pf.conf,v
  Working file: pf.conf
  head: 1.8
  branch:
  locks: strict
  access list:
  symbolic names:
  keyword substitution: kv
  total revisions: 8;     selected revisions: 8
  description:
  coatlicue firewall
  ----------------------------
  revision 1.8
  date: 2005/01/19 03:06:09;  author: root;  state: Exp;  lines: +12 -5
  transition to new DSL protocol, mostly IP addr changes

Use co -l file to check-out a file, locking it for editing. Use ci -u file to check-in a file and leave it unlocked.

NOTE WELL: if you forget an argument (-u or -l) when you use 'ci', then it checks in your changes and removes the file!! The file is still there (in the RCS sub-dir); do a co file to check it out unlocked.

Use rcsdiff file to compare the current version with the most recently checked-in version (by default).

umask
The umask command is a shell built-in, you may recall reading about it in the bash or tcsh manpages. It takes an octal value and acts as a mask when the permissions of newly created files and directories are set. E.g. by default a file gets created with permissions 666. With a umask of 022, those permissions get changed to 644. A fascist umask that lets no one else see your files would be 077.

Setgid bit on directories
Turning on the setgid bit on a directory causes files (and dirs) created in that directory to inherit the permissions of the parent directory (note that your umask will still take precedence too).
  feanor % cd classdir/
  feanor % ls -la
  total 12
  drwxrws---   2 tor  classgrp   512 Jan 18 18:16 ./
  drwxr-xr-x  16 tor  wheel     2048 Jan 18 19:52 ../
  -rw-r--r--   1 tor  classgrp     0 Jan 18 17:34 goo
  -rw-rw-r--   1 tor  classgrp     0 Jan 18 17:35 goo2

Note the permissions for the directory . and also: file goo was created with a umask of 022, while file goo2 was created with a umask of 002.

sudo
The sudo(8) command is great way to run commands as root (or another user) without actually logging as root (or that user). Benefits include:

Then we talked about shell-scripting. Two key points to start with:

  1. The first line of any shell-script must start with a #! followed by the full path to the shell used to interprete the script. E.g.
      #!/bin/sh

    (No spaces before or after the #!.)

  2. The file must be made to be executable. E.g.
      chmod +x file

You can get lots of information (of varying quality, of course) by doing a http://google.com/ search on the keywords unix shell scripting.

I have prepared a file of various shell-scripting examples that can be found here: http://bio3d.colorado.edu/tor/saclass/shell-examples

Here are the two scripts that were developed in class:

  #!/bin/sh
  #
  # hostinfo - wrapper around the uname command
  #
  # uname -a:
  # OpenBSD feanor.xbalanque.net 3.6 GENERIC#219 i386
  #
  OS=`uname -a | awk '{ print $1 }'`
  HOST=`uname -a | awk '{ print $2 }'`
  VERS=`uname -a | awk '{ print $3 }'`
  echo "OS is $OS"
  echo "HOST is $HOST"
  echo "VERS is $VERS"
  #
  if [ $OS = "OpenBSD" ]; then
      echo "the OS is openbsd"
  elif [ $OS = "Linux" ]; then
      echo "the OS is linux"
  else
      echo "the OS is unknown"
  fi

And:

  #!/bin/sh
  #
  # userinfo -
  #
  USERS=`grep /home /etc/passwd | awk -F: '{ print $1 }'`
  for user in $USERS; do
      echo "user is $user"
  done

The homework one can be found here: http://bio3d.colorado.edu/tor/saclass/HW1 It is due by midnight, Tuesday 25 January 2005.

R Jan 20 - Class four
Today we talked about Unix Filesystems. An annotated version of the slides (which we will finish next Tuesday) can be found here: http://bio3d.colorado.edu/~tor/sadocs/filesys/filesys.html . We stopped at the 'Adding a disk' topic.

Reading:

Chapter 5 - Filesystems, in the Green or Purple book
Manual pages
  mount(8)
  umount(8)
  fstab(5)
  lsof(8) - on Linux
  fstat(1) - on BSD
  filesystems(5) - on Linux
  fs(5) - on BSD
  hier(7) - compare between Linux and BSD
T Jan 25 - Class five - Adding a disk
Today we went over the process of adding a disk to a Unix system. See http://bio3d.colorado.edu/~tor/sadocs/filesys/filesys.html

The question was raised as to re-sizing an existing partition while using fdisk(8). While you can change the physical paramaters of an existant fdisk partition, it makes no updates to the filesystem that may live there and you will easily lose access to the entire filesystem. Two utilities were mentioned that allow you do re-sizing gracefully. sfdisk is a newer Linux utility that apparently does this, although the sfdisk manpage on my fedora system says the following:

  sfdisk  has  four  (main) uses: list the size of a partition, list the
  partitions on a device, check the partitions on a device, and  -  very
  dangerous - repartition a device.

Another utility that was mentioned is parted - partition editor. There appear to be at least two seperate versions

http://qtparted.sourceforge.net/
http://www.gnu.org/software/parted/

It looks like qtparted will resize ntfs partitions while the Gnu parted does not. It also looks like neither will resize OpenBSD or FreeBSD partitions. FreeBSD has the vinum project that provides RAID and re-size capabilities.

Reading:

Chapter 8 - Adding a disk, in the Green or Purple book
Manual pages
  fdisk(8)  - compare between BSD and Linux
  disklabel(8) - on BSD
  newfs(8) - on BSD
  sfdisk(8) - on Linux
  mkfs(8) - on Linux
R Jan 27 - Class six - booting, init and startup scripts
Tonight's topic was booting Unix systems and the phases involved. Notes can be found here: http://bio3d.colorado.edu/~tor/sadocs/misc/booting.html . I hoped to make it all the way to shutting down as well, but that will have to wait until next time.

Someone asked about AMD 64 support, and the answer is yes; from http://www.openbsd.org/36.html:

  # SMP support on OpenBSD/i386 and OpenBSD/amd64 platforms.

Here is a nice system to run it on ;) http://global.shuttle.com/Product/barebone/brb_OverView.asp?B_id=34

Usage information for the grub bootloader can be found here at http://www.gnu.org/software/grub/manual/html_node/index.html .

Reading:

Chapter 2 - Booting and shutting down, in the Green or Purple book
Manual pages
  boot(8) - on OpenBSD, FreeBSD and Fedora et al.
    On all systems this manpage has something new and different
    to offer; highly recommended to read them all if you can.
    The manpage is (7) on Fedora.
  biosboot(8) - OpenBSD
  installboot(8)
  boot_i386(8)
  bootparam(7) - Linux
  init(8) - both BSD and Linux
  rc(8), rc.conf(8), netstart(8) - OpenBSD
  inittab(5) - Fedora (et al. but not all) Linux
T Feb 1 - Class seven
Today's class saw the tail-end of the discussion regarding system startup and shutdown. (Cf http://bio3d.colorado.edu/~tor/sadocs/misc/booting.html )

From there we talked about system logfiles and syslogd(8) in particular. See http://bio3d.colorado.edu/~tor/sadocs/misc/syslog.html

Reading:

Chapter 11 - Syslog and Logfiles, in the Green or Purple book
Manual pages
  chkconfig(8) - Fedora (et al.) Linux
  halt(8) reboot(8) shutdown(8) init(8) - BSD and Linux
  syslogd(8) - BSD and Linux
  syslog.conf(5) - BSD and Linux
  newsyslog(8) - BSD
  logrotate(8) - Linux

Homework Two is available at http://bio3d.colorado.edu/~tor/saclass/HW2

Homework One is almost finished being graded and will be sent back tomorrow (Wednesday, 2 Feb 2005) evening.

R Feb 3 - Class eight
Class eight was cancelled.

T Feb 8 - Class nine
Tonights class covered cron(8) and moved on to software installation with the ports(7) tree on BSD.

Reading:

Chapter 9 Periodic Process, in the Green or Purple book
Manual pages
  cron(8)
  at(1)
  crontab(1) crontab(5)
  /usr/bin/run-parts  on Fedora (and other) Linux
        note that this is a just a short shell script
  periodic(8) - FreeBSD
  ports(7)
  mk.conf(5) - OpenBSD
  make.conf(5) - FreeBSD

Cron on OpenBSD is also Vixie Cron.

More info about the BSD ports collection can be found at http://www.openbsd.org/ports.html and in the 'Packages and Ports' chapter of the FreeBSD Handbook http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports.html .

The correct incantation for having sudo automatically invoked for you while building ports goes in the /etc/mk.conf file for OpenBSD (/etc/make.conf on FreeBSD) and looks like this:

  SUDO=/usr/bin/sudo

For FreeBSD you'll want it to look like this:

  SU_CMD=eval /usr/local/bin/sudo

Exam One will be held during the next class on Thursday, Feb 10 2005.

R Feb 10 - Class ten
Exam one.

T Feb 15 - Class eleven
Today's class focused entirely on adding software to BSD and Linux machines using ports(7) and packages(7) on BSD and rpm(8) on Linux.

Reading:

Manual pages
  ports(7) bsd.port.mk(5) mk.conf(5) - OpenBSD
  ports(7) make.conf(5) - FreeBSD
  packages(7) pkg_add(8) pkg_info(8) pkg_delete(8) - BSD
  rpm(8) rpm2cpio(8) - Fedora Linux

Useful URLs

http://www.openbsd.org/ports.html
http://www.freebsd.org/ports/index.html
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports.html

Something that was not discussed in class is the use of (semi) automated software update managers like RedHat's up2date(8), Fedora's yum(8), or Debian's apt_get(8). The default behavior for these types of systems is to make sure the package profile currently installed is always of the most recent versions. Here is a /etc/yum.conf config file for Fedora users that accesses a local mirror site:

  [main]
  cachedir=/var/cache/yum
  debuglevel=2
  logfile=/var/log/yum.log
  pkgpolicy=newest
  distroverpkg=redhat-release
  tolerant=1
  exactarch=1
  retries=20
  
  [COLORADO Fedora base]
  name=Colorado Fedora Core $releasever - $basearch - Base
  baseurl=ftp://mirror.colorado.edu/pub/fedora/linux/core/$releasever/$basearch/os/
  
  [COLORADO Fedora updates]
  name=Colorado Fedora Core $releasever - $basearch - Released Updates
  baseurl=ftp://mirror.colorado.edu/pub/fedora/linux/core/updates/$releasever/$basearch/

I run yum(8) by hand mostly because the labs I maintain are small and there are certain packages that need to be tested before they are actually deployed. You can change the 'cachedir' to be a shared directory and do downloads on one machine only:

  yum --download-only update

Then on all of the machines use yum with the -C option to use your local cache only:

  yum -C update

Next class will be covering user management. That would be Chapter 6 in the Green or Purple books as well as http://bio3d.colorado.edu/tor/sadocs/misc/users.html .

R Feb 17 - Class twelve
Class twelve covered user management, mostly focused on issues surrounding the /etc/passwd file. A very rough overview of the lecture can be found here: http://bio3d.colorado.edu/~tor/sadocs/misc/users.html .

Reading:

Chapter 6 Adding New Users, in the Green or Purple book
Manual pages
  passwd(5) shadow(5)- Linux or Solaris
  master.passwd(5) login.conf(5) - BSD
  pam(8) - Linux or Solaris
  passwd(1) vipw(8)
  pwd_mkdb(8) - BSD
  pwck(8) pwconv(8) - Linux or Solaris
  adduser(8) adduser.conf(5) - BSD
  useradd(8) - Linux or Solaris
  crypt(3) blowfish(3) md5(3) - BSD
  md5(1) - BSD
  dgst(1) - Linux

I recommended getting on one or more security mailing lists. Here are the URLs I showed in class

http://www.securityfocus.com/
The Bugtraq list is hosted here.

http://www.sans.org/
I recommend both the Newsbites and the <@RISK> newsletters.

Group Project: at this point you should have two machines (at least) installed for your project. One of these machines should have two network interfaces (only one in use so far..) and be running OpenBSD. The other must be running some form of 'nix. You should have user accounts setup on each machine.

Next week we will be diving into TCP/IP. That is Chapter 13 in the Green or Purple books.

T Feb 22 - Class thirteen
We finally started talking about TCP/IP! Tonight covered most of the slides that can be found here: http://bio3d.colorado.edu/tor/sadocs/tcpip/tcpip-1.html

One person asked about the PPPoE (PPP over Ethernet) protocol. Here is the RFC for it (RFC2516): ftp://ftp.rfc-editor.org/in-notes/rfc2516.txt

Reading:

Chapter 13 TCP/IP Networking, in the Green or Purple book
Manual pages
  protocols(5) - man page reference to /etc/protocols.  These
                 are values that van be found in the TYPE field
                 of an IP packet
  arp(8) rarpd(8) ethers(5)
  BSD: netintro(4) inet(4) ip(4)
  Fedora: netlink(7) ip(7)

I will start next class by going over the netmask and subnet concepts a little more and eventually get to the TCP and UDP protocols.

Homework three (due on March 1st) can be found here: http://bio3d.colorado.edu/tor/saclass/HW3

R Feb 24 - Class fourteen
Thursday's class continued the discussion on TCP/IP. We looked at the ICMP and UDP protocols as well as some networking tools. We also talked about the distinction between MAC (ethernet) addresses and IP addresses. See http://bio3d.colorado.edu/tor/sadocs/tcpip/macip.html For slides related to the MAC/IP address distinction. Slides covering UDP are at http://bio3d.colorado.edu/tor/sadocs/tcpip/tcpip-2.html . We have not yet talked about TCP - we'll do that on Tuesday.

Tools that were talked about:

traceroute(8)
Traceroute uses UDP as well as the TTL field of the IP protocol to determine the route that the packets travel to reach a destination.

tcpdump(8)
Tcpdump is a great example from the category of programs called packet sniffers. You use it to put a network interface into promiscuous mode and capture network traffic, displaying the values in each packet's protocol headers.

nc(1)
Netcat is a simple tool that can run as a client or server with UDP or TCP using user-specified port numbers. It is useful for testing connectivity to specific TCP or UDP ports as well as testing specific rules in a firewall.

If the concept of CIDR subnetting and netmasks is still confusing, here is another description of how the netmask value gets used.

First, note that most hosts have a very simple network setup in that they have one network-interface and one IP address; they live on simple networks with only a single router connecting them to the rest of the internet, the IP of the router is called a DEFAULT ROUTE.

Second, it is always easier to understand netmasks when you break down the decimal IP address values into their binary representation. To do that, it is also handy to have a small chart of the first eight powers-of-two [0-7]:

   power: 7 6 5 4 3 2 1 0    Value in decimal
          . . . . . . . 1    2^^0 =    1
          . . . . . . 1 .    2^^1 =    2
   bit    . . . . . 1 . .    2^^2 =    4
   posi-  . . . . 1 . . .    2^^3 =    8
    tion  . . . 1 . . . .    2^^4 =   16
          . . 1 . . . . .    2^^5 =   32
          . 1 . . . . . .    2^^6 =   64
          1 . . . . . . .    2^^7 =  128
                                sum ----
                                     255  = 2^^8 - 1   (minus one, cause we
                                                        start at zero!)

Computing decimal-to-binary or binary-to-decimal is easy. For eight bits the possible values are 0 thru 255.

  For example, the octect '138' is         128  +   8 + 2
  so the corresponding octet in binary is:  1 0 0 0 1 0 1 0

Now, imagine the host, coatl, on the saclass.net network with an IP address of 10.0.0.18 sending an ICMP packet to www.colorado.edu which has an IP address of 128.138.129.98. The machine coatl needs to decide where to send the packet: is it destined to a machine on the local network or is it going to a machine on a remote network? To figure out the answer, coatl compares the destination IP address (i.e. 128.138.129.98) with its own netmask (i.e. 255.255.255.0).

Coatl has an IP of 10.0.0.18 and a netmask of 255.255.255.0. Coatl's own subnet can be determined as follows:

     IP       00001010 . 00000000 . 00000000 . 00010010  (10.0.0.18)
  netmask     11111111 . 11111111 . 11111111 . 00000000  (255.255.255.0)
  logical AND -----------------------------------------
  the subnet  00001010 . 00000000 . 00000000 . 00000000  (10.0.0.0)

Do the same operation with coatl's netmask and the destination IP:

     IP       10000000 . 10001010 . 10000001 . 01100100  (128.138.129.98)
  netmask     11111111 . 11111111 . 11111111 . 00000000  (255.255.255.0)
  logical AND -----------------------------------------
  the subnet  10000000 . 10001010 . 10000001 . 00000000  (128.138.129.0)

Now compare the two subnet values:

  the subnet  00001010 . 00000000 . 00000000 . 00000000  (10.0.0.0)
  the subnet  10000000 . 10001010 . 10000001 . 00000000  (128.138.129.0)

obviously, they are not equal, so the packet is destined to a remote host.

T Mar 1 - Class fifteen - half way
Today's class finished the whirlwind overview of the TCP/IP protocols with a look at TCP itself. ( http://bio3d.colorado.edu/tor/sadocs/tcpip/tcpip-2.html ).

We then looked at configuring network information for a simple host (i.e. one network interface, one IP address). The slides I used are here: http://bio3d.colorado.edu/tor/sadocs/tcpip/simple.html .

Reading:

  arp(8)      these pages should be on all OSs
  netstat(8)    ** they will be different too **
  route(8)
  ifconfig(8)
  resolv.conf(5)
  hostname.if(5) mygate/myname(5)  OpenBSD

Questions were asked about the specifics of the URGENT and PUSH flags regarding the TCP protocol. The RFC (rfc793) has this to say:

  TRANSMISSION CONTROL PROTOCOL
     DARPA INTERNET PROGRAM
     PROTOCOL SPECIFICATION
        September 1981 !!
  
  ..
  
  The data that flows on a connection may be thought of as a stream of
  octets.  The sending user indicates in each SEND call whether the data
  in that call (and any preceeding calls) should be immediately pushed
  through to the receiving user by the setting of the PUSH flag.
  
  A sending TCP is allowed to collect data from the sending user and to
  send that data in segments at its own convenience, until the push
  function is signaled, then it must send all unsent data.  When a
  receiving TCP sees the PUSH flag, it must not wait for more data from
  the sending TCP before passing the data to the receiving process.
  
  ..
  
  TCP also provides a means to communicate to the receiver of data that
  at some point further along in the data stream than the receiver is
  currently reading there is urgent data.  TCP does not attempt to
  define what the user specifically does upon being notified of pending
  urgent data, but the general notion is that the receiving process will
  take action to process the urgent data quickly.

The PUSH flag is used extensively for interactive TCP sessions (like I mentioned in class.. if you watch an interactive SSH session you will see a packet getting sent for every keystroke!) On the other hand, if you tcpdump an FTP session, while you see the command-channel proceeds as an interactive session, the data gets transmitted over a different TCP session and that makes much less use of the PUSH flag.

Next class we will be talking about NAT and then IP firewalls.

R Mar 3 - Class sixteen
[ apologies for getting this out so late.. ]

Last Thursday's class covered the topic of NAT or Network Address Translation. NAT is how many computers, at home or in a lab, can access the Internet through a gateway machine using a single real (i.e. routable) IP address. My NAT slides are here: http://bio3d.colorado.edu/tor/sadocs/tcpip/nat.html .

There are lots of resources that can be found if you google for NAT as a keyword. Some specific sites of interest are:

http://www.netfilter.org/documentation/HOWTO/NAT-HOWTO.html
http://www.openbsd.org/faq/pf/nat.html

We ended the class by just beginning to define terminology regarding IP Firewalling. This discussion will be continued on Tuesday.

T Mar 8 - Class seventeen
Today was spent entirely focused on IP Firewalling, particularly with OpenBSD's PF software. Slides that were used are http://bio3d.colorado.edu/~tor/sadocs/tcpip/firewall.html and http://bio3d.colorado.edu/~tor/sadocs/tcpip/pf.html .

NOTE WELL that URL's given in the above slides are out of date! Here are the current versions:

http://www.openbsd.org/faq/pf/index.html
The OpenBSD PF Guide is well-written and has tons of examples. It also goes into cool details that I was not able to cover in class like tables, anchors (these are like rule-groups), queuing and lots more. There are many options including short-hand keywords that can be used to make a ruleset both shorted and more readable.

http://www.netfilter.org/documentation/
This links to several very useful HowTos about IPtables (aka netfilter) and how to make use of it in various situations. It is also the site for downloading the latest IPtables source code and nifty kernel modules that you can get extra to enable specific features in a Linux firewall (like dynamic H.323 connection tracking, logging features and so on).

Relevant OpenBSD manpages:

  pfctl(8)  pflogd(8)  pf.conf(5)
  pf(4)  pf.os(5)  authpf(8)  ftp-proxy(8)

Here is a complete PF example:

  #
  # RCS: $Id: index.pod,v 1.9 2005/03/11 02:17:58 tor Exp tor $
  #
  ext = "fxp0"
  int = "fxp1"
  my_ip = "10.0.0.2"
  
  scrub in on $ext all
  
  # NAT rules:
  nat on $ext inet from 10.1.0.0/24 to any -> $ext
  
  # PF defaults that apply to every interface
  pass out all
  pass in all
  
  # block and log all incoming traffic by default
  block             in  log on $ext           all
  block return      in  log on $ext           all
  
  # immediately block and don't log ident queries
  block return-rst in quick on $ext proto tcp from any \
                                     to any port = auth flags S/SA
  
  # block and log incoming packets from reserved address space and invalid
  # addresses, they are either spoofed or misconfigured, we can't reply to
  # them anyway (hence, no return-rst).  Block incoming traffic from our
  # own internal IP space.
  block in log quick on $ext from { 172.16.0.0/12, \
                                    192.168.0.0/16, \
                                    127.0.0.0/8, \
                                    0.0.0.0/32, \
                                    255.255.255.255/32 } to any
  # block and log outgoing packets that don't have our address as source
  block out log quick on $ext from ! $my_ip to any
  
  # Keep state for outgoing TCP sessions, allows return traffic back
  pass out log on $ext proto tcp from any to any modulate state
  pass out log on $ext proto udp from any to any keep state
  
  # ICMP rules
  pass in on $ext inet proto icmp all icmp-type \
                 { echoreq, echorep, timex, unreach }
  # UDP Rules
  pass in log on $ext inet proto udp from any to any port \
                 { = domain, = 1053 } keep state
  # TCP Rules
  pass in log on $ext inet proto tcp from any to any port \
        { = domain, = smtp,  = https, = imaps, = ssh } keep state
R Mar 10 - Class eighteen
The first half of class focused on bridging firewalls. The slides are here: http://bio3d.colorado.edu/tor/sadocs/tcpip/bridge.html . With bridging groups you can turn an old PC into a managed switch if you throw a bunch of NIC cards into it. A machine like this is very useful in situations where you want to firewall some specific subset of machines but the logical topology of the existing network is required to stay the same (i.e. no IP changes to existing machines); no network changes need to happen on any existing infrastructure, the bridging firewall box is just inserted into an exisiting network link (okay, so you need to replace an existing ethernet cable with two shorter ones.. ;)

Bridging is fairly easy to setup in both the OpenBSD and Fedora Linux environments.

Some relevant URLs are:

http://www.openbsd.org/faq/faq6.html#Bridge
http://ebtables.sourceforge.net/

The second half of class was spent covering inetd and its superior Linux cousin xinetd. There is a brief synopsis of inetd in the green or purple book, chapter 28 on Daemons. My slides are here: http://bio3d.colorado.edu/tor/sadocs/misc/inetd.html . Inetd should always be used with TCPwrappers if possible; xinetd has tcpwrapper-functionality built in.

Relevent Manual pages:

  For BSD
    brconfig(8) bridgename.if(5)
    inetd(8) inetd.conf(5)
  For Linux
    brctl(8)
    xinetd(8)  xinetd.conf(5)
  Both
    tcpd(8)  hosts.allow(5)

Next class will be spent looking several related network services centered on NFS.

T Mar 15 - Class nineteen
Class 19 was spent covering the the NFS (Network File System) protocol (version three) and associated things like RPC (this is old-school Remote Procedure Call and a dangerous thing on which NFS relies) and a quick look at automounters. The slides I used are: http://bio3d.colorado.edu/tor/sadocs/misc/rpc.html http://bio3d.colorado.edu/tor/sadocs/filesys/nfs.html and http://bio3d.colorado.edu/tor/sadocs/filesys/amd.html .

Related reading:

Chapter 17 The Network File System in the Green or Purple books
Manual pages
  exportfs(8)  exports(8)
  mountd(8) nfsd(8)
  rpc.statd(8) rpc.lockd(8)
    no statd on openbsd, check out the BUGS of rpc.lockd ;0
  showmount(8)  nfsstat(8)  rpcinfo(8)  portmap(8)
    Solaris calls portmap 'rpcbind'
http://www.openbsd.org/faq/faq6.html#NFS
http://www.tldp.org/HOWTO/NFS-HOWTO/
http://www.fsl.cs.sunysb.edu/docs/amd-book/
http://www.am-utils.org/
http://www.tldp.org/HOWTO/Automount.html
http://www.sunhelp.org/faq/autofs.html

Exam two is Thursday. It only covers things we have looked at since the previous exam. Topics include:

  ports(7) and packages(7) on BSD and rpm(8) on Fedora
    and related utilities (e.g. pkg_add)
  user management and related utilities (e.g. adduser)
  TCP/IP Networking
    protocols: ethernet arp ip icmp udp tcp
        utilities: too many to list but a partial list is
          tcpdump
          traceroute
          netstat
          route
          ifconfig
          arp
        NAT
        IP Firewalling
        NFS and related (NO AMD or automount though)

Like the last exam the focus is on understanding the concepts involved, not the specific syntax on the command line or - in particular - in config files like /etc/pf.conf. NAT is a good example.. it is important to undestand how NAT works by translating the source IP address, choosing alternate port numbers and recording this state in a database; It would be somewhat unreasonable to expect you to remember:

  nat [pass] on interface [af] from src_addr [port src_port] to \ 
     dst_addr [port dst_port] -> ext_addr [pool_type] [static-port]
T Mar 29 - Class twenty one - post spring break
Class twenty one was spent doing an overview of cryptography as it relates to computers. Slides are here: http://bio3d.colorado.edu/tor/sadocs/crypto/crypto.html .

Next class we will be looking at the related topics of ssh and openssl. After we finish that we will dive into DNS.

R Mar 31 - Class twenty two
Class twenty two was spent giving an overview of SSH and all of its uses. Slides are here: http://bio3d.colorado.edu/tor/sadocs/crypto/ssh.html . My favorite use of SSH is the on the fly tunnel where you pipe the output of some command like tar into an SSH session that untar's the data (for example) on the other side:
  tar -zcf - ./somedir | ssh otherbox 'cd /to/where/it/goes; tar -zxf -'

That is an example of using SSH in remote command execution mode. It's important that everything to be done on the remote machine appear in quotes so that it looks like one argument to SSH.

We started to look at OpenSSL and X.509 certificates but only barely got started. That will be the topic of class on Tuesday.

T Apr 5 - Class twenty three
Today we finished looking at OpenSSL and how to generate and sign your own X.509 certificates for use with various applications like apache. These are the certificates - when signed by a known Certificate Authority like verisign.com - that authenticate web transactions like credit-card purchases and so on. OpenSSL is also used to encrypt the transaction itself (i.e. https). Slides are here: http://bio3d.colorado.edu/tor/sadocs/crypto/ssl.html .

After that I presented an overview of portscanning in general. Nmap - http://insecure.org/ - is the canonical portscanner that just keeps getting better (it now does service location - e.g. finding a web-server on a random port instead of port 80 - and versioning for many services - e.g. apache-2.1.13). Nessus - http://www.nessus.org/ - is a great alternative to Nmap for system administrators because of its reporting capabilities. It also features an attack-scripting-language (NASL) and plugin architecture with a plugin collection that is frequently updated to test for recent exploits. Lecture slides are here: http://bio3d.colorado.edu/tor/sadocs/tcpip/nmap.HTML/index.html .

Next class we will start looking at DNS.

Homework Four is available here: http://bio3d.colorado.edu/tor/saclass/HW4 .

R Apr 7 - Class twenty four
Class twenty four class was the first of several that will be spent looking at the DNS (Domain Name System). There is a hefty chapter in the Purple or Green Admin Handbooks on DNS (Chapter 16 - The Domain Name System).

One thing we talked about that didn't get explained satisfactorily is the different roles that a name-server can play and how this relates to firewalling. There is a good picture, Exhibit B in the Purple or Green books (p. 413 or 379 resp), that illustrates the concept.

Imagine an organization with 10,000 or more computers (like CU Boulder, or any big corporation). The idea is that small groups of the machines, like in a single lab, all use a local caching server. The caching servers are configured to query local forwarding servers which in turn contact a single (or maybe just a few) dedicated secondary machine(s) (called the 'big server' in the picture). It is this machine(s) that is allowed outgoing DNS access through the site firewall. All of that is for serving internal client queries and none of these machines ought to be visible for querying from the public Internet. A (potentially) entirely seperate set of servers can serve DNS data through the firewall to the public.

T Apr 12 - Class twenty five
DNS continued.. we looked at zone file syntax and started to look at BIND name server configuration. To be concluded next class.

Other information:

R Apr 14 - Class twenty six
DNS continued.. we are almost finished. I'll wrap up talking about DNSSEC next class and then move on to Sendmail.

Both BIND and DHCPD are available from http://www.isc.org/ but tend to come by default with most 'nix variants. The Operations link on the left-hand bar of the main page at isc.org has some interesting data about DNS usage and other information.

Recommended Reading:

http://bio3d.colorado.edu/tor/sadocs/dns/dns.html
Purple or Green Handbook, Chapter 16 The Domain Name System
Bind ARM - comes with the BIND source
For OpenBSD, find it in file:///usr/share/doc/html/bind/Bv9ARM.html

For Fedora Linux find it in file:///usr/share/doc/bind-9.2.3/arm/Bv9ARM.html

http://www.dnssec.net/ - Lots of information about DNS security.
Dynamic Zone Updates and DHCP
http://www.lesbell.com.au/Home.nsf/0/3f0a77f02a05e665ca256bee00104c74?OpenDocument

http://ops.ietf.org/dns/dynupd/secure-ddns-howto.html

T Apr 19 - Class twenty seven
We finished DNS with a rapid look at DNSSEC. See http://www.dnssec.net/ for lots of information about DNSSEC and DNS security issues in general.

Then we looked at how to set up a server machine to be trusted via root ssh by a client. To do this you need to:

  o Enable root ssh in the clients' sshd_config files
  o On the server:
      sudo cp /etc/ssh/ssh_host_dsa_key /root/.ssh/id_dsa
      sudo cp /etc/ssh/ssh_host_dsa_key.pub /root/.ssh/id_dsa.pub
  o On the client(s):
      install the server's /root/.ssh/id_dsa.pub key into
          the client's /root/.ssh/authorized_keys file

Then we started talking about sendmail. Sendmail is covered in Chapter 19 of the Purple or Green Handbooks. We will continue to talk about sendmail next class (and the one after that..).

Homework five (due on: Thursday, 28 April after class) is available here: http://bio3d.colorado.edu/tor/saclass-2005/HW5 .

R Apr 21 - Class twenty eight
Sendmail concepts, to be continued next class.
  the 5 email agents
  addresses
  aliases
  database files: aliases access local-host-names relay-domains

T Apr 26 - Class twenty nine
Sendmail continued. Class 29 covered the basics of sendmail configuration as well as the following 'advanced' concepts:
  masquerading
  genericstable
  virtusertable
  starttls

There are several configuration examples in the slides ( http://bio3d.colorado.edu/tor/sadocs/email/email.html ) including server and client examples that incorporate X.509 certs for use with STARTTLS (you also need the server to have its key listed in the /etc/mail/access file; a good description of that is at http://www.ofb.net/~jheiss/sendmail/tlsandrelay.shtml ).

You can also look at the config on saclass.net in /usr/share/sendmail/cf/saclass.mc

Thursday we will look at some examples of using SpamAssassin with sendmail.

The last exam, Exam 3, is on Tuesday, 3 May at 7:30 PM (NOTE the time is later!). It will only cover DNS and Sendmail.

R Apr 28 - Class thirty
Last class! The last class was devoted to wrapping up Sendmail and class ended early. TOO Early as I totally forgot to add an important topic to my list of things: greylisting which I'll discuss below.

We did talk about two main issues: 1) secure, off-site relaying and 2) dealing with spam. (Greylisting falls under the latter issue.)

Secure, off-site relaying is when you want to allow specific, remote IP addresses or users to to be able to relay email through your MTA. By default all such relaying should be explicitly denied or your MTA will get blackholed. Two methods for accomplishing this were discussed: STARTTLS and pop-before-smtp; another method, SASL, can also be used but it's important to remember that SASL can be misconfigured such that authentication information gets transfered in plaintext, so be careful if you go that route. There are other solutions (SSH tunnel, for example).

STARTTLS http://www.ofb.net/~jheiss/sendmail/tlsandrelay.shtml
pop-before-smtp: http://popbsmtp.sourceforge.net/
SASL: http://www.sendmail.org/~ca/email/auth.html

Dealing with spam is a never-ending game of brinksmanship (greylisting is a good example as you'll see) but tools like SpamAssassin continue to be effective. Filtering for spam can be done at the MTA for all email and/or it can be done by the end-user with her own mailspool. Depending how SpamAssassin is installed and how you end up getting mail delivered enabling spam-filtering may be as simple as creating a .procmailrc file that looks like this:

  :0fw :spamassassin.lock
  | /usr/bin/spamc

Typically you'll need a suitable .forward file as well:

  "|IFS=' ' && exec /usr/bin/procmail -f- || exit 75 #tor"

That works for an end-user. Dealing with spam (and filtering for virii) at the MTA is more complex. We looked at a dual-sendmail method that uses amavisd-new; sendmail is configured as two seperate servers, a receiving server and a transmit server. The powerful amavisd-new daemon sits in between the two and provides any number of filters (you can configure multiple spam filters to use with it including SpamAssassin and many different Virus filters like ClamAV).

manpages: procmailrc(5) procmailex(5)
http://spamassassin.apache.org/
http://www.ijs.si/software/amavisd/
http://www.ijs.si/software/amavisd/README.sendmail-dual.txt
http://www.flakshack.com/anti-spam/wiki/index.php
http://www.clamav.net/

Don't forget that all this stuff is in the BSD ports tree (e.g. /usr/ports/mail/amavisd-new or /usr/ports/security/clamav).

Greylisting is a relatively new technique to fight spam that relies on the fact that most spammers' mailer-software is not RFC-compliant. Specifically, when email is received from an unknown sender (i.e. user@IP), the address is put into a greylist and the email is (temporarily) rejected with an error that looks like:

  451 4.7.1 Please try again later

A sending MTA that is RFC-compliant will simply queue the email and resend it after some amount of time (the default for sendmail is 4 hours, I think). When our local machine receives email from the same sender address a second time (after at least ten or fifteen minutes - this is configured on the receiving side) then the address is moved to a whitelist and the email is allowed in to our MTA. Greylisting is currently very powerful (also very controversial) but sooner-than-later the spammers will catch on and fix their mailing-software. Greylisting is part of the OpenBSD spamd(8) daemon and relies on some special tables in the pf firewall (man -M /usr/share/man spamd).

NOTE WELL that if you start greylisting on your MTA but you have a failover MX that doesn't do greylisting then you lose all the benefits: the failover MX host will queue your spam and re-send it after the required interval!

http://projects.puremagic.com/greylisting/
http://greylisting.org/
http://greylisting.org/implementations/openbsd_spamd.shtml

An un-related URL: http://labrea.sourceforge.net/ ; I stumbled across this sticky honeypot project and it looks pretty cool. Some of you might find it very interesting!