File or directory names are printed as: /path/to/file. For example: /etc/fstab

Commands are printed as bold text. For example: ls -l

Screen output is printed like this:

Hello world!
      

If commands are being entered in the screen output the commands will be printed as bold text:

$ command
Output
      

If a command is executed as root, the shell will be displayed as “#”. If a command is executed as a normal non-privileged user, the shell will be displayed as “$”.

Some sections contain extra notes. It is not necessary to know the information in notes, but notes may provide valuable information, or pointers to information. Notes are printed like this:

	Note
	This is a note.

The Linux HOWTOs are a collection of documents which cover specific subjects related to GNU/Linux. Most Linux HOWTOs are not tailored to a specific distribution, therefore they are very useful for Slackware Linux users. The linux-howtos package in the “f” software set contains the HOWTO collection. After installing this package the HOWTOs are available from the /usr/doc/Linux-HOWTOs/ directory. Slackware Linux also contains a small collection of Linux-related FAQs (FAQs are documents that answer Frequently Asked Questions). The Linux FAQs are installed to the /usr/doc/Linux-FAQs/ directory, and are available from the linux-faqs package, which is also part of the “f” software set.

Most UNIX-like commands are covered by a traditional UNIX help system called the manual pages. You can read the manual page of a program with the man command. Executing man with the name of a command as an argument shows the manual page for that command. For instance,

$ man ls
      

shows the manual page of the ls command.

If you do not know the exact name of a manual page or command, you can search through the manual pages with a keyword. The -k option is provided to make use of this facility:

$ man -k rmdir
hrmdir               (1)  - remove an empty HFS directory
rmdir                (1)  - remove empty directories
rmdir                (2)  - delete a directory
      

The manual page collection is very extensive, and covers more subjects than just commands. The following sections of manual pages are available:

If there is more than one section that has a manual page with a specific name, as with for instance rmdir, you can choose what page you want to see by adding the section number of the manual page before the manual page name. For example:

man 2 rmdir
      

If you would like to print a manual page to a printer that you have set up, you can pipe the output of man to the lpr command. When the -t option of the man command is used, man will output the manual page in Postscript format, rather than ASCII. For example, you can use the following command to print the cat manual page:

$ man -t cat | lpr
      

The Slackware Linux website may be a bit outdated at times, but it provides many useful resources:

The Slackware Linux website is available at: http://www.slackware.com/

LinuxQuestions is a large GNU/Linux forum with many helpful members. Particularly interesting is the Slackware Linux subforum, where you can seek assistance to help you with problems that you may have with Slackware Linux. The LinuxQuestions forum is available at: http://www.linuxquestions.org/

alt.os.linux.slackware is a Slackware Linux newsgroup. You can read newsgroups with a newsreader like tin or slrn, through the newsgroup server of your Internet service provider. On this newsgroup it is expected that you have read all necessary documentation before posting questions. If you have not done that, the chance of getting “flamed” is large.

One of UNIX's traditional strengths is multitasking. Multitasking means that multiple programs can be run at the same time. You may wonder why this is important, because most people use only one application at a time. Multitasking is a bare necessity for UNIX-like systems. Even if you have not started any applications, there are programs that run in the background. For instance, some programs provide network services, while others show a login prompt and wait until a user logs in on a (virtual) terminal. Programs that are running in the background are often called daemon processes^[2].

After a program is loaded from a storage medium, an instance of the program is started. This instance is called a process. A process has its own protected memory, named the process address space. The process address space has two important areas: the text area and the data area. The text area is the actual program code; it is used to tell the system what to do. The data area is used to store constant and runtime data. The operating system gives every process time to execute. On single processor systems processes are not really running simultaneously. In reality a smart scheduler in the kernel divides CPU time among processes, giving the illusion that processes run simultaneously. This process is called time-sharing. On systems with more than one CPU or CPU cores, more than one process can run simultaneously, but time-sharing is still used to divide the available CPU time among processes.

New processes are created by duplicating a running process with the fork system call. Figure 4.1, “Forking of a process” shows a fork() call in action schematically. The parent process issues a fork() call. The kernel will respond to this call by duplicating the process, and naming one process the parent, and the other process the child.

Figure 4.1. Forking of a process

Forking can be used by a program to create two processes that can run simultaneously on multiprocessor machines. However, this is often not ideal, because both processes will have their own process address space. The initial duplication of the process memory takes relatively much time, and it is difficult to share data between two processes. This problem is solved by a concept named multithreading. Multithreading means that multiple instances of the text area can run at the same time, sharing the data area. These instances, named threads, can be executed in parallel on multiple CPUs.

Operating systems store data in filesystems. A filesystem is basically a tree-like structure of directories that hold files, like the operating system, user programs and user data. Most filesystems can also store various metadata about files and directories, for instance access and modification times. In GNU/Linux there is only one filesystem hierarchy, this means GNU/Linux does not have drive letters (e.g. A:, C:, D:) for different filesystems, like DOS and Windows. The filesystem looks like a tree, with a root directory (which has no parent directory), branches, and leaves (directories with no subdirectories). The root directory is alway denoted with a slash (“/”). Directories are separated by the same character.

Figure 4.2. The filesystem structure

Figure 4.2, “The filesystem structure” shows the structure of a filesystem. You can see that the root directory / has two child directories: bin and home. The home directory has two child directories, joe and jack. The diagram shows the full pathname of each directory. The same notation is used for files. Suppose that there is a file named memo.txt in the /home/jack directory, the full path of the file is /home/jack/memo.txt.

Each directory has two special entries, “.”, and “..”. The first refers to the directory itself, the second to the parent directory. These entries can be used for making relative paths. If you are working in the jack directory, you can refer to the the /home/joe directory with ../joe.

You might wonder how it is possible to access other devices or partitions than the hard disk partition which holds the root filesystem. Linux uses the same approach as UNIX for accessing other filesystems. Linux allows the system administrator to connect a device to any directory in the filesystem structure. This process is named mounting. For example, one could mount the CD-ROM drive to the /cdrom directory. If the mount is correct, the files on the CD-ROM can be accessed through this directory. The mounting process is described in detail in Section 8.8, “Mounting filesystems”.

The Filesystem Hierarchy Standard Group has attempted to create a standard that describes which directories should be available on a GNU/Linux system. Currently, most major distributions use the Filesystem Hierarchy Standard (FHS) as a guideline. This section describes some mandatory directories on GNU/Linux systems.

Please note that GNU/Linux does not have a separate directory for each application (like Windows). Instead, files are ordered by function and type. For example, the binaries for most common user programs are stored in /usr/bin, and their libraries in /usr/lib. This is a short overview of the important directories on a Slackware Linux system:

In GNU/Linux virtually everything is represented as a file, including devices. Every GNU/Linux system has a directory with special files, named /dev. Each file in the /dev directory represents a device or pseudo-device. A device file has two special numbers associated with it, the major and the minor device number. The kernel knows which device a device file represents by these device numbers. The following example shows the device numbers for the /dev/zero device file:

$ file /dev/zero
/dev/zero: character special (1/5)
      

The file command can be used to determine the type of a file. This particular file is recognized as a device file that has 1 as the major device number, and 5 as the minor device number.

If you have installed the kernel source package, you can find a comprehensive list of all major devices with their minor and major numbers in /usr/src/linux/Documentation/devices.txt. An up-to-date list is also available on-line through the Linux Kernel Archives^[3].

The Linux kernel handles two types of devices: character and block devices. Character devices can be read byte by byte, block devices can not. Block devices are read per block (for example 4096 bytes at a time). Whether a device is a character or block device is determined by the nature of the device. For example, most storage media are block devices, and most input devices are character devices. Block devices have one distinctive advantage, namely that they can be cached. This means that commonly read or written blocks are stored in a special area of the system memory, named the cache. Memory is much faster than most storage media, so much performance can be gained by performing common block read and write operations in memory. Of course, eventually changes have to be written to the storage media to reflect the changes that were made in the cache.

There are two kinds of block devices that we are going to look into in detail, because understanding the naming of these devices is crucial for partitioning a hard disk and mounting. Almost all modern computers with an x86 architecture use ATA hard disks and CD-ROMs. Under Linux these devices are named in the following manner:

/dev/hda - master device on the first ATA channel
/dev/hdb - slave device on the first ATA channel
/dev/hdc - master device on the second ATA channel
/dev/hdd - slave device on the second ATA channel
      

On most computers with a single hard disk, the hard disk is the master device on the first ATA channel (/dev/hda), and the CD-ROM is the master device on the second ATA channel. Hard disk partitions are named after the disk that holds them plus a number. For example, /dev/hda1 is the first partition of the disk represented by the /dev/hda device file.

SCSI hard disks and CD-ROM drives follow an other naming convention. SCSI is not commonly used in most low-end machines, but USB drives and Serial ATA (SATA) drives are also represented as SCSI disks. The following device notation is used for SCSI drives:

/dev/sda - First SCSI disk
/dev/sdb - Second SCSI disk
/dev/sdc - Third SCSI disk
/dev/scd0 - First CD-ROM
/dev/scd1 - Second CD-ROM
/dev/scd2 - Third CD-ROM
      

Partitions names are constructed in the same way as ATA disks; /dev/sda1 is the first partition on the first SCSI disk.

If you use the software RAID implementation of the Linux kernel, RAID volumes are available as /dev/mdn, in which n is the volume number starting at 0.

One of the necessary configuration steps is to create a fstab file, so that the system can look up what partitions or volumes have to be mounted. The format of the /etc/fstab file is described in Section 8.8.4, “The fstab file”. As a bare minimum you will have to add entries for the / filesystem, the /proc pseudo-filesystem, the devpts pseudo-filesystem, and the swap partition.

With the sample partitioning used earlier in this chapter, you could create a /etc/fstab like this:

#  cat > /mnt/etc/fstab << EOF
/dev/hda2	 swap             swap        defaults         0   0
/dev/hda1        /                ext3        defaults         1   1
devpts           /dev/pts         devpts      gid=5,mode=620   0   0
proc             /proc            proc        defaults         0   0
EOF
      

To make the system bootable you will have to configure and install the Linux Loader (LILO). The configuration of LILO is covered in Section 19.1, “The bootloader”. For this section we will just show a sample LILO configuration, that can be used with the partition layout described in this chapter. The first step is to create the /etc/lilo.conf file:

# cat > /mnt/etc/lilo.conf << EOF
boot = /dev/hda
vga = normal
timeout = 50
image = /boot/vmlinuz
	root = /dev/hda1
	label = Slackware
	read-only
EOF
      

You can then install LILO with /mnt as the LILO root. With /mnt set as the root, LILO will use /etc/lilo.conf from the target partition:

# lilo -r /mnt
      

The configuration of networking in Slackware Linux is covered in Chapter 22, Networking configuration. This section will cover one example of a host that will use DHCP to get an IP address.

The /etc/networks file contains information about known Internet networks. Because we will get network information via DHCP, we will just use 127.0.0.1 as the local network.

# cat > /mnt/etc/networks << EOF
loopback        127.0.0.0
localnet        127.0.0.0
EOF
      

Although we will get a hostname via DHCP, we will still set up a temporary hostname:

# cat > /mnt/etc/HOSTNAME << EOF
sugaree.example.net
EOF
      

Now that the hostname is configured, the hostname and localhost should also be made resolvable, by creating a /etc/hosts database:

# cat > /mnt/etc/hosts << EOF
127.0.0.1	localhost
127.0.0.1	sugaree.example.net	sugaree	
EOF
      

We do not have to create a /etc/resolv.conf, since it will be created automatically by dhcpcd, the DHCP client. So, finally, we can set up the interface in /etc/rc.d/rc.inet1.conf:

# cat > /mnt/etc/rc.d/rc.inet1.conf << EOF
IPADDR[0]=""
NETMASK[0]=""
USE_DHCP[0]="yes"
DHCP_HOSTNAME[0]=""
EOF
      

You may wat to make a backup of the old rc.inet1.conf file first, because it contains many useful comments. Or you can use sed to change this specific option:

# sed -i 's/USE_DHCP\[0\]=""/USE_DHCP[0]="yes"/' \
   /mnt/etc/rc.d/rc.inet1.conf
      

Depending on the purpose of the system that is being installed, it should be decided which initialization scripts should be started. The number of services that are available depends on what packages you have installed. You can get get a list of available scripts with ls:

# ls -l /mnt/etc/rc.d/rc.*
      

If the executable bits are set on a script, it will be started, otherwise it will not. Obviously you should keep essential scripts executable, including the runlevel-specific scripts. You can set the executable bit on a script with:

# chmod +x /mnt/etc/rc.d/rc.scriptname
      

Or remove it with:

# chmod -x /mnt/etc/rc.d/rc.scriptname
      

GNU/Linux uses a cache for loading dynamic libraries. Besides that many programs rely on generic version numbers of libraries (e.g. /usr/lib/libgtk-x11-2.0.so, rather than /usr/lib/libgtk-x11-2.0.so.0.600.8). The cache and library symlinks can be updated in one ldconfig run:

# chroot /mnt /sbin/ldconfig
      

You may not know the chroot command; it is a command that executes a command with a different root than the active root. In this example the root directory is changed to /mnt, and from there chroot runs /sbin/ldconfig binary. After the command has finished the system will return to the shell, which uses the original root. To use chroot with other commands this ldconfig command has to be executed once first, because without initializing the dynamic linker run-time bindings most commands will not execute, due to unresolvable library dependencies.

Now that the dynamic library cache and links are set up, you can execute commands on the installed system. We will make use of this to set the root password. The passwd command can be used to set the password for an existing user (the root user is part of the initial /etc/passwd file). We will use the chroot command again to execute the command on the target partition:

# chroot /mnt /usr/bin/passwd root
      

On UNIX-like systems it is important to set the timezone correctly, because it is used by various parts of the system. For instance, the timezone is used by NTP to synchronize the system time correctly, or by different networking programs to compute the time difference between a client and a server. On Slackware Linux the timezone can be set by linking /etc/localtime to a timezone file. You can find the timezone for your region by browsing the /mnt/usr/share/zoneinfo directory. Most timezones can be found in the subdirectories of their respective regions. For example to use Europe/Amsterdam as the timezone, you can execute the following commands:

# cd /mnt
# rm -rf etc/localtime
# ln -sf /usr/share/zoneinfo/Europe/Amsterdam etc/localtime
      

After setting the time zone, programs still do not know whether the hardware clock is set to the local time, or to the Coordinated Universal Time (UTC) standard. If you use another operating system on the same machine that does not use UTC it is best to set the hardware clock to the local time. On UNIX-like systems it is a custom to set the system time to UTC. You can set what time the system clock uses, by creating the file /etc/hardwareclock, and putting the word localtime in it if your clock is set to the local time, or UTC when it is set to UTC. For example, to use UTC, you can create /etc/hardwareclock in the following manner:

# echo "UTC" > /mnt/etc/hardwareclock
      

If you are going to use X11, you will have to initialize the font cache for TrueType and Type1 fonts. This can be done with the fc-cache command:

# chroot /mnt /usr/bin/fc-cache
      

The first group of shortcuts have characters as their logic unit, meaning that they allow command line editing operations on characters. Table 7.1, “Moving by character” provides an overview of the shortcuts that are used to move through a line by character.

These shortcuts are simple, and don't do anything unexpected. Suppose that you have typed the following line:

find ~/music -name '*.ogg' - -print 
      

The cursor will be at the end. You can now move to the start of the line by holding Ctrl-b:

find ~/music - -name '*.ogg' -print
      

Likewise, you can go back again to the end by holding Ctrl-f. There is an error in this line, since there is one erroneous dash. To remove this dash, you can use one of the character deletion shortcuts.

You can delete the dash in two manners. The first way is to move the cursor to the dash:

find ~/music - -name '*.ogg' -print
      

and then press Ctrl-d twice. This will delete the dash character, and the space that follows the dash:

find ~/music -name '*.ogg' -print
      

Looking at the original fragment, the other approach is to position the cursor on the space after the dash:

find ~/music -  -name '*.ogg' -print
      

and then press Ctrl-h twice to delete the two preceding characters, namely the dash and the space before the dash. The result will be the same, except that the cursor will move:

find ~/music -name '*.ogg' -print
      

One of the nice features of most modern shells is that you can transpose (swap) characters. This is handy if you make a typing error in which two characters are swapped. Table 7.3, “Swapping characters” lists the shortcut for transposing characters.

Suppose that you have typed the following command:

cat myreport.ttx
      

The extension contains a typing error if you intended to cat myreport.txt. This can be corrected with the character transpose shortcut. First move to the second character of the pair of characters that are in the wrong order:

cat myreport.ttx
      

You can then press Ctrl-t. The characters will be swapped, and the cursor will be put behind the swapped characters:

cat myreport.txt 
      

It if often tedious to move at character level. Fortunately the Korn and Bash shells can also move through lines at a word level. Words are sequences of characters that are separated by a special character, such as a space. Table 7.4, “Moving by word” summarizes the shortcuts that can be used to navigate through a line by word.

As you can see the letters in these shortcuts are equal to those of moving forward and backwards by character. The movement logic is a bit curious. Moving forward puts the cursor to the end of the current word, not to the first character of the next word as you may have predicted. Let's look at a quick example. In the beginning the cursor is on the first character of the line.

find ~/music -name '*.ogg' -print
      

Pressing Esc f will move the cursor behind the last character of the first word, which is find in this case:

find ~/music -name '*.ogg' -print
      

Going forward once more will put the cursor behind ~/music:

find ~/music -name '*.ogg' -print
      

Backwards movement puts the cursor on the first character of the current word, or on the first character of the previous word if the cursor is currently on the first character of a word. So, moving back one word in the previous example will put the cursor on the first letter of “music”:

find ~/music -name '*.ogg' -print
      

Deleting words works equal to moving by word, but the characters that are encountered are deleted. Table 7.5, “Deleting words” lists the shortcuts that are used to delete words.

Finally, there are some shortcuts that are useful to manipulate words. These shortcuts are listed in Table 7.6, “Modifying words”.

Transposition swaps words. If normal words are used, it's behavior is predictable. For instance, if we have the following line with the cursor on “two”

one two three
      

Word transposition will swap “two” and “one”:

two one three
      

But if there are any non-word characters, the shell will swap the word with the previous word while preserving the order of non-word characters. This is very handy for editing arguments to commands. Suppose that you made an error, and mixed up the file extension you want to look for, and the print parameter:

find ~/music -name '*.print' -ogg
      

You can fix this by putting the cursor on the second faulty word, in this case “ogg”, and transposing the two words. This will give the result that we want:

find ~/music -name '*.ogg' -print
      

Finally, there are some shortcuts that change the capitalization of words. The Alt-u shortcut makes all characters uppercase, starting at the current cursor position till the end of the word. So, if we have the lowercase name “alice”, uppercasing the name with the cursor on “i” gives “alICE”. Alt-l has the same behavior, but changes letters to lowercase. So, using Alt-l on “alICE” with the cursor on “I” will change the string to “alice”. Alt-c changes just the character the cursor is on, or the next word character that is encountered, to uppercase. For instance, pressing Alt-c with the cursor on “a” in “alice” will yield “Alice”.

The highest level we can edit is the line itself. Table 7.7, “Moving through lines” lists the two movement shortcuts.

Suppose that the cursor is somewhere halfway a line:

find ~/music -name '*.ogg' -print
      

Pressing Ctrl-e once will move the cursor to the end of the line:

find ~/music -name '*.ogg' -print 
      

Pressing Ctrl-a will move the cursor to the beginning of the line:

find ~/music -name '*.ogg' -print
      

You can also delete characters by line level. The shortcuts are listed in Table 7.8, “Deleting lines”. These shortcuts work like movement, but deletes all characters that are encountered. Ctrl-k will delete the character the cursor is on, but Ctrl-x Backspace will not. Moving to the beginning of the line with Ctrl-a, followed by Ctrl-k, is a fast trick to remove a line completely.

As you can see in the table above the “*” character matches a string of characters. For example, *.html matches everything ending with .html, d*.html matches everything starting with a d and ending with .html.

Suppose that you would like to list all files in the current directory with the .html extension, the following command will do the job:

$ ls *.html
book.html        installation.html     pkgmgmt.html  usermgmt.html
filesystem.html  internet.html         printer.html  xfree86.html
gfdl.html        introduction.html     proc.html
help.html        slackware-basics.html shell.html
      

Likewise we could remove all files starting with an in:

$ rm in*
      

The “?” wildcard works as the “*” wildcard, but matches single characters. Suppose that we have three files, file1.txt, file2.txt and file3.txt. The string file?.txt matches all three of these files, but it does not match file10.txt (“10” are two characters).

The “[]” wildcard matches every character between the brackets. Suppose we have the files from the previous example, file1.txt, file2.txt and file3.txt. The string file[23].txt matches file2.txt and file3.txt, but not file1.txt.

The shell allows you to take use of stdin and stdout using the “<” and “>”. Data is redirected in which way the sharp bracket points. In the following example we will redirect the md5 summaries calculated for a set of files to a file named md5sums:

$ md5sum * > md5sums
$ cat md5sums 
6be249ef5cacb10014740f61793734a8  test1
220d2cc4d5d5fed2aa52f0f48da38ebe  test2
631172a1cfca3c7cf9e8d0a16e6e8cfe  test3
      

As we can see in the cat output the output of the md5sum * output was redirected to the md5sums file. We can also use redirection to provide input to a command:

$ md5sum < test1
6be249ef5cacb10014740f61793734a8  -
      

This feeds the contents of the test1 to md5sum.

You can also connect the input and output of commands using so-called pipes. A pipe between commands can be made with the “|” character. Two or more combined commands are called a pipeline. Figure 7.2, “A pipeline” shows a schematic overview of a pipeline consisting of two commands.

Figure 7.2. A pipeline

The “syntax” of a pipeline is: command1 | command2 ... | commandn. If you know how the most basic UNIX-like commands work you can now let these commands work together. Let's look at a quick example:

$ cat /usr/share/dict/american-english | grep "aba" | wc -l
123
      

The first command, cat, reads the dictionary file /usr/share/dict/american-english. The output of the cat command is piped to grep, which prints out all files containing the phrase “aba”. In turn, the output of “grep” is piped to wc -l, which counts the number of lines it receives from stdin. Finally, when the stream is finished wc prints the number of lines it counted. So, combined three commands to count the number of words containing the phrase “aba” in this particular dictionary.

There are hundreds of small utilities that handle specific tasks. As you can imagine, together these commands provide a very powerful toolbox by making combinations using pipes.

The filesystem consists of two types of elements: data and metadata. The metadata describes the actual data blocks that are on the disk. Most filesystems use information nodes (inodes) to provide store metadata. Most filesystems store the following data in their inodes:

If you are not a UNIX or Linux afficiendo, these names will probably sound bogus to you, but we will clear them up in the following sections. At any rate, you can probably deduct the relation between inodes and data from this table, and specifically the i_block field: every inode has pointers to the data blocks that the inode provides information for. Together, the inode and data blocks are the actual file on the filesystem.

You may wonder by now where the names of files (and directories) reside, since there is no file name field in the inode. Actually, the names of the files are separated from the inode and data blocks, which allows you to do groovy stuff, like giving the same file more than one name. The filenames are stored in so-called directory entries. These entries specify a filename and the inode of the file. Since directories are also represented by inodes, a directory structure can also be constructed in this manner.

We can simply show how this all works by illustrating what the kernel does when we execute the command cat /home/daniel/note.txt

As we have described earlier, Linux is a multi-user system. This means that each user has his/her own files (that are usually located in the home directory). Besides that users can be members of a group, which may give the user additional privileges.

As you have seen in the inode field table, every file has a owner and a group. Traditional UNIX access control gives read, write, or executable permissions to the file owner, file group, and other users. These permissions are stored in the mode field of the inode. The mode field represents the file permissions as a four digit octal number. The first digit represents some special options, the second digit stores the owner permissions, the third the group permissions, and the fourth the permissions for other users. The permissions are established by digit by using or adding one of the number in Table 8.2, “Meaning of numbers in the mode octet”

Now, suppose that a file has mode 0644, this means that the file is readable and writable by the owner (6), and readable by the file group (4) and others (4).

Most users do not want to deal with octal numbers, so that is why many utilities can also deal with an alphabetic representation of file permissions. The letters that are listed in Table 8.2, “Meaning of numbers in the mode octet” between parentheses are used in this notation. In the following example information about a file with 0644 permissions is printed. The numbers are replaced by three rwx triplets (the first character can list special mode options).

$ ls -l note.txt
-rw-r--r--  1 daniel daniel 5 Aug 28 19:39 note.txt
      

Over the years these traditional UNIX permissions have proven not to be sufficient in some cases. The POSIX 1003.1e specification aimed to extend the UNIX access control model with Access Control Lists (ACLs). Unfortunately this effort stalled, though some systems (like GNU/Linux) have implemented ACLs^[4]. Access control lists follow the same semantics as normal file permissions, but give you the opportunity to add rwx triplets for additional users and groups.

The following example shows the access control list of a file. As you can see, the permissions look like normal UNIX permissions (the access rights for the user, group, and others are specified). But there is also an additional entry for the user joe.

user::rwx
user:joe:r--
group::---
mask::r--
other::---
      

To make matters even more complex (and sophisticated), some GNU/Linux systems add more fine-grained access control through Mandatory Access Control Frameworks (MAC) like SELinux and AppArmor. But these access control frameworks are beyond the scope of this book.

A directory entry that points to an inode is named a hard link. Most files are only linked once, but nothing holds you from linking a file twice. This will increase the links_count field of the inode. This is a nice way for the system to see which inodes and data blocks are free to use. If links_count is set to zero, the inode is not referred to anymore, and can be reclaimed.

Figure 8.1. The structure of a hard link

Hard links have two limitations. First of all, hard links can not interlink between filessystems, since they point to inodes. Every filesystem has its own inodes and corresponding inode numbers. Besides that, most filesystems do not allow you to create hard links to directories. Allowing creation of hard links to directories could produce directory loops, potentially leading to deadlocks and filesystem inconsistencies. In addition to that, most implementations of rm and rmdir do not know how to deal with such extra directory hard links.

Symbolic links do not have these limitations, because they point to file names, rather than inodes. When the symbolic link is used, the operating system will follow the path to that link. Symbolic links can also refer to a file that does not exist, since it just contains a name. Such links are called dangling links.

Figure 8.2. The structure of a symbolic link

Note

If you ever get into system administration, it is good to be aware of the security implications of hard links. If the /home directory is on the same filesystem as any system binaries, a user will be able to create hard links to binaries. In the case that a vulnerable program is upgraded, the link in the user's home directory will keep pointing to the old program binary, effectively giving the user continuing access to a vulnerable binary. For this reason it is a good idea to put any directories that users can write to on different filesystems. In practice, this means that it is a good idea to put at least /home and /tmp on separate filesystems.

One of the most common things that you will want to do is to list all or certain files. The ls command serves this purpose very well. Using ls without any arguments will show the contents of the actual directory:

$ ls
dns.txt  network-hosts.txt  papers
      

If you use a GNU/Linux distribution, you may also see some fancy coloring based on the type of file. The standard output is handy to skim through the contents of a directory, but if you want more information, you can use the -l parameter. This provides a so-called long listing for each file:

$ ls -l
total 36
-rw-rw-r--  1 daniel daniel 12235 Sep  4 15:56 dns.txt
-rw-rw-r--  1 daniel daniel  7295 Sep  4 15:56 network-hosts.txt
drwxrwxr-x  2 daniel daniel  4096 Sep  4 15:55 papers
      

This gives a lot more information about the three directory entries that we have found with ls. The first column shows the file permissions. The line that shows the papers entry starts with a “d”, meaning that this entry represents a directory. The second column shows the number of hard links pointing to the inode that a directory entry points to. If this is higher than 1, there is some other filename for the same file. Directory entries usually have at least two hard links, namely the link in the parent directory and the link in the directory itself (each directory has a . entry, which refers to the directory itself). The third and the fourth columns list the file owner and group respectively. The fifth column contains the file size in bytes. The sixth column the last modification time and date of the file. And finally, the last column shows the name of this entry.

Files that start with a period (.) will not be shown by most applications, including ls. You can list these files too, by adding the -a option to ls:

$ ls -la
total 60
drwxrwxr-x   3 daniel daniel  4096 Sep 11 10:01 .
drwx------  88 daniel daniel  4096 Sep 11 10:01 ..
-rw-rw-r--   1 daniel daniel 12235 Sep  4 15:56 dns.txt
-rw-rw-r--   1 daniel daniel  7295 Sep  4 15:56 network-hosts.txt
drwxrwxr-x   2 daniel daniel  4096 Sep  4 15:55 papers
-rw-rw-r--   1 daniel daniel     5 Sep 11 10:01 .settings
      

As you can see, three more entries have appeared. First of all, the .settings file is now shown. Besides that you can see two additional directory entries, . and ... These represent the current directory and the parent directory respectively.

Earlier in this chapter (Section 8.1.1, “inodes, directories and data”) we talked about inodes. The inode number that a directory entry points to can be shown with the -i parameter. Suppose that I have created a hard link to the inode that points to the same inode as dns.txt, they should have the same inode number. The following ls output shows that this is true:

$ ls -i dns*
3162388 dns-newhardlink.txt
3162388 dns.txt
      

Sometimes you will need some help to determine the type of a file. This is where the file utility becomes handy. Suppose that I find a file named HelloWorld.class somewhere on my disk. I suppose that this is a file that holds Java bytecode, but we can use file to check this:

$ file HelloWorld.class
HelloWorld.class: compiled Java class data, version 49.0
      

That is definitely Java bytecode. file is quite smart, and handles most things you throw at it. For instance, you could ask it to provide information about a device node:

$ file /dev/zero
/dev/zero: character special (1/5)
      

Or a symbolic link:

$ file /usr/X11R6/bin/X
/usr/X11R6/bin/X: symbolic link to `Xorg'
      

If you are rather interested in the file /usr/X11R6/bin/X links to, you can use the -L option of file:

$ file -L /usr/X11R6/bin/X
/usr/X11R6/bin/X: setuid writable, executable, regular file, no read permission
      

You may wonder why file can determine the file type relatively easy. Most files start of with a so-called magic number, this is a unique number that tells programs that can read the file what kind of file it is. The file program uses a file which describes many file types and their magic numbers. For instance, the magic file on my system contains the following lines for Java compiled class files:

# Java ByteCode
# From Larry Schwimmer (schwim@cs.stanford.edu)
0       belong          0xcafebabe      compiled Java class data,
>6      beshort x       version %d.
>4      beshort x       \b%d
      

This entry says that if a file starts with a long (32-bit) hexadecimal magic number 0xcafebabe^[5], it is a file that holds “compiled Java class data”. The short that follows determines the class file format version.

While we will look at more advanced file integrity checking later, we will have a short look at the cksum utility. cksum can calculate a cyclic redundancy check (CRC) for an input file. This is a mathematically sound method for calculating a unique number for a file. You can use this number to check whether a file is unchanged (for example, after downloading a file from a server). You can specify the file to calculate a CRC for as a parameter to cksum, and cksum will print the CRC, the file size in bytes, and the file name:

$ cksum myfile
1817811752 22638 myfile
      

Slackware Linux also provides utilities for calculating checksums based on one-way hashes (for instance MD5 or SHA-1).

Since most files on UNIX systems are usually text files, they are easy to view from a character-based terminal or terminal emulator. The most primitive way of looking at the contents of a file is by using cat. cat reads files that were specified as a parameter line by line, and will write the lines to the standard output. So, you can write the contents of the file note.txt to the terminal with cat note.txt. While some systems and most terminal emulators provide support for scrolling, this is not a practical way to view large files. You can pipe the output of cat to the less paginator:

$ cat note.txt | less
      

or let less read the file directly:

$ less note.txt
      

The less paginator lets you scroll forward and backward through a file. Table 8.3, “less command keys” provides an overview of the most important keys that are used to control less

The command keys that can be quantized can be prefixed by a number. For instance 11j scrolls forward eleven lines, and 3n searches the third match of the previously specified regular expression.

Slackware Linux also provides an alternative to less, the older “more” command. We will not go into more here, less is more comfortable, and also more popular these days.

The ls -l output that we have seen earlier provides information about the size of a file. While this usually provides enough information about the size of files, you might want to gather information about collections of files or directories. This is where the du command comes in. By default, du prints the file size per directory. For example:

$ du ~/qconcord
72      /home/daniel/qconcord/src
24      /home/daniel/qconcord/ui
132     /home/daniel/qconcord
      

By default, du represents the size in 1024 byte units. You can explicitly specify that du should use 1024 byte units by adding the -k flag. This is useful for writing scripts, because some other systems default to using 512-byte blocks. For example:

$ du -k ~/qconcord
72	/home/daniel/qconcord/src
24	/home/daniel/qconcord/ui
132	/home/daniel/qconcord
      

If you would also like to see per-file disk usage, you can add the -a flag:

$ du -k -a ~/qconcord
8       /home/daniel/qconcord/ChangeLog
8       /home/daniel/qconcord/src/concordanceform.h
8       /home/daniel/qconcord/src/textfile.cpp
12      /home/daniel/qconcord/src/concordancemainwindow.cpp
12      /home/daniel/qconcord/src/concordanceform.cpp
8       /home/daniel/qconcord/src/concordancemainwindow.h
8       /home/daniel/qconcord/src/main.cpp
8       /home/daniel/qconcord/src/textfile.h
72      /home/daniel/qconcord/src
12      /home/daniel/qconcord/Makefile
16      /home/daniel/qconcord/ui/concordanceformbase.ui
24      /home/daniel/qconcord/ui
8       /home/daniel/qconcord/qconcord.pro
132     /home/daniel/qconcord
      

You can also use the name of a file or a wildcard as a parameter. But this will not print the sizes of files in subdirectories, unless -a is used:

$ du -k -a ~/qconcord/*
8       /home/daniel/qconcord/ChangeLog
12      /home/daniel/qconcord/Makefile
8       /home/daniel/qconcord/qconcord.pro
8       /home/daniel/qconcord/src/concordanceform.h
8       /home/daniel/qconcord/src/textfile.cpp
12      /home/daniel/qconcord/src/concordancemainwindow.cpp
12      /home/daniel/qconcord/src/concordanceform.cpp
8       /home/daniel/qconcord/src/concordancemainwindow.h
8       /home/daniel/qconcord/src/main.cpp
8       /home/daniel/qconcord/src/textfile.h
72      /home/daniel/qconcord/src
16      /home/daniel/qconcord/ui/concordanceformbase.ui
24      /home/daniel/qconcord/ui
      

If you want to see the total sum of the disk usage of the files and subdirectories that a directory holds, use the -s flag:

$ du -k -s ~/qconcord
132     /home/daniel/qconcord
      

The ls command that we have looked at in Section 8.2.1, “Listing files” can also be used to list directories in various ways. As we have seen, the default ls output includes directories, and directories can be identified using the first output column of a long listing:

$ ls -l
total 36
-rw-rw-r--  1 daniel daniel 12235 Sep  4 15:56 dns.txt
-rw-rw-r--  1 daniel daniel  7295 Sep  4 15:56 network-hosts.txt
drwxrwxr-x  2 daniel daniel  4096 Sep  4 15:55 papers
      

If a directory name, or if wildcards are specified, ls will list the contents of the directory, or the directories that match the wildcard respectively. For example, if there is a directory papers, ls paper* will list the contents of this directory paper. This is often annoying if you would just like to see the matches, and not the contents of the matching directories. The -d avoid that this recursion happens:

$ ls -ld paper*
drwxrwxr-x  2 daniel daniel  4096 Sep  4 15:55 papers
      

You can also recursively list the contents of a directory, and its subdirectories with the -R parameter:

$ ls -R
.:
dns.txt  network-hosts.txt  papers

./papers:
cs  phil

./papers/cs:
entr.pdf

./papers/phil:
logics.pdf
      

UNIX provides the mkdir command to create directories. If a relative path is specified, the directory is created in the current active directory. The basic syntax is very simple: mkdir <name>, for example:

$ mkdir mydir
      

By default, mkdir only creates one directory level. So, if you use mkdir to create mydir/mysubdir, mkdir will fail if mydir does not exist already. If you would like to create both directories at once, use the -p parameter:

$ mkdir -p mydir/mysubdir
      

rmdir removes a directory. Its behavior is comparable to mkdir. rmdir mydir/mysubdir removes mydir/subdir, while rmdir -p mydir/mysubdir removes mydir/mysubdir and then mydir.

If a subdirectory that we want to remove contains directory entries, rmdir will fail. If you would like to remove a directory, including all its contents, use the rm command instead.

Files and directories can be copied with the cp command. In its most basic syntax the source and the target file are specified. The following example will make a copy of file1 named file2:

$ cp file1 file2
      

It is not surprising that relative and absolute paths do also work:

$ cp file1 somedir/file2
$ cp file1 /home/joe/design_documents/file2
      

You can also specify a directory as the second parameter. If this is the case, cp will make a copy of the file in that directory, giving it the same file name as the original file. If there is more than one parameter, the last parameter will be used as the target directory. For instance

$ cp file1 file2 somedir
      

will copy both file1 and file2 to the directory somedir. You can not copy multiple files to one file. You will have to use cat instead:

$ cat file1 file2 > combined_file
      

You can also use cp to copy directories, by adding the -R. This will recursively copy a directory and all its subdirectories. If the target directory exists, the source directory or directories will be placed under the target directory. If the target directory does not exist, it will be created if there is only one source directory.

$ cp -r mytree tree_copy
$ mkdir trees
$ cp -r mytree trees
      

After executing these commands, there are two copies of the directory mytree, tree_copy and trees/mytree. Trying to copy two directories to a nonexistent target directory will fail:

$ cp -R mytree mytree2 newdir
usage: cp [-R [-H | -L | -P]] [-f | -i] [-pv] src target
       cp [-R [-H | -L | -P]] [-f | -i] [-pv] src1 ... srcN directory
      

	Note
	Traditionally, the -r has been available on many UNIX systems to recursively copy directories. However, the behavior of this parameter can be implementation-dependent, and the Single UNIX Specification version 3 states that it may be removed in future versions of the standard.

When you are copying files recursively, it is a good idea to specify the behavior of what cp should do when a symbolic link is encountered explicitly, if you want to use cp in portable scripts. The Single UNIX Specification version 3 does not specify how they should be handled by default. If -P is used, symbolic links will not be followed, effectively copying the link itself. If -H is used, symbolic links specified as a parameter to cp may be followed, depending on the type and content of the file. If -L is used, symbolic links that were specified as a parameter to cp and symbolic links that were encountered while copying recursively may be followed, depending on the content of the file.

If you want to preserve the ownership, SGID/SUID bits, and the modification and access times of a file, you can use the -p flag. This will try to preserve these properties in the file or directory copy. Good implementations of cp provide some additional protection as well - if the target file already exists, it may not be overwritten if the relevant metadata could not be preserved.

The UNIX command for moving files, mv, can move or rename files or directories. What actually happens depends on the location of the files or directories. If the source and destination files or directories are on the same filesystem, mv usually just creates new hard links, effectively renaming the files or directories. If both are on different filesystems, the files are actually copied, and the source files or directories are unlinked.

The syntax of mv is comparable to cp. The most basic syntax renames file1 to file2:

$ mv file1 file2
      

The same syntax can be used for two directories as well, which will rename the directory given as the first parameter to the second parameter.

When the last parameter is an existing directory, the file or directory that is specified as the first parameter, is copied to that directory. In this case you can specify multiple files or directories as well. For instance:

$ targetdir
$ mv file1 directory1 targetdir
      

This creates the directory targetdir, and moves file1 and directory1 to this directory.

Files and directories can be removed with the rm (1) command. This command unlinks files and directories. If there are no other links to a file, its inode and disk blocks can be reclaimed for new files. Files can be removed by providing the files that should be removed as a parameter to rm (1) . If the file is not writable, rm (1) will ask for confirmation. For instance, to remove file1 and file2, you can execute:

$ rm file1 file2
      

If you have to remove a large number of files that require a confirmation before they can be deleted, or if you want to use rm (1) to remove files from a script that will not be run on a terminal, add the -f parameter to override the use of prompts. Files that are not writable, are deleted with the -f flag if the file ownership allows this. This parameter will also suppress printing of errors to stderr if a file that should be removed was not found.

Directories can be removed recursively as well with the -r parameter. rm (1) will traverse the directory structure, unlinking and removing directories as they are encountered. The same semantics are used as when normal files are removed, as far as the -f flag is concerned. To give a short example, you can recursively remove all files and directories in the notes directory with:

$ rm -r notes
      

Since rm (1) command uses the unlink (2) function, data blocks are not rewritten to an uninitialized state. The information in data blocks is only overwritten when they are reallocated and used at a later time. To remove files including their data blocks securely, some systems provide a shred (1) command that overwrites data blocks with random data. But this is not effective on many modern (journaling) filesystems, because they don't write data in place.

The unlink (1) commands provides a one on one implementation of the unlink (2) function. It is of relatively little use, because it can not remove directories.

As we have seen earlier, every file has an owner (user) ID and a group ID stored in the inode. The chown (1) command can be used to set these fields. This can be done by the numeric IDs, or their names. For instance, to change the owner of the file note.txt to john, and its group to staff, the following command is used:

$ chown john:staff note.txt
      

You can also omit either components, to only set one of both fields. If you want to set the user name, you can also omit the colon. So, the command above can be split up in two steps:

$ chown john note.txt
$ chown :staff note.txt
      

If you want to change the owner of a directory, and all the files or directories it holds, you can add the -R to chown (1) :

$ chown -R john:staff notes
      

If user and group names were specified, rather than IDs, the names are converted by chown (1) . This conversion usually relies on the system-wide password database. If you are operating on a filesystem that uses another password database (e.g. if you mount a root filesystem from another system for recovery), it is often useful to change file ownership by the user or group ID. In this manner, you can keep the relevant user/group name to ID mappings in tact. So, changing the ownership of note to UID 1000 and GUID 1000 is done in the following (predictable) manner:

$ chown 1000:1000 note.txt
      

After reading the introduction to filesystem permissions in Section 8.1.2, “File permissions”, changing the permission bits that are stored in the inode is fairly easy with the chmod (1) command. chmod (1) accepts both numeric and symbolic representations of permissions. Representing the permissions of a file numerically is very handy, because it allows setting all relevant permissions tersely. For instance:

$ chmod 0644 note.txt
      

Make note.txt readable and writable for the owner of the file, and readable for the file group and others.

Symbolic permissions work with addition or subtraction of rights, and allow for relative changes of file permissions. The syntax for symbolic permissions is:

[ugo][-+][rwxst]
      

The first component specifies the user classes to which the permission change applies (user, group or other). Multiple characters of this component can be combined. The second component takes away rights (-), or adds rights (+). The third component is the access specifier (read, write, execute, set UID/GID on execution, sticky). Multiple components can be specified for this component too. Let's look at some examples to clear this up:

ug+rw        # Give read/write rights to the file user and group
chmod go-x   # Take away execute rights from the file group and others.
chmod ugo-wx # Disallow all user classes to write to the file and to
             # execute the file.
      

These commands can be used in the following manner with chmod:

$ chmod ug+rw note.txt
$ chmod go-x script1.sh
$ chmod ugo-x script2.sh
      

Permissions of files and directories can be changed recursively with the -R. The following command makes the directory notes world-readable, including its contents:

$ chmod -R ugo+r notes
      

Extra care should be taken with directories, because the x flag has a special meaning in a directory context. Users that have execute rights on directories can access a directory. User that don't have execute rights on directories can not. Because of this particular behavior, it is often easier to change the permissions of a directory structure and its files with help of the find (1) command .

There are a few extra permission bits that can be set that have a special meaning. The SUID and SGID are the most interesting bits of these extra bits. These bits change the active user ID or group ID to that of the owner or group of the file when the file is executed. The su(1) command is a good example of a file that usually has the SUID bit set:

$ ls -l /bin/su
-rwsr-xr-x  1 root root 60772 Aug 13 12:26 /bin/su
      

This means that the su command runs as the user root when it is executed. The SUID bit can be set with the s modifier. For instance, if the SUID bit was not set on /bin/su this could be done with:

$ chmod u+s /bin/su
      

	Note
	Please be aware that the SUID and SGID bits have security implications. If a program with these bits set contain a bug, it may be exploited to get privileges of the file owner or group. For this reason it is good manner to keep the number of files with the SUID and SGID bits set to an absolute minimum.

The sticky bit is also interesting when it comes to directory. It disallows users to rename of unlink files that they do not own, in directories that they do have write access to. This is usually used on world-writeable directories, like the temporary directory (/tmp) on many UNIX systems. The sticky tag can be set with the t modifier:

$ chmod g+t /tmp
      

The question that remains is what initial permissions are used when a file is created. This depends on two factors: the mode flag that was passed to the open(2) system call, that is used to create a file, and the active file creation mask. The file creation mask can be represented as an octal number. The effective permissions for creating the file are determined as mode & ~mask. Or, if represented in an octal fashion, you can substract the digits of the mask from the mode. For instance, if a file is created with permissions 0666 (readable and writable by the file user, file group, and others), and the effective file creation mask is 0022, the effective file permission will be 0644. Let's look at anothere example. Suppose that files are still created with 0666 permissions, and you are more paranoid, and want to take away all read and write permissions for the file group and others. This means you have to set the file creation mask to 0066, because substracting 0066 from 0666 yields 0600

The effective file creation mask can be queried and set with the umask command, that is normally a built-in shell command. The effective mask can be printed by running umask without any parameters:

$ umask
0002
      

The mask can be set by giving the octal mask number as a parameter. For instance:

$ umask 0066
      

We can verify that this works by creating an empty file:

$ touch test
$ ls -l test
-rw-------  1 daniel daniel 0 Oct 24 00:10 test2
      

Access Control lists (ACLs) are an extension to traditional UNIX file permissions, that allow for more fine-grained access control. Most systems that support filesystem ACLs implement them as they were specified in the POSIX.1e and POSIX.2c draft specifications. Notable UNIX and UNIX-like systems that implement ACLs according to this draft are FreeBSD, Solaris, and Linux.

As we have seen in Section 8.1.2, “File permissions” access control lists allows you to use read, write and execute triplets for additional users or groups. In contrast to the traditional file permissions, additional access control lists are note stored directly in the node, but in extended attributes that are associated with files. Two thing to be aware of when you use access control lists is that not all systems support them, and not all programs support them.

$ ls -l index.html
-rw-r-----+ 1 daniel daniel 3254 2006-10-31 17:11 index.html
	

$ getfacl index.html
# file: index.html
# owner: daniel
# group: daniel
user::rw-
group::---
group:www-data:r--
mask::r--
other::---
	

$ getfacl links.html
# file: links.html
# owner: daniel
# group: daniel
user::rw-
group::rw-                      #effective:r--
group:www-data:rw-              #effective:r--
mask::r--
other::---
	

$ getfacl reports
# file: reports
# owner: daniel
# group: daniel
user::rwx
group::r-x
group:www-data:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x
default:group:www-data:r-x
default:mask::r-x
default:other::---
	

$ touch reports/test
$ getfacl reports/test
# file: reports/test
# owner: daniel
# group: daniel
user::rw-
group::r-x                      #effective:r--
group:www-data:r-x              #effective:r--
mask::r--
other::---
	

$ setfacl -m group:friends:rw- report.txt
	  

$ getfacl report.txt
# file: report.txt
# owner: daniel
# group: daniel
user::rw-
group::r--
group:friends:rw-
mask::rw-
other::r--
	  

$ setfacl -m group:friends:rw-,group:foes:--- report.txt
	  

$ setfacl -x group:friends: report.txt
	  

$ setfacl --set user::rw-,group::r--,other:---,group:friends:rwx report.txt
	  

$ setfacl -b -m group:friends:rw- report.txt
	  

$ setfacl -d -m group:friends:rwx reports
$ getfacl reports
# file: reports
# owner: daniel
# group: daniel
user::rwx
group::r-x
other::r-x
default:user::rwx
default:group::r-x
default:group:friends:rwx
default:mask::rwx
default:other::r-x
	  

$ getfacl report.txt > ref
	  

$ setfacl -M ref report2.txt
	  

$ setfacl -b -M ref report2.txt
	  

$ getfacl report.txt | setfacl -b -M - report2.txt
	  

The find command is without doubt the most comprehensive utility to find files on UNIX systems. Besides that it works in a simple and predictable way: find will traverse the directory tree or trees that are specified as a parameter to find. Besides that a user can specify an expression that will be evaluated for each file and directory. The name of a file or directory will be printed if the expression evaluates to true. The first argument that starts with a dash (-), exclamation mark (!, or an opening parenthesis ((, signifies the start of the expression. The expression can consist of various operands. To wrap it up, the syntax of find is: find paths expression.

The simplest use of find is to use no expression. Since this matches every directory and subdirectory entry, all files and directories will be printed. For instance:

$ find .
.
./economic
./economic/report.txt
./economic/report2.txt
./technical
./technical/report2.txt
./technical/report.txt
      

You can also specify multiple directories:

$ find economic technical
economic
economic/report.txt
economic/report2.txt
technical
technical/report2.txt
technical/report.txt
      

8.6.1.1. Operands that limit by object name or type

One common scenario for finding files or directories is to look them up by name. The -name operand can be used to match objects that have a certain name, or match a particular wildcard. For instance, using the operand -name 'report.txt' will only be true for files or directories with the name report.txt. For example:

$ find economic technical -name 'report.txt'
economic/report.txt
technical/report.txt
	

The same thing holds for wildcards:

$ find economic technical -name '*2.txt'
economic/report2.txt
technical/report2.txt
	

	Note
	When using find you will want to pass the wildcard to find, rather than letting the shell expand it. So, make sure that patterns are either quoted, or that wildcards are escaped.

It is also possible to evaluate the type of the object with the -type c operand, where c specifies the type to be matched. Table 8.5, “Parameters for the '-type' operand” lists the various object types that can be used.

Table 8.5. Parameters for the '-type' operand

Parameter	Meaning
b	Block device file
c	Character device file
d	Directory
f	Regular file
l	Symbolic link
p	FIFO
s	Socket

So, for instance, if you would like to match directories, you could use the d parameter to -type operand:

$ find . -type d
.
./economic
./technical
	

We will look at forming a complex expression at the end of this section about find, but at this moment it is handy to know that you can make a boolean 'and' expression by specifying multiple operands. For instance operand1 operand2 is true if both operand1 and operand2 are true for the object that is being evaluated. So, you could combine the -name and -type operands to find all directories that start with eco:

$ find . -name 'eco*' -type d
./economic
	

8.6.1.2. Operands that limit by object ownership or permissions

Besides matching objects by their name or type, you can also match them by their active permissions or the object ownership. This is often useful to find files that have incorrect permissions or ownership.

The owner (user) or group of an object can be matched with respectively the -user username and -group groupname variants. The name of a user or group will be interpreted as a user ID or group ID of the name is decimal, and could not be found on the system with getpwnam(3) or getgrnam(3). So, if you would like to match all objects of which joe is the owner, you can use -user joe as an operand:

$ find . -user joe
./secret/report.txt
	

Or to find all objects with the group friend as the file group:

$ find . -group friends
./secret/report.txt
	

The operand for checking file permissions -perm is less trivial. Like the chmod command this operator can work with octal and symbolic permission notations. We will start with looking at the octal notation. If an octal number is specified as a parameter to the -perm operand, it will match all objects that have exactly that permissions. For instance, -perm 0600 will match all objects that are only readable and writable by the user, and have no additional flags set:

$ find . -perm 0600
./secret/report.txt
	

If a dash is added as a prefix to a number, it will match every object that has at least the bits set that are specified in the octal number. A useful example is to find all files which have at least writable bits set for other users with -perm -0002. This can help you to find device nodes or other objects with insecure permissions.

$ find /dev -perm -0002
/dev/null
/dev/zero
/dev/ctty
/dev/random
/dev/fd/0
/dev/fd/1
/dev/fd/2
/dev/psm0
/dev/bpsm0
/dev/ptyp0
	

	Note
	Some device nodes have to be world-writable for a UNIX system to function correctly. For instance, the /dev/null device is always writable.

The symbolic notation of -perm parameters uses the same notation as the chmod command. Symbolic permissions are built with a file mode where all bits are cleared, so it is never necessary to use a dash to take away rights. This also prevents ambiguity that could arise with the dash prefix. Like the octal syntax, prefixing the permission with a dash will match objects that have at least the specified permission bits set. The use of symbolic names is quite predictable - the following two commands repeat the previous examples with symbolic permissions:

$ find . -perm u+rw
./secret/report.txt
	

$ find /dev -perm -o+w
/dev/null
/dev/zero
/dev/ctty
/dev/random
/dev/fd/0
/dev/fd/1
/dev/fd/2
/dev/psm0
/dev/bpsm0
/dev/ptyp0
	

$ find /etc -mtime 1
/etc
/etc/group
/etc/master.passwd
/etc/spwd.db
/etc/passwd
/etc/pwd.db
	

$ find /etc -mtime -2
/etc
/etc/network/run
/etc/network/run/ifstate
/etc/resolv.conf
/etc/default
/etc/default/locale
[...]
	

$ find . -newer economic/report2.txt
.
./technical
./technical/report2.txt
./technical/report.txt
./secret
./secret/report.txt
	

$ find / -name 'bin' -type d
/usr/bin
/bin
	

$ find / -name 'bin' -type d -xdev
/bin
	

$ find . -depth
./economic/report.txt
./economic/report2.txt
./economic
./technical/report2.txt
./technical/report.txt
./technical
.
	

$ find . -perm 0666 -exec chmod 0644 {} \;
	

$ find . -name '*.txt' -exec grep -q 'gross income' {} \; -print
./economic/report2.txt
	

$ find . -perm 0666 -exec chmod 0644 {} +
	

$ find . \( -perm 0666 -o -perm 0664 \) -exec chmod 0644 {} \;
	

The which command is not part of the Single UNIX Specification version 3, but it is provided by most sysmtems. which locates a command that is in the user's path (as set by the PATH environment variable), printing its full path. Providing the name of a command as its parameter will show the full path:

$ which ls
/bin/ls
      

You can also query the paths of multiple commands:

$ which ls cat
/bin/ls
/bin/cat
      

which returns a non-zero return value if the command could not be found.

This whereis command searches binaries, manual pages and sources of a command in some predefined places. For instance, the following command shows the path of the ls and the ls(1) manual page:

$ whereis ls
ls: /bin/ls /usr/share/man/man1/ls.1.gz
      

Slackware Linux also provides the locate command that searches through a file database that can be generated periodically with the updatedb command. Since it uses a prebuilt database of the filesystem, it is a lot faster than command, especially when directory entry information has not been cached yet. Though, the locate/updatedb combo has some downsides:

With filesystems becoming faster, and by applying common sense when formulating find queries, locate does not really seem worth the hassle. Of course, your mileage may vary. That said, the basic usage of locate is locate filename. For example:

$ locate locate
/usr/bin/locate
/usr/lib/locate
/usr/lib/locate/bigram
/usr/lib/locate/code
/usr/lib/locate/frcode
[...]
      

Sooner or later a GNU/Linux user will encounter tar archives, tar is the standard format for archiving files on GNU/Linux. It is often used in conjunction with gzip or bzip2. Both commands can compress files and archives. Table 8.6, “Archive file extensions” lists frequently used archive extensions, and what they mean.

The difference between bzip2 and gzip is that bzip2 can find repeating information in larger blocks, resulting in better compression. But bzip2 is also a lot slower, because it does more data analysis.

Since many software and data in the GNU/Linux world is archived with tar it is important to get used to extracting tar archives. The first thing you will often want to do when you receive a tar archive is to list its contents. This can be achieved by using the t parameter. However, if we just execute tar with this parameter and the name of the archive it will just sit and wait until you enter something to the standard input:

$ tar t test.tar
      

This happens because tar reads data from its standard input. If you forgot how redirection works, it is a good idea to reread Section 7.7, “Redirections and pipes”. Let's see what happens if we redirect our tar archive to tar:

$ tar t < test.tar
test/
test/test2
test/test1
      

That looks more like the output you probably expected. This archive seems to contain a directory test, which contains the files test2 and test2. It is also possible to specify the archive file name as an parameter to tar, by using the f parameter:

$ tar tf test.tar
test/
test/test2
test/test1
      

This looks like an archive that contains useful files ;). We can now go ahead, and extract this archive by using the x parameter:

$ tar xf test.tar
      

We can now verify that tar really extracted the archive by listing the contents of the directory with ls:

$ ls test/
test1  test2
      

Extracting or listing files from a gzipped or bzipped archive is not much more difficult. This can be done by adding a z or b for respectively archives compressed with gzip or bzip2. For example, we can list the contents of a gzipped archive with:

$ tar ztf archive2.tar.gz
      

And a bzipped archive can be extracted with:

$ tar bxf archive3.tar.bz2
      

You can create archives with the c parameter. Suppose that we have the directory test shown in the previous example. We can make an archive with the test directory and the files in this directory with:

$ tar cf important-files.tar test
      

This will create the important-files.tar archive (which is specified with the f parameter). We can now verify the archive:

$ tar tf important-files.tar
test/
test/test2
test/test1
      

Creating a gzipped or bzipped archive goes along the same lines as extracting compressed archives: add a z for gzipping an archive, or b for bzipping an archive. Suppose that we wanted to create a gzip compressed version of the archive created above. We can do this with:

tar zcf important-files.tar.gz test
      

Like most Unices Linux uses a technique named “mounting” to access filesystems. Mounting means that a filesystem is connected to a directory in the root filesystem. One could for example mount a CD-ROM drive to the /mnt/cdrom directory. Linux supports many kinds of filesystems, like Ext2, Ext3, ReiserFS, JFS, XFS, ISO9660 (used for CD-ROMs), UDF (used on some DVDs) and DOS/Windows filesystems, like FAT, FAT32 and NTFS. These filesystems can reside on many kinds of media, for example hard drives, CD-ROMs and Flash drives. This section explains how filesystems can be mounted and unmounted.

The mount is used to mount filesystems. The basic syntax is: “mount /dev/devname /mountpoint”. The device name can be any block device, like hard disks or CD-ROM drives. The mount point can be an arbitrary point in the root filesystem. Let's look at an example:

# mount /dev/cdrom /mnt/cdrom
      

This mounts the /dev/cdrom on the /mnt/cdrom mountpoint. The /dev/cdrom device name is normally a link to the real CD-ROM device name (for example, /dev/hdc). As you can see, the concept is actually very simple, it just takes some time to learn the device names ;). Sometimes it is necessary to specify which kind of filesystem you are trying to mount. The filesystem type can be specified by adding the -t parameter:

# mount -t vfat /dev/sda1 /mnt/flash
      

This mounts the vfat filesystem on /dev/sda1 to /mnt/flash.

The umount command is used to unmount filesystems. umount accepts two kinds of parameters, mount points or devices. For example:

# umount /mnt/cdrom
# umount /dev/sda1
      

The first command unmounts the filesystem that was mounted on /mnt/cdrom, the second commands unmounts the filesystem on /dev/sda1.

The GNU/Linux system has a special file, /etc/fstab, that specifies which filesystems should be mounted during the system boot. Let's look at an example:

/dev/hda10       swap             swap        defaults         0   0
/dev/hda5        /                xfs         defaults         1   1
/dev/hda6        /var             xfs         defaults         1   2
/dev/hda7        /tmp             xfs         defaults         1   2
/dev/hda8        /home            xfs         defaults         1   2
/dev/hda9        /usr             xfs         defaults         1   2
/dev/cdrom       /mnt/cdrom       iso9660     noauto,owner,ro  0   0
/dev/fd0         /mnt/floppy      auto        noauto,owner     0   0
devpts           /dev/pts         devpts      gid=5,mode=620   0   0
proc             /proc            proc        defaults         0   0
      

As you can see each entry in the fstab file has five entries: fs_spec, fs_file, fs_vfstype, fs_mntops, fs_freq, and fs_passno. We are now going to look at each entry.

There are two security mechanisms for securing files: signing files and encrypting files. Signing a file means that a special digital signature is generated for a file. You, or other persons can use the signature to verify the integrity of the file. File encryption encodes a file in a way that only a person for which the file was intended to read can read the file.

This system relies on two keys: the private and the public key. Public keys are used to encrypt files, and files can only be decrypted with the private key. This means that one can sent his public key out to other persons. Others can use this key to send encrypted files, that only the person with the private key can decode. Of course, this means that the security of this system depends on how well the private is kept secret.

Slackware Linux provides an excellent tool for signing and encrypting files, named GnuPG. GnuPG can be installed from the “n” disk set.

Generating public and private keys is a bit complicated, because GnuPG uses DSA keys by default. DSA is an encryption algorithm, the problem is that the maximum key length of DSA is 1024 bits, this is considered too short for the longer term. That is why it is a good idea to use 2048 bit RSA keys. This section describers how this can be done.

Note

1024-bit keys were believed to be secure for a long time. But Bernstein's paper Circuits for Integer Factorization: a Proposal contests this, the bottom line is that it is quite feasible for national security agencies to produce hardware that can break keys in a relatively short amount of time. Besides that it has be shown that 512-bit RSA keys can be broken in a relatively short time using common hardware. More information about these issues can by found in this e-mail to the cypherpunks list: http://lists.saigon.com/vault/security/encryption/rsa1024.html

We can generate a key by executing:

$ gpg --gen-key
      

The first question is what kind of key you would like to make. We will choose (4) RSA (sign only):

Please select what kind of key you want:
   (1) DSA and ElGamal (default)
   (2) DSA (sign only)
   (4) RSA (sign only)
Your selection? 4
      

You will then be asked what the size of the key you want to generate has to be. Type in 2048 to generate a 2048 bit key, and press enter to continue.

What keysize do you want? (1024) 2048
      

The next question is simple to answer, just choose what you like. Generally speaking it is not a bad idea to let the key be valid infinitely. You can always deactivate the key with a special revocation certificate.

Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
      

GnuPG will then ask for confirmation. After confirming your name and e-mail address will be requested. GnuPG will also ask for a comment, you can leave this blank, or you could fill in something like “Work” or “Private”, to indicate what the key is used for. For example:

Real name: John Doe
Email address: john@doe.com
Comment: Work              
You selected this USER-ID:
    "John Doe (Work) <john@doe.com>"
      

GnuPG will the ask you to confirm your user ID. After confirming it GnuPG will ask you to enter a password. Be sure to use a good password:

You need a Passphrase to protect your secret key.    

Enter passphrase:
      

After entering the password twice GnuPG will generate the keys. But we are not done yet. GnuPG has only generated a key for signing information, not for encryption of information. To continue, have a look at the output, and look for the key ID. In the information about the key you will see pub 2048R/. The key ID is printed after this fragment. In this example:

public and secret key created and signed.
key marked as ultimately trusted.

pub  2048R/8D080768 2004-07-16 John Doe (Work) <john@doe.com>
     Key fingerprint = 625A 269A 16B9 C652 B953  8B64 389A E0C9 8D08 0768
      

the key ID is 8D080768. If you lost the output of the key generation you can still find the key ID in the output of the gpg --list-keys command. Use the key ID to tell GnuPG that you want to edit your key:

$ gpg --edit-key <Key ID>
      

With the example key above the command would be:

$ gpg --edit-key 8D080768
      

GnuPG will now display a command prompt. Execute the addkey command on this command prompt:

Command> addkey
      

GnuPG will now ask the password you used for your key:

Key is protected.

You need a passphrase to unlock the secret key for
user: "John Doe (Work) <john@doe.com>"
2048-bit RSA key, ID 8D080768, created 2004-07-16

Enter passphrase:
      

After entering the password GnuPG will ask you what kind of key you would like to create. Choose RSA (encrypt only), and fill in the information like you did earlier (be sure to use a 2048 bit key). For example:

Please select what kind of key you want:
   (2) DSA (sign only)
   (3) ElGamal (encrypt only)
   (4) RSA (sign only)
   (5) RSA (encrypt only)
Your selection? 5
What keysize do you want? (1024) 2048
Requested keysize is 2048 bits       
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
      

And confirm that the information is correct. After the key is generated you can leave the GnuPG command prompt, and save the new key with the save command:

Command> save
      

Congratulations! You have now generated the necessary keys to encrypt and decrypt e-mails and files. You can now configure your e-mail client to use GnuPG. It is a good idea to store the contents of the .gnupg directory on some reliable medium, and store that in a safe place! If your private key is lost you can't decrypt files and messages that were encrypted with your public key. If the private key, and your password are stolen, the security of this system is completely compromised.

To make GnuPG useful, you have to give your public key to people who send you files or e-mails. They can use your public key to encrypt files, or use it to verify whether a file has a correct signature or not. The key can be exported using the --export parameter. It is also a good idea to specify the --output parameter, this will save the key in a file. The following command would save the public key of John Doe, used in earlier examples, to the file key.gpg:

$ gpg --output key.gpg --export john@doe.com
      

This saves the key in binary format. Often it is more convenient to use the so-called “ASCII armored output”, which fits better for adding the key to e-mails, or websites. You export an ASCII armored version of the key by adding the --armor parameter:

$ gpg --armor --output key.gpg --export john@doe.com
      

If you look at the key.gpg file you will notice that the ASCII armored key is a much more comfortable format.

With GPG you can make a signature for a file. This signature is unique, because your signature can only be made with your private key. This means that other people can check whether the file was really sent by you, and whether it was in any way altered or not. Files can be signed with the --detach-sign parameter. Let us look at an example. This command will make a signature for the memo.txt file. The signature will be stored in memo.txt.sig.

$ gpg --output memo.txt.sig --detach-sign memo.txt

You need a passphrase to unlock the secret key for
user: "John Doe (Work) <john@doe.com>"
2048-bit RSA key, ID 8D080768, created 2004-07-16

Enter passphrase:
      

As you can see, GnuPG will ask you to enter the password for your private key. After you have entered the right key the signature file (memo.txt.sig) will be created.

You can verify a file with its signature using the --verify parameter. Specify the signature file as a parameter to the --verify parameter. The file that needs to be verified can be specified as the final parameter:

$ gpg --verify memo.txt.sig memo.txt
gpg: Signature made Tue Jul 20 23:47:45 2004 CEST using RSA key ID 8D080768
gpg: Good signature from "John Doe (Work) <john@doe.com>"
      

This will confirm that the file was indeed signed by John Doe (Work) <john@doe.com>, with the key 8D080768, and that the file is unchanged. Suppose the file was changed, GnuPG would have complained about it loudly:

$ gpg --verify memo.txt.sig memo.txt
gpg: Signature made Tue Jul 20 23:47:45 2004 CEST using RSA key ID 8D080768
gpg: BAD signature from "John Doe (Work) <john@doe.com>"
      

One of the main features of GnuPG is encryption. Due to its use of asymmetric cryptography, the person who encrypts a file and the person who decrypts a file do not need to share a key. You can encrypt a file with the public key of another person, and that other person can decrypt it with his or her private key. You can encrypt files with the --encrypt. If you do not specify a user ID for which the file should be encrypted, GnuPG will prompt for the user ID. You can specify the user ID with the -r parameter. In the following example, the file secret.txt will be encrypted for another person named John Doe:

$ gpg --encrypt -r "John Doe" secret.txt
      

The user ID is quoted with double quotes for making sure that the ID is interpreted as a single program argument. After the encryption is completed, the encrypted version of the file will be available as secret.txt.gpg.

The user who receives the file can decrypt it with the --decrypt parameter of the gpg command:

$ gpg --output secret.txt --decrypt secret.txt.gpg

You need a passphrase to unlock the secret key for
user: "John Doe (Work) <john@doe.com>"
2048-bit RSA key, ID 8D080768, created 2004-07-16 (main key ID EC3ED1AB)

Enter passphrase:

gpg: encrypted with 2048-bit RSA key, ID 8D080768, created 2004-07-16
      "John Doe (Work) <john@doe.com>"
      

In this example the --output parameter is used store the decrypted content in secret.txt.

The most simple text filter is the cat, it does nothing else than sending the data from stdin to stdout:

$ echo "hello world" | cat
hello world
      

Another useful feature is that you can let it send the contents of a file to the standard output:

$ cat file.txt
Hello, this is the content of file.txt
      

cat really lives up to its name when multiple files are added as arguments. This will concatenate the files, in the sense that it will send the contents of all files to the standard output, in the same order as they were specified as an argument. The following screen snippet demonstrates this:

$ cat file.txt file1.txt file2.txt
Hello, this is the content of file.txt
Hello, this is the content of file1.txt
Hello, this is the content of file2.txt
      

The wc command provides statistics about a text file or text stream. Without any parameters, it will print the number of lines, the number of words, and the number of bytes respectively. A word is delimited by one white space character, or a sequence of whitespace characters.

The following example shows the number of lines, words, and bytes in the canonical “Hello world!” example:

$ echo "Hello world!" | wc 
       1       2      13
      

If you would like to print just one of these components, you can use one of the -l (lines), -w (words), or -c (bytes) parameters. For instance, adding just the -l parameter will show the number of lines in a file:

$ wc -l /usr/share/dict/words 
  235882 /usr/share/dict/words
      

Or, you can print additional fields by adding a parameter:

$ wc -lc /usr/share/dict/words
 235882 2493082 /usr/share/dict/words
      

Please note that, no matter the order in which the options were specified, the output order will always be the same (lines, words, bytes).

Since -c prints the number bytes, this parameter may not represent the number of characters that a text holds, because the character set in use maybe be wider than one byte. To this end, the -m parameter has been added which prints the number of characters in a text, independent of the character set. -c and -m are substitutes, and can never be used at the same time.

The statistics that wc provides are more useful than they may seem on the surface. For example, the -l parameter is often used as a counter for the output of a command. This is convenient, because many commands seperate logical units by a newline. Suppose that you would like to count the number of files in your home directory having a filename ending with .txt. You could do this by combining find to find the relevant files and wc to count the number of occurences:

$ find ~ -name '*.txt' -type f | wc -l
      

The tr command can be used to do common character operations, like swapping characters, deleting characters, and squeezing character sequences. Depending on the operation, one or two sets of characters should be specified. Besides normal characters, there are some special character sequences that can be used:

$ echo 'Hello world!' | tr 'eo' 'ia'
Hilla warld!
	

$ echo 'Hello world!' | tr 'eaiou' '[@*]'
H@ll@ w@rld!
	

$ echo 'abcdef' | tr 'abcdef' '@[-*4]!'
@----!
	

$ echo "Let's squeeze this." | tr -s 'e'
Let's squeze this.
	

$ echo "eenie meenie minie moe" | tr 'aeiou' '[@*]' | tr -s '@'
@n@ m@n@ m@n@ m@
	

$ echo 'Hello world!' | tr -d 'lr'
Heo wod!
	

The cut command is provided by UNIX systems to “cut” one or more columns from a file or stream, printing it to the standard output. It is often useful to selectively pick some information from a text. cut provides three approaches to cutting information from files:

In all three approaches, you can specify the element to choose by its number starting at 1. You can specify a range by using a dash (-). So, M-N means the Mth to the Nth element. Leaving M out (-N) selects all elements from the first element to the Nth element. Leaving N out (M-) selects the Mth element to the last element. Multiple elements or ranges can be combined by separating them by commas (,). So, for instance, 1,3- selects the first element and the third to the last element.

Data can be cut by field with the -f fields parameter. By default, the horizontal tab is used a separator. Let's have a look at cut in action with a tiny Dutch to English dictionary:

$ cat dictionary
appel   apple
banaan  banana
peer    pear
      

We can get all English words by selecting the first field:

$ cut -f 2 dictionary
apple
banana
pear
      

That was quite easy. Now let's do the same thing with a file that has a colon as the field separator. We can easily try this by converting the dictionary with the tr command that we have seen earlier, replacing all tabs with colons:

$ tr '\t' ':' < dictionary > dictionary-new
$ cat dictionary-new
appel:apple
banaan:banana
peer:pear
      

If we use the same command as in the previous example, we do not get the correct output:

$ cut -f 2 dictionary-new
appel:apple
banaan:banana
peer:pear
      

What happens here is that the delimiter could not be found. If a line does not contain the delimiter that is being used, the default behavior of cut is to print the complete line. You can prevent this with the -s parameter.

To use a different delimiter than the horizontal tab, add the -d delimter_char parameter to set the delimiting character. So, in this case of our dictionary-new file, we will ask cut to use the colon as a delimiter:

$ cut -d ':' -f 2 dictionary-new
apple
banana
pear
      

If a field that was specified does not exist in a line, that particular field is not printed.

The -b bytes and -c characters respectively select bytes and characters from the text. On older systems a character used to be a byte wide. But newer systems can provide character sets that are wider than one byte. So, if you want to be sure to grab complete characters, use the -c parameter. An entertaining example of seeing the -c parameter in action is to find the ten most common sets of the first three characters of a word. Most UNIX systems provide a list of words that are separated by a new line. We can use cut to get the first three characters of the words in the word list, add uniq to count identical three character sequences, and use sort to sort them reverse-numerically (sort is described in Section 9.1.5, “Sorting text”). Finally, we will use head to get the ten most frequent sequences:

$ cut -c 1-4 /usr/share/dict/words | uniq -c | sort -nr | head
    254 inte
    206 comp
    169 cons
    161 cont
    150 over
    125 tran
    111 comm
    100 disc
     99 conf
     96 reco
      

Having concluded with that nice piece of UNIX commands in action, we will move on to the paste command, which combines files in columns in a single text stream.

Usage of paste is very simple, it will combine all files given as an argument, separated by a tab. With the list of English and Dutch words, we can generate a tiny dictionary:

$ paste dictionary-en dictionary-nl
apple   appel
banana  banaan
pear    peer
      

You can also combine more than two files:

$ paste dictionary-en dictionary-nl dictionary-de 
apple   appel   Apfel
banana  banaan  Banane
pear    peer    Birne
      

If one of the files is longer, the column order is maintained, and empty entries are used to fill up the entries of the shorter files.

You can use another delimiter by adding the -d delimiter parameter. For example, we can make a colon-separated dictionary:

$ paste -d ':' dictionary-en dictionary-nl
apple:appel
banana:banaan
pear:peer
      

Normally, paste combines files as different columns. You can also let paste use the lines of each file as columns, and put the columns of each file on a separate line. This is done with the -s parameter:

$ paste -s dictionary-en dictionary-nl dictionary-de
apple   banana  pear
appel   banaan  peer
Apfel   Banane  Birne
      

UNIX offers the sort command to sort text. sort can also check whether a file is in sorted order, and merge two sorted files. sort can sort in dictionary and numerical orders. The default sort order is the dictionary order. This means that text lines are compared character by character, sorted as specified in the current collating sequence (which is specified through the LC_COLLATE environment variable). This has a catch when you are sorting numbers, for instance, if you have the numbers 1 to 10 on different lines, the sequence will be 1, 10, 2, 3, etc. This is caused by the per-character interpretation of the dictionary sort. If you want to sort lines by number, use the numerical sort.

If no additional parameters are specified, sort sorts the input lines in dictionary order. For instance:

$ cat << EOF | sort
orange
apple
banana
EOF
apple
banana
orange
      

As you can see, the input is correctly ordered. Sometimes there are two identical lines. You can merge identical lines by adding the -u parameter. The two samples listed below illustrate this.

$ cat << EOF | sort
orange
apple
banana
banana
EOF
apple
banana
banana
orange
$ cat << EOF | sort -u
orange
apple
banana
banana
EOF
apple
banana
orange
      

There are some additional parameters that can be helpful to modify the results a bit:

You can sort files numerically by adding the -n parameter. This parameter stops reading the input line when a non-numeric character was found. The leading minus sign, decimal point, thousands separator, radix character (that separates an exponential from a normal number), and blanks can be used as a part of a number. These characters are interpreted where applicable.

The following example shows numerical sort in action, by piping the output of du to sort. This works because du specifies the size of each file as the first field.

$ du -a /bin | sort -n
0       /bin/kernelversion
0       /bin/ksh
0       /bin/lsmod.modutils
0       /bin/lspci
0       /bin/mt
0       /bin/netcat
[...]
      

In this case, the output is probably not useful if you want to read the output in a paginator, because the smallest files are listed first. This is where the -r parameter becomes handy. This reverses the sort order.

$ du -a /bin | sort -nr
4692    /bin
1036    /bin/ksh93
668     /bin/bash
416     /bin/busybox
236     /bin/tar
156     /bin/ip
[...]
      

The -r parameter also works with dictionary sorts.

Quite often, files use a layout with multiple columns, and you may want to sort a file by a different column than the first column. For instance, consider the following score file named score.txt:

John:US:4
Herman:NL:3
Klaus:DE:5
Heinz:DE:3
      

Suppose that we would like to sort the entries in this file by the two-letter country name. sort allows us to sort a file by a column with the -k col1[,col2] parameter. Where col1 up to col2 are used as fields for sorting the input. If col2 is not specified, all fields up till the end of the line are used. So, if you want to use just one column, use -k col1,col1. You can also specify the the starting character within a column by adding a period (.) and a character index. For instance, -k 2.3,4.2 means that the second column starting from the third character, the third column, and the fourth column up to (and including) the second character.

There is yet another particularity when it comes to sorting by columns: by default, sort uses a blank as the column separator. If you use a different separator character, you will have to use the -t char parameter, that is used to specify the field separator.

With the -t and -k parameters combined, we can sort the scores file by country code:

$ sort -t ':' -k 2,2 scores.txt
Heinz:DE:3
Klaus:DE:5
Herman:NL:3
John:US:4
    

So, how can we sort the file by the score? Obviously, we have to ask sort to use the third column. But sort uses a dictionary sort by default^[6]. You could use the -n, but sort also allows a more sophisticated approach. You can append the one or more of the n, r>, f, d, i, or b to the column specifier. These letters represent the sort parameters with the same name. If you add just the starting column, append it to that column, otherwise, add it to the ending column.

The following command sorts the file by score:

$ sort -t ':' -k 3n /home/daniel/scores.txt
Heinz:DE:3
Herman:NL:3
John:US:4
Klaus:DE:5
    

It is good to follow this approach, rather than using the parameter variants, because sort allows you to use more than one -k parameter. And, adding these flags to the column specification, will allow you to sort by different columns in different ways. For example using sort with the -k 3,3n -k 2,2 parameters will sort all lines numerically by the third column. If some lines have identical numbers in the third column, these lines can be sorted further with a dictionary sort of the second column.

If you want to check whether a file is already sorted, you can use the -c parameter. If the file was in a sorted order, sort will return the value 0, otherwise 1. We can check this by echoing the value of the ? variable, which holds the return value of the last executed command.

$ sort -c scores.txt ; echo $?
1
$ sort scores.txt | sort -c ; echo $?
0
    

The second command shows that this actually works, by piping the output of the sort of scores.txt to sort.

Finally, you can merge two sorted files with the -m parameter, keeping the correct sort order. This is faster than concatenating both files, and resorting them.

# sort -m scores-sorted.txt scores-sorted2.txt
    

Since text streams, and text files are very important in UNIX, it is often useful to show the differences between two text files. The main utilities for working with file differences are diff and patch. diff shows the differences between files. The output of diff can be processed by patch to apply the changes between two files to a file. “diffs” are also form the base of version/source management systems. The following sections describe diff and patch. To have some material to work with, the following two C source files are used to demonstrate these commands. These files are named hello.c and hello2.c respectively.

#include <stdio.h>

void usage(char *programName);

int main(int argc, char *argv[]) {
  if (argc == 1) {
    usage(argv[0]);
    return 1;
  }

  printf("Hello %s!\n", argv[1]);

  return 0;
}

void usage(char *programName) {
  printf("Usage: %s name\n", programName);
}

      

#include <stdio.h>
#include <time.h>

void usage(char *programName);

int main(int argc, char *argv[]) {
  if (argc == 1) {
    usage(argv[0]);
    return 1;
  }

  printf("Hello %s!\n", argv[1]);

  time_t curTime = time(NULL);
  printf("The date is %s\n", asctime(localtime(&curTime)));


  return 0;
}

void usage(char *programName) {
  printf("Usage: %s name\n", programName);
}
      

$ diff hello.c hello2.c
1a2 
> #include <time.h> 
12a14,17
>   time_t curTime = time(NULL);
>   printf("The date is %s\n", asctime(localtime(&curTime)));
>
	

$ diff hello2.c hello.c
2d1 
< #include <time.h> 
14,16d12
<   time_t curTime = time(NULL);
<   printf("The date is %s\n", asctime(localtime(&curTime)));
<
	

$ diff -u hello.c hello2.c
--- hello.c     2006-11-26 20:28:55.000000000 +0100 
+++ hello2.c    2006-11-26 21:27:52.000000000 +0100 
@@ -1,4 +1,5 @@ 
 #include <stdio.h> 
+#include <time.h> 

 void usage(char *programName);

@@ -10,6 +11,9 @@

   printf("Hello %s!\n", argv[1]);

+  time_t curTime = time(NULL);
+  printf("The date is %s\n", asctime(localtime(&curTime)));
+
   return 0;
 }

	

$ diff -u hello2.c hello.c

--- hello2.c    2006-11-26 21:27:52.000000000 +0100
+++ hello.c     2006-11-26 20:28:55.000000000 +0100
@@ -1,5 +1,4 @@
 #include <stdio.h>
-#include <time.h>

 void usage(char *programName);

@@ -11,9 +10,6 @@

   printf("Hello %s!\n", argv[1]);

-  time_t curTime = time(NULL);
-  printf("The date is %s\n", asctime(localtime(&curTime)));
-
   return 0;
 }


	

$ diff -ru hello.orig hello
diff -ru hello.orig/hello.c hello/hello.c

--- hello.orig/hello.c  2006-12-04 17:37:14.000000000 +0100
+++ hello/hello.c       2006-12-04 17:37:48.000000000 +0100
@@ -1,4 +1,5 @@
 #include <stdio.h>
+#include <time.h>

 void usage(char *programName);

@@ -10,6 +11,9 @@

   printf("Hello %s!\n", argv[1]);

+  time_t curTime = time(NULL);
+  printf("The date is %s\n", asctime(localtime(&curTime)));
+
   return 0;
 }


	

$ diff -ruN hello.orig hello

diff -ruN hello.orig/hello.c hello/hello.c
--- hello.orig/hello.c  2006-12-04 17:37:14.000000000 +0100
+++ hello/hello.c       2006-12-04 17:37:48.000000000 +0100
@@ -1,4 +1,5 @@
 #include <stdio.h>
+#include <time.h>

 void usage(char *programName);

@@ -10,6 +11,9 @@

   printf("Hello %s!\n", argv[1]);

+  time_t curTime = time(NULL);
+  printf("The date is %s\n", asctime(localtime(&curTime)));
+
   return 0;
 }

diff -ruN hello.orig/Makefile hello/Makefile
--- hello.orig/Makefile 1970-01-01 01:00:00.000000000 +0100
+++ hello/Makefile      2006-12-04 17:39:44.000000000 +0100
@@ -0,0 +1,2 @@
+hello: hello.c
+       gcc -Wall -o $@ $<

	

$ diff -u hello.c hello2.c > hello_add_date.diff
	

$ cat diff1 diff2 diff3 > combined_diff
	

$ patch < hello_add_date.diff
patching file hello.c
	

$ diff -u hello.c hello2.c
	

$ patch -R < hello_add_date.diff
	

$ patch helloworld.c < hello_add_date.diff
patching file helloworld.c
	

$ cd hello.orig
$ patch -p 1 < ../hello.diff
	

$ patch -p 1 -d hello.orig < hello.diff
patching file hello.c
patching file Makefile
	

$ patch -b < hello_add_date.diff
$ ls -l hello.c*
-rw-r--r-- 1 daniel daniel 382 2006-12-04 21:41 hello.c
-rw-r--r-- 1 daniel daniel 272 2006-12-04 21:12 hello.c.orig
	

In daily life, you will often want to some text that matches to a certain pattern, rather than a literal string. Many UNIX utilities implement a language for matching text patterns, regular expressions (regexps). Over time the regular expression language has grown, there are now basically three regular expression syntaxes:

POSIX regexps are mostly a superset of traditional UNIX regexps, and PCREs a superset of POSIX regexps. The syntax that an application supports differs per application, but almost all applications support at least POSIX regexps.

Each syntactical unit in a regexp expresses one of the following things:

This section describes traditional UNIX regexps. Because of a lack of standardisation, the exact syntax may differ a bit per utility. Usually, the manual page of a command provides more detailed information about the supported basic or traditional regular expressions. It is a good idea to learn traditional regexps, but to use POSIX regexps for your own scripts.

POSIX regular expressions build upon traditional regular expressions, adding some other useful primitives. Another comforting difference is that grouping parenthesises, quantification accolades, and the alternation sign (|) are not backslash-escaped. If they are escaped, they will match the literal characters instead, thus resulting in the opposite behavior of traditional regular expressions. Most people find POSIX extended regular expressions much more comfortable, making them more widely used.

We have now arrived at one of the most important utilties of the UNIX System, and the first occasion to try and use regular expressions. The grep command is used to search a text stream or a file for a pattern. This pattern is a regular expression, and can either be a basic regular expression or a POSIX extended regular expression (when the -E parameter is used). By default, grep will write the lines that were matched to the standard output. In the most basic syntax, you can specify a regular expression as an argument, and grep will search matches in the text from the standard input. This is a nice manner to practice a bit with regular expressions.

$ grep '^\(ab\)\{2,3\}$'
ab
abab
abab
ababab
ababab
abababab
      

The example listed above shows a basic regular expression in action, that matches a line solely consisting of two or three times the ab string. You can do the same thing with POSIX extended regular expressions, by adding the -E (for extended) parameter:

$ grep -E '^(ab){2,3}$'
ab
abab
abab
ababab
ababab
abababab
      

Since the default behavior of grep is to read from the standard input, you can add it to a pipeline to get the interesting parts of the output of the preceding commands in the pipeline. For instance, if you would like to search for the string 2006 in the third column in a file, you could combine the cut and grep command:

$ cut -f 3 | grep '2006'
      

Naturally, grep can also directly read a file, rather than the standard input. As usual, this is done by adding the files to be read as the last arguments. The following example will print all lines from the /etc/passwd file that start with the string daniel:.

$ grep "^daniel" /etc/passwd
daniel:*:1001:1001:Daniel de Kok:/home/daniel:/bin/sh
      

With the -r option, grep will recursively traverse a directory structure, trying to find matches in each file that was encountered during the traversal. Though, it is better to combine grep with find and the -exec operand in scripts that have to be portable.

$ grep -r 'somepattern' somedir
      

is the non-portable functional equivalent of

$ find /somedir -type f -exec grep 'somepattern' {} \; -print
      

grep can also print all lines that do not match the pattern that was used. This is done by adding the -v parameter:

$ grep -Ev '^(ab){2,3}$'
ab
ab
abab
ababab
abababab
abababab
      

If you want to use the pattern in a case-insensitive manner, you can add the -i parameter. For example:

$ grep -i "a"
a
a
A
A
      

You can also match a string literally with the -F parameter:

$ grep -F 'aa*'
a
aa*
aa*
      

As we have seen, you can use the alternation character (|) to match either of two or more subpatterns. If two patterns that you would like to match differ a lot, it is often more comfortable to make two separate patterns. grep allows you to use more than one pattern by separating patterns with a newline character. So, for example, if you would like to print lines that match either the a or b pattern, this can be done easily by starting a new line:

$ grep 'a
b'
a
a
b
b
c
      

This works, because quotes are used, and the shell passes quoted parameters literally. Though, it must be admitted that this is not quite pretty. grep accepts one or more -e pattern parameters, giving the opportunity to specify more than one parameter on one line. The grep invocation in the previous example could be rewritten as:

$ grep -e 'a' -e 'b'
      

A running instance of a program is called a process. Each process has its own protected memory, named the process address space. This address space consists of two areas: the text area and the data area. The text area contains the actual program code, and tells the system what to do. The data area stores constant and runtime data of a process. Since there are many processes on a system, and only one or a few processors, the operating system kernel divides processor time between processes. This process is called time-sharing.

Table 10.1, “The structure of a process” lists the most important fields of information that a kernel stores about a process. Each process can be identified uniquely by its PID (process identifier), which is an unsigned number. As we will see later, a user can easily retrieve the PID of a process. Each process is associated with a UID (user ID) and GID (group ID) on the system. Each process has a real UID, which is the UID as which the process was started, and the effective UID, which is the UID as which the process operates. Normally, the effective UID is equal to the real UID, but some programs ask the system to change its effective UID. The effective UID determines is used for access control. This means that if a user named joe starts a command, say less, less can only open files that joe has read rights for. In parallel, a process also has an real GID and an effective GID.

Many processes open files, the handle used to operate on a file is called a file descriptor. The kernel manages a list of open file descriptors for each process. The fd field contains a pointer to the list of open files. The vmspace field points to the process address space of the process.

Figure 10.1. Process states

Not every process is in need of CPU time at a given moment. For instance, some processes maybe waiting for some I/O (Input/Output) operation to complete or may be terminated. Not taking subtleties in account, processes are normally started, running, ready (to run), blocked (waiting for I/O), or terminated. Figure 10.1, “Process states” shows the lifecycle of a process. A process that is terminated, but for which the process table entry is not reclaimed, is often called a zombie process. Zombie processes are useful to let the parent process read the exit status of the process, or reserve the process table entry temporarily.

New processes are created with the fork() system call. This system call copies the process address space and process information of the caller, and gives the new process, named the child process, a different PID. The child process will continue execution at the same point as the parent, but will get a different return value from the fork() system call. Based on this return value the code of the parent and child can decide how to continue executing. The following piece of C code shows a fork() call in action:

#include <sys/types.h>
#include <stdio.h>
#include <unistd.h>

int main() {
  pid_t pid = fork();
  if (pid == 0)
    printf("Hi, I am the child!\n");
  else
    printf("Hi, I am the parent, the child PID is %d!\n", pid);

  return 0;
}
      

. This little program calls fork(), storing the return value of fork() in the variable pid. fork() returns the value 0 to the child, and the PID of the child to the parent. Since this is the case, we can use a simple conditional structure to check the value of the pid variable, and print an appropriate message.

You may wonder how it is possible to start new programs, since the fork() call duplicates an existing process. That is a good questions, since with fork() alone, it is not possible to execute new programs. UNIX kernels also provide a set of system calls, starting with exec, that load a new program image in the current process. We saw at the start of this chapter that a process is a running program -- a process was constructed in memory from the program image that is stored on a storage medium. So, the exec family of system calls gives a running process the facilities to replace its contents with a program stored on some medium. In itself, this is not wildly useful, because every time the an exec call is done, the original calling code (or program) is removed from the process. This can be witnessed in the following C program:

#include <stdio.h>
#include <unistd.h>

int main() {
  execve("/bin/ls", NULL, NULL);

  /* This will never be printed, unless execve() fails. */
  printf("Hello world!\n");

  return 0;
}
      

This program executes ls with the execve() call. The message printed with printf() will never be shown, because the running program image is replaced with that of ls. Though, a combination of fork() and the exec functions are very powerful. A process can fork itself, and let the child “sacrifice” itself to run another program. The following program demonstrates this pattern:

#include <sys/types.h>
#include <stdio.h>
#include <unistd.h>

int main() {
  pid_t pid = fork();
  if (pid == 0)
    execve("/bin/ls", NULL, NULL);

  printf("Hello world!");

  return 0;
}
      

This program forks itself first. The program image of the child process will be replaced with ls, while the parent process prints the “Hello world!” message to the screen and returns.

This procedure is followed by many programs, including the shell, when a command is executed from the shell prompt. In fact all processes on a UNIX system are directly or indirectly derrived from the init process, which is the first program that is started on a UNIX system.

Although forks are very useful to build some parallelism^[7], they can be to expensive for some purposes. Copying the whole process takes some time, and there is cost involved if processes want to share data. This is solved by offering a more lightweight alternative, namely allowing more than one thread of execution. Each thread of execution is executed separately, but the process data is shared between the threads.

Writing good multithreaded programs requires good knowledge of data sharing and locking. Since all data is shared, uncareful programming can lead to bugs like race conditions.

UNIX systems provide the ps command to show a list of running processes. Unfortunately, this command is an example of the pains of the lack of standardization. The BSD and System V variants of ps have their own sets of options. Fortunately, GNU/Linux implements both the System V and BSD-style parameters, as well as some (GNU-style) long options. Options preceded by a dash are interpreted as System V options and options without a dash as BSD options. We will describe the System V-style options in this section.

If ps is used without any parameters, it shows all processes owned by the user that invokes ps and that are attached to the same controlling terminal. For example:

$ ps
  PID TTY          TIME CMD
 8844 pts/5    00:00:00 bash
 8862 pts/5    00:00:00 ps
      

A lot of useful information can be distilled from this output. As you can see, two processes are listed: the shell that we used to call ps (bash), and the ps command itself. In this case there are four information fields. PID is the process ID of a process, TTY the controlling terminal, TIME the amount of CPU time the proces has used, and CMD the command or program of which a copy is running. The fields that are shown by default may vary a bit per system, but usually at least these fields are shown, with somewhat varying field labels.

Sometime you may want to have a broader view of processes that are running. Adding the -a option shows all processes that are associated with terminals. For instance:

$ ps -a
  PID TTY          TIME CMD
 7487 pts/1    00:00:00 less
 8556 pts/4    00:00:10 emacs-x
11324 pts/3    00:00:00 ps
      

As you can see, processes with different controlling terminals are shown. Though, in contrast to the plain ps output, only processes that control the terminal at the moment are shown. For instance, the shell that was used to call ps is not shown.

You can also print all processes that are running, including processes that are not associated with a terminal, by using the -A option:

$ ps -A | head -n 10
  PID TTY          TIME CMD
    1 ?        00:00:01 init
    2 ?        00:00:00 migration/0
    3 ?        00:00:00 ksoftirqd/0
    4 ?        00:00:00 watchdog/0
    5 ?        00:00:00 migration/1
    6 ?        00:00:00 ksoftirqd/1
    7 ?        00:00:00 watchdog/1
    8 ?        00:00:00 events/0
    9 ?        00:00:00 events/1
      

You can print all processes with a certain user ID, with the -U option. This option accepts a user name as a parameter, or multiple user names that are separated by a comma. The following command shows all processes that have xfs or rpc as their user ID:

$ ps -U xfs,rpc
  PID TTY          TIME CMD
 2409 ?        00:00:00 portmap
 2784 ?        00:00:00 xfs
      

Likewise, you can also print processes with a particular group ID, with the -G option:

$ ps -G messagebus,haldaemon
  PID TTY          TIME CMD
 8233 ?        00:00:00 dbus-daemon
11312 ?        00:00:00 hald
11320 ?        00:00:00 hald-addon-keyb
11323 ?        00:00:00 hald-addon-acpi
      

If you would like to have a list for a physical or pseudo-terminal, you can use the -t option:

$ ps -t tty2
  PID TTY          TIME CMD
 2655 tty2     00:00:00 getty
      

Signals are a crude, but effective form of inter-process communication (IPC). A signal is basically a number that is delivered to a process that has a special meaning. For all signals there are default signal handlers. Processes can install their own signal handlers, or choose to ignore signals. Some signals (normally SIGKILL and SIGSTOP) can not be ignored. All signals have convenient symbolic names.

Only a few signals are normally interesting for interactive use on UNIX(-like) systems. These are (followed by their number):

The kill command is used to send a signal to a process. By default, kill sends the SIGTERM signal. To send this signal, the process ID of the process that you would like to send this signal to should be added as a parameter. For instance:

$ kill 15631
      

To send another signal, you can use one of two options: -signalnumber or -signalname. So, the following commands both send the SIGKILL signal to the process with process ID 15631:

$ kill -9 15631
      

$ kill -SIGKILL 15631
      

In an act of altruism you can be nice to other users of computer resources. If you plan to run a CPU-time intensive process, but do not want that to interfere with work of other users on the system (or other processes), you can assign some grade of 'niceness' to a process. Practically, this means that you will be able to influence the scheduling priority of a process. Nicer processes get a lower scheduling priority. The normal niceness of a process is 0, and can be changed by executing a program with the nice command. The -n [niceness] option can be used to specify the niceness:

$ nice -n 20 cputimewaster
      

The maximum number for niceness is implementation-dependent. If a program was started with nice, but no niceness was specified, the niceness will be set to 10. In case you were wondering: yes, you can also be rude, but this right is restricted to the root user. You can boost the priority of a process by specifying a negative value as the niceness.

You can also modify the niceness of a running processes with the renice command. This can be done for specific process IDs (-p PIDs), users (-u user/uid), and effective groups (-g group/gid). The new niceness is specified as the first parameter.

The niceness of a process can only be increased. And, of course, no user except for root can affect the niceness of processes of other users.

Lets look at an example, to set the niceness of a process with PID 3108 to 14, you could use the following command:

$ renice 14 -p 3108
      

A job that is in the foreground (thus, a job that potentially accepts userinput from the terminal) can be stopped by pressing Ctrl-z (pressing both the 'Ctrl' and 'z' keys simultaneously). This will stop the job, and will handle control over the terminal back to the shell. Let's try this with the sleep command, which waits for the number of seconds provided as an argument:

$ sleep 3600
Ctrl-z
[1]+  Stopped                 sleep 3600
      

The process group, which we will refer to as a job has been stopped now, meaning the the sleep has stopped counting - it's execution is completely stopped. You can retrieve a list of jobs with the jobs command:

$ jobs
[1]+  Stopped                 sleep 3600
      

This shows the job number (1), its state, and the command that was used to start this job. Let's run another program, stop that too, and have another look at the job listing.

$ cat
Ctrl-z
[2]+  Stopped                 cat
$ jobs
[1]-  Stopped                 sleep 3600
[2]+  Stopped                 cat
      

As expected, the second job is also stopped, and was assigned job number 2. The plus sign (+) following the first job has changed to a minus (-) sign, while the second job is now marked by a plus sign. The plus sign is used to indicate the current job. The bg and fg commands that we will look at shortly, will operate on the current job if no job was specified as a parameter.

Usually, when you are working with jobs, you will want to move jobs to the foreground again. This is done with the fg command. Executing fg without a parameter will put the current job in the foreground. Most shells will print the command that is moved to the foreground to give an indication of what process was moved to the foreground:

$ fg
cat
      

Of course, it's not always useful to put the current job in the foreground. You can put another job in the foreground by adding the job number preceded by the percentage sign (%) as an argument to fg:

$ fg %1
sleep 3600
      

Switching jobs my stopping them and putting them in the foreground is often very useful when the shell is used interactively. For example, suppose that you are editing a file with a text editor, and would like to execute some other command and then continue editing. You could stop the editor with Ctrl-z, execute a command, and put the editor in the foreground again with fg.

Besides running in the foreground, jobs can also run in the background. This means that they are running, but input from the terminal is not redirected to background processes. Most shells do configure background jobs to direct output to the terminal of the shell where they were started.

A process that is stopped can be continued in the background with the bg command:

$ sleep 3600
  
[1]+  Stopped                 sleep 3600
$ bg
[1]+ sleep 3600 &
$
      

You can see that the job is indeed running with jobs:

$ jobs
[1]+  Running                 sleep 3600 &
      

Like fg, you can also move another job than the current job to the background by specifying its job number:

$ bg %1
[1]+ sleep 3600 &
      

You can also run put a job directly in the background when it is started, by adding an trailing ampersand (&) to a command or pipeline. For instance:

$ sleep 3600 &
[1] 5078
      

Each LaTeX document has some basic minimal structure that describes the document. Let's look at an example:

\documentclass[10pt,a4paper]{article} 

\title{The history of symmetric ciphers} 
\author{John Doe} 

\begin{document} 

This is a basic document. 

\end{document} 
      

You can already see the basic syntactical structure of LaTeX commands. A command is started with a backslash, followed by the name of the command. Each macro has a mandatory argument that is placed in accolades, and an optional argument that is placed in square brackets.

Once you have a LaTeX file, you can use the latex command to generate a DVI (Device Independent format) file:

$ latex crypto.tex
This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4)
entering extended mode
(./crypto.tex
LaTeX2e <2003/12/01>
Babel <v3.8d> and hyphenation patterns for american, french, german, ngerman, b
ahasa, basque, bulgarian, catalan, croatian, czech, danish, dutch, esperanto, e
stonian, finnish, greek, icelandic, irish, italian, latin, magyar, norsk, polis
h, portuges, romanian, russian, serbian, slovak, slovene, spanish, swedish, tur
kish, ukrainian, nohyphenation, loaded.
(/usr/share/texmf/tex/latex/base/article.cls
Document Class: article 2004/02/16 v1.4f Standard LaTeX document class
(/usr/share/texmf/tex/latex/base/size10.clo)) (./crypto.aux) [1] (./crypto.aux)
 )
 Output written on crypto.dvi (1 page, 248 bytes).
 Transcript written on crypto.log.
      

As the LaTeX command reports a DVI file is created after running the latex command. You can view this file with an X viewer for DVI files, xdvi:

$ xdvi crypto.dvi
      

This file is not directly printable (although DVI files can be printed with xdvi). An ideal format for printable files is Postscript. You can generate a Postscript file from the DVI file with one simple command:

$ dvips -o crypto.ps crypto.dvi
      

The -o specifies the output (file) for the Postscript document. If this parameter is not specified, the output will be piped through lpr, which will schedule the document to be printed.

PDF (Portable Document Format) is another popular format for electronic documents. PDF can easily be generated from Postscript:

$ ps2pdf crypto.ps
      

The resulting output file will be the same as the input file, with the .ps extension replaced with .pdf.

Now that you know how to create a basic LaTeX document, it is a good time to add some more structure. Structure is added using sections, subsections, and subsubsections. These structural elements are made with respectively the \section, \subsection and \subsubsection commands. The mandatory parameter for a section is the title for the section. Normal sections, subsections and subsubsections are automatically numbered, and added to the table of contents. By adding a star after a section command, for example \section*{title} section numbering is suppressed, and the section is not added to the table of contents. The following example demonstrates how you can use sections:

\documentclass[10pt,a4paper]{article}

\title{The history of symmetric ciphers}
\author{John Doe}

\begin{document}

\section{Pre-war ciphers}

To be done.

\section{Modern ciphers}

\subsection*{Rijndael}

Rijndael is a modern block cipher that was designed by Joan Daemen and
Vincent Rijmen. 

In the year 2000 the US National Institute of Standards and Technologies
selected Rijndael as the winner in the contest for becoming the Advanced
Encryption Standard, the successor of DES.

\end{document}
      

The example above is pretty straightforward, but this is a good time to look at how LaTeX treats end of line characters and empty lines. Empty lines are ignored by LaTeX, making the text a continuous flow. An empty line starts a new paragraph. All paragraphs but the first paragraph are stared with a extra space left of the first word.

Usually you may want to work with different font styles too. LaTeX has some commands that can be used to change the appearance of the current font. The most commonly used font commands are \emph for emphasized text, and \textbf. Have a look at Table 11.2, “LaTeX font styles” for a more extensive list of font styles. Emphasis and bold text are demonstrated in this example paragraph:

Working with font styles is easy. \emp{This text is emphasized} and
\textbf{this text is bold}.
      

After invoking the mutt command, an overview of all e-mails will show up. You can browse through the list of e-mails with the up and down arrow keys, or the <k> and <j> keys.

To read an e-mail, use the <Enter> key, after selecting an e-mail in the overview of e-mails. When reading an e-mail you can use the <Page Up> and <Page Down> to browse through an e-mail. You can still use the navigational keys used to browse the list of e-mail to browse to other e-mails.

If an e-mail has any attachments, you can see them by pressing the <v> key. You can view individual attachments by selecting them and pressing the <Enter> key. To save an attachment to a file, press the <s> key.

You can compose a new e-mail with the <c> key, or reply to a selected e-mail with the <r> key. Mutt will ask you to specify the recipient (To:), and a subject (Subject:). After entering this information an editor is launched (vi is used by default), which you can use to compose the e-mail. After saving the e-mail, and quitting the editor, mutt will give you the opportunity to make any changes to the e-mail. If you decide that you want to alter the e-mail, you can restart the editor with the <e> key. You can change the recipient or the subject with respectively <t> or <s>. Finally, you can send the e-mail by pressing <y>. If you would like to cancel the e-mail, press <q>. Mutt will ask you whether you want to postpone the e-mail. If you do so, you will be given the opportunity to re-do the e-mail the next time you compose a message.