Bash Tips and Tricks – 2

After a period of hibernation, I’m meeting you again with some more interesting tips and tricks related to the bash. Hope you’ll find it more interesting than the previous post.

Enabling and disabling an alias

To list the configured aliases you can use the command ‘alias’ you’ll see something like this,

$ alias
alias ls='ls --color=auto'
alias rm='rm -i'

As you can see, rm is aliased as ‘rm -i‘ (to prompt before every removal). So if you try to remove any file using ‘rm‘, its going to prompt you for confirmation.

$ rm file.txt
rm: remove regular empty file `file.txt'? y

Now if you want the use ‘rm‘ command without the alias additions like rm -i, you can do it in two ways:

1 – Un-aliasing a command by simply prefixing the command with a ‘\’

$ \rm file.txt

2 – Using ‘unalias‘ command

$ unalias rm

The above ‘rm‘ one is just an example to illustrate this, you can also do ‘rm -f‘ for the same

Highlight match with color in grep command

Like bash ‘ls‘ command, grep supports color in its output. Which means you can highlight the text that grep matches with color.
This is controlled by ‘–color‘ option with grep command which basically surround the matching string with the marker find in GREP_COLOR environment variable.

$ grep --color=auto <pattern> <file>

You can also change this color by setting the GREP_COLOR environment variable to different combinations from the color code list given below.

For example, to highlight the matched pattern with foreground color black and background color yellow, you can say..

$ export GREP_COLOR='1;30;43'

The set display attributes list:

0 Reset all attributes
1 Bright
2 Dim
4 Underscore
5 Blink
7 Reverse
8 Hidden

Foreground Colours
30 Black
31 Red
32 Green
33 Yellow
34 Blue
35 Magenta
36 Cyan
37 White

Background Colours
40 Black
41 Red
42 Green
43 Yellow
44 Blue
45 Magenta
46 Cyan
47 White

Handling ‘argument list too long’

I have nearly 200,000 files in one of my log directory out of which number of files created in 2007 is 120,000. So whenever I try to do apply some command such as rm, ls or cp etc. on those big set of “*2007*.log” files, I used to get,

$ ls *2007*.log
bash: /bin/ls: Argument list too long

$ mv *2007*.log /backup
bash: /bin/mv: Argument list too long

“Argument list too long” error is occurring due to the limitation of the above commands to handle large number of arguments. But you can get the job done easily using the ‘find‘ command. For example, to copy the files to a separate location, you can say,

$ find .  -name "*2007*.log" -exec cp {} /backup/ \;

Same results can be achieved by the following as well..

find .  -name "k*2007*.log" | while read FILE
    do
    ...
    <some operation on $FILE>
    ...
done

Process substitution

This trick allows you to use a process almost anywhere you can use a file. To illustrate, let’s consider the diff command. Most versions of diff require you to pass exactly two file names as arguments. But what if we want to diff something, like the contents of a directory, that doesn’t necessarily exist in a file? This is where we can use process substitution. For example, to diff the contents of two directories, you could use:

diff <(find dir1) <(find dir2)

The syntax <(command) creates a named pipe, and attaches command’s STDOUT to the pipe. So, anything that reads from the pipe will actually be reading the output of command. To prove this to yourself, try the following:

$ echo <(/bin/true)
/dev/fd/63

$ ls -l <(/bin/true)
lr-x------  1 chamith chamith 64 Jul 13 21:50 /dev/fd/63 -> pipe:[723168]

$ file <(/bin/true)
/dev/fd/63: broken symbolic link to pipe:[728714]

Similarly, you can use the syntax >(command) to have the command read from the pipe. As an example:

tar cvf >(gzip -c > dir.tar.gz) dir

Obviously, there are better ways to accomplish taring and compressing, but the point was to use process substitution.

pushd / popd

Bash will keep a history of the directories you visit, you just have to ask. Bash stores the history in a stack and uses the commands pushd and popd to manage the stack.

pushd foo – move the current directory onto the stack and change to the ,em>foo directory.
popd – pops the top directory off of the stack and moves you into it.

We’re opening files all over the file system, internal code, vendor code, templates, configuration files, logs. Because of this we like the ability to take a detour on the file system and still navigate back to our working directory of the day. I think these commands are so useful that I alias’d them in my .bashrc :

alias cd="pushd"
alias bd="popd"

Now the ‘cd’ command manages the stack for me as well as changing directories. Aliasing popd to bd is an easy to remember and easy to type way to move back up the stack, think “change dir” and “back dir”

Hope you’ll find this post useful. Feel free to share your ideas about this post.

13 security practices for SysAdmins

This information has been compiled to help system administrators certify that good security practices are being used BEFORE a computer is connected to the network.

Installing System Patches

It is recommended that based on the requirement, you install every patch recommended for your computer which isn’t
yet installed. Since some patches restore default configurations, it’s important that patches are put in place before any further security precautions are taken.

Before Recording System Defaults

Before starting to record system defaults, a directory should be created to store them. For example;

mkdir /usr/adm/checks

If an unauthorized user does gain access to root privileges on the computer and changes the accounting system, the
administrator will still have an original copy of it for comparison. For safety, the system administrator should check the files against the original about once a month.

Recording SUID and SGID Programs

Before any software is added to the basic operating system release, the system administrator should check for SUID and SGID programs. If unauthorized access occurs, frequently the intruder will leave a program that enables privileged
re-entry. The list of SUID and SGID programs should be stored both on and off the computer. The version on the computer will be used by a daily cron job to check for changes, while the version stored off of the computer will ensure that even if root access is acquired, a record of the system’s original state is available.

The command to list SUID and SGID files is:

find / -type f \( -perm -002000 -o -perm -004000 \)

-type f: looks only at regular files
-perm:   checks for permissions

-002000: checks for SGID programs
-004000: checks for SUID programs

Check and Record Permissions on all Device Files

By changing the permissions on device files, an unauthorized user can gain access to devices, using this access to change files, impersonate another user, or listen in on conversations. Record the permissions on the device files on and off the computer using:

ls -al /dev/* | sort > /usr/adm/checks/devices

Passwords and Shells on System Accounts

Check the system password file to ensure that all accounts have passwords. Many vendors ship their computers with no passwords on the system accounts. System accounts such as bin, lp, and sync should have a ‘*’ for the password field. No account should be left without a password.

Also, the system administrator should check to see if the computer comes with any passwords already assigned. Some
vendors give default passwords to system accounts. Since anyone who has the same type of system knows what the default passwords are, passwords should be changed immediately.

Every account needs to have a shell assigned to it. Most administrative accounts should have /bin/nologin as the shell, which
would disallow crackers from gaining shell access using obscure system holes.

Expire Inactive Accounts

Computers with large numbers of users tend to have accounts that become inactive. The beginning of a new fiscal year often
brings changes in who is using the computer, as users’ funding sources change. The system administrator needs to be sensitive to those accounts that become inactive, and disable them by replacing the password field in the /etc/password file with an ‘*’. If the user has left important data on the computer, eventually they will contact the system administrator to make arrangements to retrieve the data. Once this data is retrieved, the account should be removed.

Restrict Root Login to the Console

The ability to login to the root account should be restricted to the console. Anyone not at the console should have to use ’su’ to
become root. Tries to ’su’ are recorded in a file in /usr/adm such as /usr/adm/sulog, for accounting purposes

Check for Duplicate Groups

Replace any duplicated group with a group of its own. This will remove ambiguity and make membership in a group clearer.

Do Not Establish Guest Accounts

Do not establish accounts for guest usage. These accounts, often appearing as an account with login guest and password
account, are common holes exploited by unauthorized users. Every user of the computer should undergo the same security procedures, receive the same security briefing, and be held accountable to the same standards. When users are finished using the computer, their accounts should be removed from the password file.

‘remote’ Commands

Commands preceded by the letter ‘r’, such as ‘rlogin‘ or ‘rsh‘, should be disabled. They are a source of many attacks on sites
across the Internet. If you must use ‘r’ commands, make sure you filter the TCP ports (512,513,514) at the router; it is important to note this will only stop outsiders from abusing the commands.

Double Check the System Before Long Weekends

Double check the computer before long weekends to ensure there are no security problems with it. A backup just
before a long weekend is advisable.

Do a Monthly System Check

Run the cron script against the cron stored on the removable media in case the unauthorized user gained root access and altered the system without being noticed.

System Security Diary

Keep a diary of the security checks done on the computer and what their results are. Also, document what actions are taken if holes are found or problems occur. If there is a problem, others will want to know what the system administrator has been doing to secure the computer.

Hope these tips would help you in your day-to-day life.

Bash Tips and Tricks – 1

Bash, or the Bourne Again Shell, is the default shell in most Linux distributions. The popularity of the bash shell amongst Linux and UNIX users is no accident. It has many features to enhance user-friendliness and productivity. Unfortunately, you can’t take advantage of those features unless you know they exist.

When I first started using Linux, the only bash feature I took advantage of was going back through the command history using the up arrow. I soon learned additional features by watching others and asking questions. In this article, I’d like to share some bash tricks I’ve learned over the years.

This article isn’t meant to cover all of the features of the bash shell; that would require a book, and plenty of books are available that cover this topic. Instead, this article is a summary of the bash tricks I use most often and would be lost without.

Brace Expansion

One of my favorite bash tricks is brace expansion. Brace expansion takes a list of strings separated by commas and expands those strings into separate arguments for you. The list is enclosed by braces, the symbols { and }, and there should be no spaces around the commas. For example:

$ echo {one,two,red,blue}
one two red blue

Using brace expansion as illustrated in this simple example doesn’t offer too much to the user. In fact, the above example requires typing two more characters than simply typing:

echo one two red blue

which produces the same result. However, brace expansion becomes quite useful when the brace-enclosed list occurs immediately before, after or inside another string:

$ echo {one,two,red,blue}baloon
onebaloon twobaloon redbaloon bluebaloon

$ echo fish{one,two,red,blue}
fishone fishtwo fishred fishblue

$ echo fi{one,two,red,blue}sh
fionesh fitwosh firedsh fibluesh

Notice that there are no spaces inside the brackets or between the brackets and the adjoining strings. If you include spaces, it breaks things:

$ echo {one, two, red, blue }fi
{one, two, red, blue }fi

$ echo "{one,two,red,blue} fi"
{one,two,red,blue} fi

However, you can use spaces if they’re enclosed in quotes outside the braces or within an item in the comma-separated list:

$ echo {"one ","two ","red ","blue "}fish
one fish two fish red fish blue fish

$ echo {one,two,red,blue}" fish"
one fish two fish red fish blue fish

You also can nest braces, but you must use some caution here too:

$ echo {{1,2,3},1,2,3}
1 2 3 1 2 3

$ echo {{1,2,3}1,2,3}
11 21 31 2 3

Now, after all these examples, you might be thinking to yourself, “Those are great parlor tricks, but why should I care about brace expansion?”

Brace expansion becomes useful when you need to make a backup of a file. This is why it’s my favorite shell trick. I use it almost every day when I need to make a backup of a config file before changing it. For example, if I’m making a change to my Apache configuration, I can do the following and save some typing:

$ cp /etc/apache2/apache2.conf{,.bak}

Notice that there is no character between the opening brace and the first comma. It’s perfectly acceptable to do this and is useful when adding characters to an existing filename or when one argument is a substring of the other. Then, if I need to see what changes I made later in the day, I use the diff command and reverse the order of the strings inside the braces:

$ diff /etc/apache2/apache2.conf{.bak,}
1050a1051
> # I added this comment earlier

Redirecting Standard Error

Have you ever looked for a file using the find command, only to learn the file you were looking for is lost in a sea of permission denied error messages that quickly fill your terminal window?

If you are the administrator of the system, you can become root and execute find again as root. Because root can read any file, you don’t get that error anymore. Unfortunately, not everyone has root access on the system being used. Besides, it’s bad practice to be root unless it’s absolutely necessary. So what can you do?

One thing you can do is redirect your output to a file. Basic output redirection should be nothing new to anyone who has spent a reasonable amount of time using any UNIX or Linux shell, so I won’t go into detail regarding the basics of output redirection. To save the useful output from the find command, you can redirect the output to a file:

$ find /  -name foo > output.txt

You still see the error messages on the screen but not the path of the file you’re looking for. Instead, that is placed in the file output.txt. When the find command completes, you can cat the file output.txt to get the location(s) of the file(s) you want.

That’s an acceptable solution, but there’s a better way. Instead of redirecting the standard output to a file, you can redirect the error messages to a file. This can be done by placing a 2 directly in front of the redirection angle bracket. If you are not interested in the error messages, you simply can send them to /dev/null:

$ find /  -name foo 2> /dev/null

This shows you the location of file foo, if it exists, without those pesky permission denied error messages. I almost always invoke the find command in this way.

The number 2 represents the standard error output stream. Standard error is where most commands send their error messages. Normal (non-error) output is sent to standard output, which can be represented by the number 1. Because most redirected output is the standard output, output redirection works only on the standard output stream by default. This makes the following two commands equivalent:

$ find / -name foo > output.txt
$ find / -name foo 1> output.txt

Sometimes you might want to save both the error messages and the standard output to file. This often is done with cron jobs, when you want to save all the output to a log file. This also can be done by directing both output streams to the same file:

$ find / -name foo > output.txt 2> output.txt

This works, but again, there’s a better way to do it. You can tie the standard error stream to the standard output stream using an ampersand. Once you do this, the error messages goes to wherever you redirect the standard output:

$ find / -name foo > output.txt 2>&1

One caveat about doing this is that the tying operation goes at the end of the command generating the output. This is important if piping the output to another command. This line works as expected:

find -name test.sh 2>&1 | tee /tmp/output2.txt

but this line doesn’t:

find -name test.sh | tee /tmp/output2.txt 2>&1

and neither does this one:

find -name test.sh 2>&1 > /tmp/output.txt

I started this discussion on output redirection using the find command as an example, and all the examples used the find command. This discussion isn’t limited to the output of find, however. Many other commands can generate enough error messages to obscure the one or two lines of output you need.

Output redirection isn’t limited to bash, either. All UNIX/Linux shells support output redirection using the same syntax.

Using Loops from the Command Line

One last tip I’d like to offer is using loops from the command line. The command line is not the place to write complicated scripts that include multiple loops or branching. For small loops, though, it can be a great time saver. Unfortunately, I don’t see many people taking advantage of this. Instead, I frequently see people use the up arrow key to go back in the command history and modify the previous command for each iteration.

If you are not familiar with creating for loops or other types of loops, many good books on shell scripting discuss this topic. A discussion on for loops in general is an article in itself.

You can write loops interactively in two ways. The first way, and the method I prefer, is to separate each line with a semicolon. A simple loop to make a backup copy of all the files in a directory would look like this:

$ for file in * ; do cp $file $file.bak; done

Another way to write loops is to press Enter after each line instead of inserting a semicolon. bash recognizes that you are creating a loop from the use of the for keyword, and it prompts you for the next line with a secondary prompt. It knows you are done when you enter the keyword done, signifying that your loop is complete:

$ for file in *
> do cp $file $file.bak
> done

And Now for Something Completely Different

When I originally conceived this article, I was going to name it “Stupid bash Tricks”, and show off some unusual, esoteric bash commands I’ve learned. The tone of the article has changed since then, but there is one stupid bash trick I’d like to share.

About 2 years ago, a Linux system I was responsible for ran out of memory. Even simple commands, such as ls, failed with an insufficient memory error. The obvious solution to this problem was simply to reboot. One of the other system administrators wanted to look at a file that may have held clues to the problem, but he couldn’t remember the exact name of the file. We could switch to different directories, because the cd command is part of bash, but we couldn’t get a list of the files, because even ls would fail. To get around this problem, the other system administrator created a simple loop to show us the files in the directory:

$ for file in *; do echo $file; done

This worked when ls wouldn’t, because echo is a part of the bash shell, so it already is loaded into memory. It’s an interesting solution to an unusual problem. Now, can anyone suggest a way to display the contents of a file using only bash built-ins?

Conclusion

Bash has many great features to make life easier for its users. I hope this summary of bash tricks I like to use has shown you some new ways to take advantage of the power bash has to offer.

Mail server setup with Qmail

What is Qmail?

Qmail is an Internet Mail Transfer Agent (MTA) for UNIX-like operating systems. It’s a drop-in replacement for the Sendmail system provided with UNIX operating systems. Qmail uses the Simple Mail Transfer Protocol (SMTP) to exchange messages with MTA’s on other systems.

Why Qmail?

Your operating system might already have an MTA, probably Postfix or Sendmail, so if you’re reading this document you’re probably looking for something different. Some of the advantages of Qmail over vendor-provided MTA’s include:

• Security – Qmail was designed for high security. Sendmail has a long history of serious security problems. When Sendmail was written, the internet was a much friendlier place. Everyone knew everyone else, and there was little need to design and code for high security. Today’s Internet is a much more hostile environment for network servers. Sendmail’s author, Eric Allman, and the current maintainer, Claus Assman, have done a good job of tightening up the program, but nothing short of a redesign can achieve “true” security.
• Performance – Qmail parallelizes mail delivery, performing up to 20 deliveries simultaneously, by default.
• Reliability – Once Qmail accepts a message, it guarantees that it won’t be lost. Qmail also supports a new mailbox format that works reliably even over NFS without locking.
• Simplicity – Qmail is smaller than any other equivalently-featured MTA.

The Qmail web page, has a comprehensive list of Qmail’s features.

Comparison with other MTA’s

A book could be written about this topic, but it would be tedious reading. Here’s a quick comparison of Qmail with some of the most common UNIX MTA’s.

MTA	Maturity	Security	Features	Performance	Sendmailish	Modular
Qmail	medium	high	high	high	addons	yes
Sendmail	high	low	high	low	x	no
Postfix	medium	high	high	high	yes	yes
Exim	medium	low	high	medium	yes	no
Courier	low	medium	high	medium	optional	yes

NOTE: Sendmailish means the MTA behaves like Sendmail in some ways that would make a switch from Sendmail to the alternative MTA more user-transparent, such as the use of .forward files, /etc/aliases, and delivery to /var/spool/mail.

Preparation

Before 2007-11-30, Qmail’s restrictive licensing regarding the distribution of pre-built packages meant that it was usually installed from a source code distribution. This may change in the future, especially if daemontools and ucspi-tcp are placed in the public domain. For now, though, source code is still the preferred distribution method for Qmail.

Before installing Qmail on a system, especially if this is your first Qmail installation, there are a few things you need to think about.

• If possible, install Qmail on a staging environment. This will give you a chance to make mistakes without losing important mail or interrupting mail service to your users.
• If you don’t have a spare, and your system is already handling mail using sendmail, smail, or some other MTA, you can install and test most pieces of Qmail without interfering with the existing service.
• When migrating a system from some other MTA to Qmail–even if you’ve got some Qmail experience under your belt–it’s a good idea to formulate a plan.

Note: The Qmail bin directory must reside on a file-system that allows the use of executable and setuid() files. Some OS distributions automatically mount /var with the nosuid or noexec options enabled. On such systems, either these options should be disabled or /var/qmail/bin should reside on another filesystem without these options enabled.

Download the soure

OK, so you’ve got a system meeting the requirements ready for installing Qmail. The first step is to download the source code for Qmail and any other add-ons. You’ll need qmail, of course, and you should probably also get ucspi-tcp and daemontools:

• Qmail – http://www.qmail.org/netqmail-1.06.tar.gz
• ucspi-tcp – http://cr.yp.to/ucspi-tcp/ucspi-tcp-0.88.tar.gz
• daemontools – http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
  

Note: If any of the links fail, it’s probably because the package has been updated. In that case, you should go to http://cr.yp.to/software.html and follow the links to download the current version. It’s possible that upgraded versions aren’t compatible with the following instructions, so be sure to read the release notes in the “Upgrading from previous versions…” sections.

Unpack the distribution

To continue from this point onwards, you need a working C compiler and the tarballs. Next, copy or move the tarballs to the directory you want to do the work in. /usr/local/src is a good choice for qmail and ucspi-tcp. daemontools should be built under /package.

At this time you probably want to become root, if you’re not already.

    su
    umask 022
    mkdir -p /usr/local/src
    mv netqmail-1.06.tar.gz ucspi-tcp-0.88.tar.gz /usr/local/src
    mkdir -p /package
    mv daemontools-0.76.tar.gz /package
    chmod 1755 /package

Now you can unpack the packages.

    cd /usr/local/src
    gunzip netqmail-1.06.tar.gz
    tar xpf netqmail-1.06.tar
    gunzip ucspi-tcp-0.88.tar.gz
    tar xpf ucspi-tcp-0.88.tar
    rm *.tar      # optional, unless space is very tight
    cd /package
    gunzip daemontools-0.76.tar.gz
    tar xpf daemontools-0.76.tar
    rm *.tar      # optional, again

There should now be directories called /usr/local/src/netqmail-1.06, /usr/local/src/ucspi-tcp-0.88, and /package/admin/daemontools-0.76.

Create Directories

Since Qmail’s installation program creates the subdirectories as they’re needed, you only need to create the Qmail “home” directory:

    mkdir /var/qmail

And on to the next section.

Create users and groups

The easiest way to create the necessary users and groups is to create a little script file to do it for you. In the source directory you’ll find a file called INSTALL.ids. It contains the command lines for many platforms, so copying the file to another name and editing that is quick and easy.

    cd /usr/local/src/netqmail-1.06
    cp INSTALL.ids IDS

Then, using your favorite editor, remove all of the file except the lines you want. For example, here’s what IDS would look like for Linux after editing:

    groupadd nofiles
    useradd qmaild -g nofiles -d /var/qmail -s /usr/sbin/nologin
    useradd alias -g nofiles -d /var/qmail/alias -s /usr/sbin/nologin
    useradd qmaill -g nofiles -d /var/qmail -s /usr/sbin/nologin
    useradd qmailp -g nofiles -d /var/qmail -s /usr/sbin/nologin
    groupadd qmail
    useradd qmailq -g qmail -d /var/qmail -s /usr/sbin/nologin
    useradd qmailr -g qmail -d /var/qmail -s /usr/sbin/nologin
    useradd qmails -g qmail -d /var/qmail -s /usr/sbin/nologin

Then to run it, either use chmod to make it executable or run it with sh:

    chmod 700 IDS
    ./IDS

Let’s build Qmail

Now you can start building Qmail. Change to the /usr/local/src/netqmail-1.05/netqmail-1.05 directory and let’s get started:

    cd /usr/local/src/netqmail-1.06

Now type the following:

    make setup check

After the build is complete, you’ll need to do your post installation configuration. A couple of scripts are provided to make this job a lot easier.

If your DNS is configured properly, this script should be all you need at this point:

    ./config

If, for some reason, config can’t find your hostname in DNS, you’ll have to run the config-fast script:

    ./config-fast the.full.hostname

For example, if your domain is example.com and the hostname of your computer is foobar, your config-fast line would look like this:

    ./config-fast foobar.example.com

Install ucspi-tcp

Earlier, you unpacked the qmail, ucspi-tcp, and daemontools tarballs. Now change to the ucspi-tcp directory:

    cd /usr/local/src/ucspi-tcp-0.88

Then do:

    patch < /usr/local/src/netqmail-1.06/other-patches/ucspi-tcp-0.88.errno.patch
    make
    make setup check

That’s it. ucspi-tcp is installed.

Install daemontools

Change to the daemontools build directory:

    cd /package/admin/daemontools-0.76

Then do:

    cd src
    patch < /usr/local/src/netqmail-1.06/other-patches/daemontools-0.76.errno.patch
    cd ..
    package/install

Start Qmail

The /var/qmail/boot directory contains example qmail boot scripts for different configurations: /var/spool/mail vs. $HOME/Mailbox, using procmail or dot-forward, and various combinations of these. Feel free to examine these, but for our installation, we’ll use the following script:

/var/qmail/rc

#!/bin/sh

# Using stdout for logging
# Using control/defaultdelivery from qmail-local to deliver messages by default

exec env - PATH="/var/qmail/bin:$PATH" \
qmail-start "`cat /var/qmail/control/defaultdelivery`"

Note: This script uses backquotes (`), not single quotes ('). For best results, copy and paste the scripts in this guide instead of retyping them.

Use your editor to create the above /var/qmail/rc, then execute these commands:

    chmod 755 /var/qmail/rc
    mkdir /var/log/qmail

At this point you need to decide the default delivery mode for messages that aren’t delivered by a .qmail file. The following table outlines some common choices.

Mailbox format	Name	Location	defaultdelivery	Comments
mbox	Mailbox	$HOME	./Mailbox	most common, works with most MUA’s
maildir	Maildir	$HOME	./Maildir/	more reliable, less MUA support
mbox	username	/var/spool/mail	See INSTALL.vsm	traditional UNIX mailbox

To select your default mailbox type, just enter the defaultdelivery value from the table into /var/qmail/control/defaultdelivery. E.g., to select the standard Qmail Mailbox delivery, do:

    echo ./Maildir > /var/qmail/control/defaultdelivery

System startup files

If you were to manually execute the /var/qmail/rc script, qmail would be partially started. But we want qmail started up automatically every time the system is booted and we want it shut down cleanly when the system is halted.

This is accomplished by creating a startup/shutdown script like the following in /var/qmail/bin/qmailctl:

#!/bin/sh

# description: the qmail MTA

PATH=/var/qmail/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin
export PATH

QMAILDUID=`id -u qmaild`
NOFILESGID=`id -g qmaild`

case "$1" in
  start)
    echo "Starting qmail"
    if svok /service/qmail-send ; then
      svc -u /service/qmail-send /service/qmail-send/log
    else
      echo "qmail-send supervise not running"
    fi
    if svok /service/qmail-smtpd ; then
      svc -u /service/qmail-smtpd /service/qmail-smtpd/log
    else
      echo "qmail-smtpd supervise not running"
    fi
    if [ -d /var/lock/subsys ]; then
      touch /var/lock/subsys/qmail
    fi
    ;;
  stop)
    echo "Stopping qmail..."
    echo "  qmail-smtpd"
    svc -d /service/qmail-smtpd /service/qmail-smtpd/log
    echo "  qmail-send"
    svc -d /service/qmail-send /service/qmail-send/log
    if [ -f /var/lock/subsys/qmail ]; then
      rm /var/lock/subsys/qmail
    fi
    ;;
  stat)
    svstat /service/qmail-send
    svstat /service/qmail-send/log
    svstat /service/qmail-smtpd
    svstat /service/qmail-smtpd/log
    qmail-qstat
    ;;
  doqueue|alrm|flush)
    echo "Flushing timeout table and sending ALRM signal to qmail-send."
    /var/qmail/bin/qmail-tcpok
    svc -a /service/qmail-send
    ;;
  queue)
    qmail-qstat
    qmail-qread
    ;;
  reload|hup)
    echo "Sending HUP signal to qmail-send."
    svc -h /service/qmail-send
    ;;
  pause)
    echo "Pausing qmail-send"
    svc -p /service/qmail-send
    echo "Pausing qmail-smtpd"
    svc -p /service/qmail-smtpd
    ;;
  cont)
    echo "Continuing qmail-send"
    svc -c /service/qmail-send
    echo "Continuing qmail-smtpd"
    svc -c /service/qmail-smtpd
    ;;
  restart)
    echo "Restarting qmail:"
    echo "* Stopping qmail-smtpd."
    svc -d /service/qmail-smtpd /service/qmail-smtpd/log
    echo "* Sending qmail-send SIGTERM and restarting."
    svc -t /service/qmail-send /service/qmail-send/log
    echo "* Restarting qmail-smtpd."
    svc -u /service/qmail-smtpd /service/qmail-smtpd/log
    ;;
  cdb)
    tcprules /etc/tcp.smtp.cdb /etc/tcp.smtp.tmp < /etc/tcp.smtp
    chmod 644 /etc/tcp.smtp.cdb
    echo "Reloaded /etc/tcp.smtp."
    ;;
  help)
    cat <<HELP
   stop -- stops mail service (smtp connections refused, nothing goes out)
  start -- starts mail service (smtp connection accepted, mail can go out)
  pause -- temporarily stops mail service (connections accepted, nothing leaves)
   cont -- continues paused mail service
   stat -- displays status of mail service
    cdb -- rebuild the tcpserver cdb file for smtp
restart -- stops and restarts smtp, sends qmail-send a TERM & restarts it
doqueue -- schedules queued messages for immediate delivery
 reload -- sends qmail-send HUP, rereading locals and virtualdomains
  queue -- shows status of queue
   alrm -- same as doqueue
  flush -- same as doqueue
    hup -- same as reload
HELP
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|doqueue|flush|reload|stat|pause|cont|cdb|queue|help}"
    exit 1
    ;;
esac

exit 0

Create the script using your editor.

Make the qmailctl script executable and link it to a directory in your path:

    chmod 755 /var/qmail/bin/qmailctl
    ln -s /var/qmail/bin/qmailctl /usr/bin

The supervise scripts

Now create the supervise directories for the Qmail services:

    mkdir -p /var/qmail/supervise/qmail-send/log
    mkdir -p /var/qmail/supervise/qmail-smtpd/log

Create the /var/qmail/supervise/qmail-send/run file:

#!/bin/sh
exec /var/qmail/rc

Create the /var/qmail/supervise/qmail-send/log/run file:

#!/bin/sh
exec /usr/local/bin/setuidgid qmaill /usr/local/bin/multilog t /var/log/qmail

Create the /var/qmail/supervise/qmail-smtpd/run file:

#!/bin/sh

QMAILDUID=`id -u qmaild`
NOFILESGID=`id -g qmaild`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
LOCAL=`head -1 /var/qmail/control/me`

if [ -z "$QMAILDUID" -o -z "$NOFILESGID" -o -z "$MAXSMTPD" -o -z "$LOCAL" ]; then
    echo QMAILDUID, NOFILESGID, MAXSMTPD, or LOCAL is unset in
    echo /var/qmail/supervise/qmail-smtpd/run
    exit 1
fi

if [ ! -f /var/qmail/control/rcpthosts ]; then
    echo "No /var/qmail/control/rcpthosts!"
    echo "Refusing to start SMTP listener because it'll create an open relay"
    exit 1
fi

exec /usr/local/bin/softlimit -m 5000000 \
    /usr/local/bin/tcpserver -v -R -l "$LOCAL" -x /etc/tcp.smtp.cdb -c "$MAXSMTPD" \
        -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp /var/qmail/bin/qmail-smtpd 2>&1

NOTE: concurrencyincoming isn’t a standard qmail control file. It’s a feature of the above script. Also, that’s -1 (dash one) on the LOCAL line and -l (dash ell) on the tcpserver line.

Create the concurrencyincoming control file:

    echo 20 > /var/qmail/control/concurrencyincoming
    chmod 644 /var/qmail/control/concurrencyincoming

Create the /var/qmail/supervise/qmail-smtpd/log/run file:

#!/bin/sh
exec /usr/local/bin/setuidgid qmaill /usr/local/bin/multilog t /var/log/qmail/smtpd

Make the run files executable:

    chmod 755 /var/qmail/supervise/qmail-send/run
    chmod 755 /var/qmail/supervise/qmail-send/log/run
    chmod 755 /var/qmail/supervise/qmail-smtpd/run
    chmod 755 /var/qmail/supervise/qmail-smtpd/log/run

Then set up the log directories:

    mkdir -p /var/log/qmail/smtpd
    chown qmaill /var/log/qmail /var/log/qmail/smtpd

Finally, link the supervise directories into /service:

    ln -s /var/qmail/supervise/qmail-send /var/qmail/supervise/qmail-smtpd /service

The /service directory is created when daemontools is installed.

NOTE: The Qmail system will start automatically shortly after these links are created. If you don’t want it running yet, do:

    qmailctl stop

SMTP access controll

Allow the local host to inject mail via SMTP:

    echo '127.:allow,RELAYCLIENT=""' >>/etc/tcp.smtp
    qmailctl cdb

Verify that nothing is listening to the SMTP port (25). Culprits could be the old MTA, inetd, or xinetd. The following command should produce no output (unless the qmail-smtpd service is running):

    netstat -a | grep smtp

If something is running, make sure it’s not Qmail by doing:

    qmailctl stop

The repeat the netstat check:

    netstat -a | grep smtp

Create system aliases

There are three system aliases that should be created on all qmail installations:

Alias	Purpose
postmaster	RFC 2821 required, points to the mail adminstrator (you)
mailer-daemon	de facto standard recipient for some bounces
root	redirects mail from privileged account to the system administrator
abuse	de facto standard recipient for abuse complaints

To create these aliases, decide where you want each of them to go (a local user or a remote address) and create and populate the appropriate .qmail files. For example, say local user dave is both the system and mail administrator:

    echo dave > /var/qmail/alias/.qmail-root
    echo dave > /var/qmail/alias/.qmail-postmaster
    ln -s .qmail-postmaster /var/qmail/alias/.qmail-mailer-daemon
    ln -s .qmail-postmaster /var/qmail/alias/.qmail-abuse
    chmod 644 /var/qmail/alias/.qmail-root /var/qmail/alias/.qmail-postmaster

Start Qmail

If you stopped qmail above after creating the links in /service, you should restart it now:

    qmailctl start

Test the installation

Qmail should now be running. First run qmailctl stat to verify that the services are up and running:

    # qmailctl stat
    /service/qmail-send: up (pid 30303) 187 seconds
    /service/qmail-send/log: up (pid 30304) 187 seconds
    /service/qmail-smtpd: up (pid 30305) 187 seconds
    /service/qmail-smtpd/log: up (pid 30308) 187 seconds
    messages in queue: 0
    messages in queue but not yet preprocessed: 0

All four services should be “up” for more than a second. If they’re not, you’ve probably got a typo in the associated run script or you skipped one or more steps in creating the necessary files, directories, or links. Go back through the installation step-by-step and double check your work. You can also download and run the inst_check script, available from http://www.filedropper.com/qmailinstcheck . For example:

    # sh inst_check
    ! /var/log/qmail has wrong owner, should be qmaill
    ...try: chown qmaill /var/log/qmail
    #

If inst_check finds problems, fix them and re-run it. When everything looks right, inst_check will report:

    Congratulations, your Qmail installation looks good!

Configuration

All of Qmail’s system configuration files, (with the extension .qmail) files in ~alias, reside in /var/qmail/control. The qmail-control man page contains a table like the following:

Control	Default	Used by	Purpose
badmailfrom	none	qmail-smtpd	blacklisted From addresses
bouncefrom	MAILER-DAEMON	qmail-send	username of bounce sender
bouncehost	me	qmail-send	hostname of bounce sender
concurrencyincoming	none	/service/qmail-smtpd/run	max simultaneous incoming SMTP connections
concurrencylocal	10	qmail-send	max simultaneous local deliveries
concurrencyremote	20	qmail-send	max simultaneous remote deliveries
defaultdelivery	none	/var/qmail/rc	default .qmail file
defaultdomain	me	qmail-inject	default domain name
defaulthost	me	qmail-inject	default host name
databytes	0	qmail-smtpd	max number of bytes in message (0=no limit)
doublebouncehost	me	qmail-send	host name of double bounce sender
doublebounceto	postmaster	qmail-send	user to receive double bounces
envnoathost	me	qmail-send	default domain for addresses without “@”
helohost	me	qmail-remote	host name used in SMTP HELO command
idhost	me	qmail-inject	host name for Message-ID’s
localiphost	me	qmail-smtpd	name substituted for local IP address
locals	me	qmail-send	domains that we deliver locally
me	FQDN of system	various	default for many control files
morercpthosts	none	qmail-smtpd	secondary rcpthosts database
percenthack	none	qmail-send	domains that can use “%”-style relaying
plusdomain	me	qmail-inject	domain substituted for trailing “+”
qmqpservers	none	qmail-qmqpc	IP addresses of QMQP servers
queuelifetime	604800	qmail-send	seconds a message can remain in queue
rcpthosts	none	qmail-smtpd	domains that we accept mail for
smtpgreeting	me	qmail-smtpd	SMTP greeting message
smtproutes	none	qmail-remote	artificial SMTP routes
timeoutconnect	60	qmail-remote	how long, in seconds, to wait for SMTP connection
timeoutremote	1200	qmail-remote	how long, in seconds, to wait for remote server
timeoutsmtpd	1200	qmail-smtpd	how long, in seconds, to wait for SMTP client
virtualdomains	none	qmail-send	virtual domains and users

For more information about a particular control file, see the man page for the module listed under “Used by”.

I think you have successfully setup up your Qmail SMTP server. I’m hoping to meet you again with another couple of HOWTOs on “Qmail configuration – smarthosts, multiple domains, relaying, etc.” and “Running a POP server with Qmail”

Cheers!

Configure Squid to control web access

Squid is a proxy server and web cache daemon. It has a wide variety of uses, from speeding up a web server by caching repeated requests, to caching web, DNS and other computer network lookups for a group of people sharing network resources, to aiding security by filtering traffic. Squid is primarily used for HTTP and FTP and it includes limited support for several other protocols such as TLS, SSL, Internet Gopher and HTTPS and the development version of Squid includes IPv6 and ICAP support too.

In this article I’m not going to cover the installation process of Squid-cache. My focus will be on the access control based configuration of Squid-cache for various requirements and also I’ll be covering how to fine tune the other applications to work with Squid, such as the firewall. In other words I’m gonna talk about access-controls (ACLs) in squid.conf and some post configurations.

The “/etc/squid/squid.conf” file

The main Squid configuration file is squid.conf, and, like most Linux applications, Squid needs to be restarted for changes to the configuration file can take effect.

Squid will fail to start if you don’t give your server a hostname. You can set this with the visible_hostname parameter. Here, the hostname is set to the real name of the server ‘myhost’.

visible_hostname myhost

You can limit users’ ability to browse the Internet with access control lists (ACLs). Each ACL line defines a particular type of activity, such as an access time or source network, they are then linked to an http_access statement that tells Squid whether or not to deny or allow traffic that matches the ACL.

Squid matches each Web access request it receives by checking the http_access list from top to bottom. If it finds a match, it enforces the allow or deny statement and stops reading further. You have to be careful not to place a deny statement in the list that blocks a similar allow statement below it.

NOTE: The final http_access statement denies everything, so it is best to place new http_access statements above that statement.

Squid has a minimum required set of ACL statements in the ACCESS_CONTROL section of the squid.conf file. It is best to put new customized entries right after this list to improve the readability.

Restricting web access by time

You can create access control lists with time parameters. For example, you can allow only business hour access from the home network, while always restricting access to host 192.168.1.10.

#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/24
acl business_hours time M T W H F 9:00-17:00
acl RestrictedHost src 192.168.1.10

#
# Add this at the top of the http_access section of squid.conf
#
http_access deny RestrictedHost
http_access allow home_network business_hours

Or, you can allow morning access only:

#
# Add this to the bottom of the ACL section of squid.conf
#
acl morning_hours time 08:00-12:00

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow morning_hours

Restricting access to specific URLs

Squid is also capable of reading files containing lists of web sites and/or domains for use in ACLs. In this example we create to lists in files named /etc/squid/allowed-sites.acl and /etc/squid/restricted-sites.acl

# File: /etc/squid/allowed-sites.acl
www.gnu.org
mysite.com

# File: /etc/squid/restricted-sites.acl
www.restricted.com
illegal.com

These can then be used to always block the restricted sites and permit the allowed sites during working hours. This can be illustrated by expanding our previous example slightly.

#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/24
acl business_hours time M T W H F 9:00-17:00
acl GoodSites dstdomain "/etc/allowed-sites.acl"
acl BadSites  dstdomain "/etc/restricted-sites.acl"

#
# Add this at the top of the http_access section of squid.conf
#
http_access deny BadSites
http_access allow home_network business_hours GoodSites

Restricting web access by IP address

You can create an access control list that restricts web access to users on certain networks. In this case, it’s an ACL that defines a home network of 192.168.1.0.

#
# Add this to the bottom of the ACL section of squid.conf
#
acl home_network src 192.168.1.0/255.255.255.0

You also have to add a corresponding http_access statement that allows traffic that matches the ACL:

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow home_network

Password based authentication using NCSA

You can configure Squid to prompt users for a username and password when they are browsing any URLs. Squid comes with a program called ncsa_auth that reads any NCSA-compliant encrypted password file. You can use the htpasswd program that comes installed with Apache to create your passwords. Here is how it’s done:

First you need to create the password file. Here the name of the password file should be /etc/squid/squid_passwd, and you need to make sure that it’s universally readable.

[root]# touch /etc/squid/squid_passwd
[root]# chmod o+r /etc/squid/squid_passwd

Then use the htpasswd program to add users to the password file. You can add users at anytime without having to restart Squid. In this case, you add a username called ‘test_user’:

[root]# htpasswd /etc/squid/squid_passwd test_user
New password:
Re-type new password:
Adding password for user test_user

Now you have to locate the ncsa_auth file.

[root]# locate ncsa_auth
/usr/lib/squid/ncsa_auth

Edit squid.conf; specifically, you need to define the authentication program in squid.conf, which is in this case ncsa_auth. Next, create an ACL named ncsa_users with the REQUIRED keyword that forces Squid to use the NCSA auth_param method you defined previously. Finally, create an http_access entry that allows traffic that matches the ncsa_users ACL entry. Here’s a simple user authentication example; the order of the statements are important:

#
# Add this to the auth_param section of squid.conf
#
auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd

#
# Add this to the bottom of the ACL section of squid.conf
#
acl ncsa_users proxy_auth REQUIRED

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow ncsa_users

This will enable the password based authentication and allows access only during business hours. Once again, the order of the statements is important:

#
# Add this to the auth_param section of squid.conf
#
auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd

#
# Add this to the bottom of the ACL section of squid.conf
#
acl ncsa_users proxy_auth REQUIRED
acl business_hours time M T W H F 9:00-17:00

#
# Add this at the top of the http_access section of squid.conf
#
http_access allow ncsa_users business_hours

Remember to restart Squid for the changes to take effect.

Forcing users to use your Squid Server

If you are using access controls on Squid, you may also want to configure your firewall to allow only HTTP Internet access to only the Squid server. This forces your users to browse the Web through the Squid proxy. Also it is possible to limit HTTP Internet access to only the Squid server without having to modify the browser settings on your client PCs. This called a transparent proxy configuration. It is usually achieved by configuring a firewall between the client PCs and the WAN to redirect all HTTP (TCP port 80) traffic to the Squid server on TCP port 3128, which is the Squid server’s default TCP port.

Squid transparent proxy configuration

Your first step will be to modify your squid.conf to create a transparent proxy. The procedure is different depending on your version of Squid. In older versions of Squid ( < 2.6), transparent proxy was achieved through the use of the httpd_accel options which were originally developed for http acceleration. In these cases, the configuration syntax in squid.conf would be as follows:

httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on

Newer versions of Squid simply require you to add the word “transparent” to the default “http_port 3128″ statement. In this example, Squid not only listens on TCP port 3128 for proxy connections, but will also do so in transparent mode.

http_port 3128 transparent

Configuring iptables to support the Squid tansparent proxy

In this example, assuming the Squid server and firewall are in the same server, all HTTP traffic from the home network is redirecting to the firewall itself on the Squid port of 3128 and then only the firewall itself has access the Internet on port 80.

iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128
iptables -A INPUT -j ACCEPT -m state --state NEW,ESTABLISHED,RELATED -i eth1 -p tcp --dport 3128
iptables -A OUTPUT -j ACCEPT -m state --state ESTABLISHED,RELATED -o eth1 -p tcp --sport 80

Note: This example is specific to HTTP traffic. You won’t be able to adapt this example to support HTTPS web browsing on TCP port 443, as that protocol specifically doesn’t allow the insertion of a “man in the middle” server for security purposes. One solution is to add IP masquerading statements for port 443, or any other important traffic, immediately after the code snippet. This will allow non HTTP traffic to access the Internet without being cached by Squid.

Reference:

http://www.squid-cache.org/

Special thanks to Peter Harrison.

Securing your wireless network

Wireless networking products are so ubiquitous and inexpensive that just about anyone can set up a WLAN in a matter of minutes with less than 10,000 rupees ($100) worth of equipment. This widespread use of wireless networks means that there may be dozens of potential network intruders lurking within range of your home or office WLAN.When you connect two computers using a wireless connection, the data is sent via radio waves on a certain channel. Thus anyone with a receiver (could be a wireless card) can analyze the data being sent. This is called “sniffing”.

Most WLAN hardware has gotten easy enough to set up that many users simply plug it in and start using the network without giving much thought to security. If you are running an “open” network, a cracker with a laptop can listen in and analyze everything that you are doing online online – the websites you visit, the emails you send, even the usernames and passwords you exchange with servers. After connecting to your network, he may be able to scan and connect to other machines as well. Sharing your WiFi by keeping your access point “open” is regarded as nice, but there are instances where you want to secure your data.  Here are some of the things you can do to protect your wireless network:

SSID cloaking

Wireless networks identify themselves by a SSID, which can be something like “mynetwork“. Computers with a wireless card whose SSID is set to “mywireles” can connect to each other. Access points send out periodic beacons which are meant to indicate their presence. These beacons also usually broadcast the respective SSID. Thus anyone with a sniffer can find out that there is a network with a “open” SSID and connect to that. A basic form of security is to disable the broadcast SSID. When this is done, the access point doesn’t identify it self when sending out his beacon packets. An intruder who doesn’t know the SSID wont be able to connect to the network. The weakness of this method is that the network’s SSID is sent via other data packets as well. If you listen long enough to the communications between two networks, the SSID can be easily found, making connecting as easy as before.

MAC address filtering

A MAC address is the hardware address of the wireless card. The network uses this to identify where to send data packets. If you have a wireless network with a router and two wireless cards connected to it, you will see two machines connected with two unique MAC addresses. (Here is an example for a MAC address – 00:1C:F0:3A:39:12). Since a MAC address is unique for each network card (like a finger print), another method of security is to ask the wireless router  to accept connections only from certain MAC addresses. Using this method, you could ask the router to only connect machines known to you.

The weakness in this method is that you can set the hardware MAC address of a wireless card  to what ever you wish. If an attacker listens to a wireless network for long enough, he can get a list of connected computers along with their MAC address. Then all he has to do is to wait till one of the computers disconnect from the wireless access point and set his wireless network card’s MAC address to that number and connect to the network.  As far as the access point is concerned, the new connection will be from a known client. This technique is called “MAC address spoofing“.

WEP (Wired Equivalent Privacy)

This is a security method where the computers in a wireless network use a pre-shared security key to encrypt data. Since the data is encrypted before transmission you cannot decrypt WEP enabled network traffic if you don’t have the access key. The problem with WEP is a design limitation – it is inherently insecure at high volumes of traffic. If you have enough data that is transmitted in a WEP encrypted network, you can subject the data obtained o a statistical analysis and guess the security key with near one hundred percent accuracy. Once you have obtained the key, the network is completely decrypted and can be accessed  like an “open” network. Because of these problems, security experts no longer recommend the use of WEP for securing a network. But, if you find that some of your wireless devices only support WEP encryption (this is often the case with non-PC devices like media players, PDAs, and DVRs), avoid the temptation to skip encryption entirely, using WEP is still far superior to having no encryption at all.

WPA (Wireless Protected Access)

Due to the weakness of the WEP system, a stronger security model was needed. The WPA encryption method is much stronger than WEP and is more resistant to attempts at guessing the security key. However one weakness in WPA is the use of weak passwords. An attacker can guess the security key by subjecting captured WPA authentication packets to a dictionary attack. However, WPA is a secure method far superior to WEP if you use a proper password with alternating letters and numbers and no dictionary words.

IpSec (IP Security)

This is the strongest security method available. IpSec is initiated by the computers connected to the network themselves, independent of the medium of transmission (wired or wireless). This method can be used to establish a secure encrypted channel of communication between two computers. The data is authenticated as well, meaning that no outsider is able to insert data packets or generate false packets. The disadvantage of IpSec is that it is difficult to setup without trained, professional help.

Remote administration

Most WLAN routers have the ability to be remotely administered via the Internet. Ideally, you should use this feature only if it lets you define a specific IP address or limited range of addresses that will be able to access the router. Otherwise, almost anyone anywhere could potentially find and access your router. As a rule, unless you absolutely need this capability, it’s best to keep remote administration turned off. (It’s usually turned off by default, but it’s always a good idea to check. ) Although wireless network security has always been problematic, viable solutions are slowly emerging.

Although IpSec is by far the most secure encryption method to use on the network, I also recommend WPA for combining both security and ease of setup.

How BitTorrent works

In 2001, a computer geek named Bram Cohen (25) visited the annual DefCon ‘hacker’ convention in Las Vegas to show off his new toy… BitTorrent. It then debuted at CodeCon 2002 the following year and gained popularity, being featured on an online P2P magazine as well as a few respectable news sites. Fast forward a few years, and a significant percentage of internet traffic belongs to BitTorrent usage, the MPAA and RIAA hate it and Microsoft is at tempting (unsuccessfully) to copy it.

How it works

Let’s take a general example here. Imagine you are in a room with three other people. One of these people (let’s call him A) has a complete jigsaw puzzle with him. The objective is to transfer a copy of the jigsaw to each of the other three people in the room in the most efficient way possible. Initially ‘A’ transfers a portion of the file to a person in the room. Simultaneously, he transfers a different portion to the other, and yet another portion to you. Now we have three different portions of the jigsaw puzzle, which if put together make one hundred percent. While ‘A’ continues to transfer the rest of the file, each person in the room can now transfer their portions to each other as well.

Let’s get a little geeky and use BitTorrent lingo, which you will need to know if you ever decide to venture into torrent territory. The jigsaw here is a file..let’s say a movie (movie.avi). ‘A’, who has the complete file is called a ‘seed‘. The person who doesn’t have the complete file is called a ‘leech‘ or ‘leecher‘. Both these types can be commonly called a ‘peer‘. For example, if someone says “I’m downloading this file off 1 seed and 3 leechers”, that means he’s connected to and downloading from 4 peers. So what happens is, a seed sends a different parts of the file to each leecher. Each leacher in turn transfers parts of the file to other leechers. This way, if a seed disconnects, 100% of the file will still be able to downloaded since there’s a full copy amongst the leechers.

Do it yourself

Now that you know how the protocol works, it’s time foe some hands-on experience. You’ll be happy to know that all software need to delve into the BitTorrent world is free…and free of spyware/malware/adware and other annoying *ware’s.

In the BitTorrent world, all you download off websites are small files called ‘torrents’ (with a .torrent extension) Once you download the torrent onto your computer, you run it with a BitTorrent client. The BitTorrent client will handle the rest for you.

BitTorrent client

There are as many BitTorrent client’s out there as there are budding rock groups in the world. Each client have it’s own streangths and weaknesses. I recommend ‘qbittorrent‘ which I myself use, though there are other really good torrent clients. The official BitTorrent client is available at BitTorrent.com and a simple Google search will link you to many more.

The client is the software that keeps track of all your downloads. It’ll also present you with more information than you’ll know what to do with. Your client can allow you to ‘chat’ to others, prioritize certain files in the torrent, etc. If I just confused you..that’s okay, you probably don’t need those features anyway.

Torrents

After you have downloaded and installed a BitTorrent client, you will need to find yourself a torrent of your choice. These can be downloaded off from the websites. Usually these sites have search facility which allows you to search for the torrents you want. You don’t have to worry about the file size, because torrent files are usually less than 50kb’s in size. (If you are using ‘qbittorrent‘ it has a build in torrent search engine which enables you to download torrents from various torrent sites)

At this point many users new to torrents are surprised at how a 700mb ‘movie.avi‘ downloaded in 5 seconds. It didn’t. What actually downloaded is the TORRENT file. Think of a torrent as a small file telling your BitTorrent client where to find the actual ‘movie.avi‘ file from.

Great! At this point you have the items on my very short ‘what you need’ list. The complicated part is over. Next open the torrent file with your chosen BitTorrent client. That’s it! Now you can leave your computer and indulge in your favorite extra curricular activity while the torrent downloads.

Future of BitTorrent

BitTorrent, Inc. is continuously working on making a more efficient delivery network. BitTorrent DNA^TM is their next step in the evolution of digital content delivery; it combines the extreme efficiency and organic scalability of peer networking with the control and reliability of a traditional content delivery network (CDN). BitTorrent DNA uses one or more existing origin servers or CDNs to seed a managed peer network.

Use of the peer network is tightly controlled by a specialized tracker operated by BitTorrent, Inc. and accessible to BitTorrent DNA customers through a web-based dashboard that provides control and reporting tools.

Thanks to Raveen Wijayatilake and BitTorrent, Inc.

System Administrators’ Code of Ethics

Professionalism

• I will maintain professional conduct in the workplace and will not allow personal feelings or beliefs to cause me to treat people unfairly or unprofessionally.

Personal Integrity

• I will be honest in my professional dealings and forthcoming about my competence and the impact of my mistakes. I will seek assistance from others when required.

• I will avoid conflicts of interest and biases whenever possible. When my advice is sought, if I have a conflict of interest or bias, I will declare it if appropriate, and recuse myself if necessary.

Privacy

• I will access private information on computer systems only when it is necessary in the course of my technical duties. I will maintain and protect the confidentiality of any information to which I may have access, regardless of the method by which I came into knowledge of it.

Laws and Policies

• I will educate myself and others on relevant laws, regulations, and policies regarding the performance of my duties.

Communication

• I will communicate with management, users, and colleagues about computer matters of mutual interest. I will strive to listen to and understand the needs of all parties.

System Integrity

• I will strive to ensure the necessary integrity, reliability, and availability of the systems for which I am responsible.

• I will design and maintain each system in a manner to support the purpose of the system to the organization.

Education

• I will continue to update and enhance my technical knowledge and other work-related skills. I will share my knowledge and experience with others.

Responsibility to Computing Community

• I will cooperate with the larger computing community to maintain the integrity of network and computing resources.

Social Responsibility

• As an informed professional, I will encourage the writing and adoption of relevant policies and laws consistent with these ethical principles.

Ethical Responsibility

• I will strive to build and maintain a safe, healthy, and productive workplace.

• I will do my best to make decisions consistent with the safety, privacy, and well-being of my community and the public, and to disclose promptly factors that might pose unexamined risks or dangers.

• I will accept and offer honest criticism of technical work as appropriate and will credit properly the contributions of others.

• I will lead by example, maintaining a high ethical standard and degree of professionalism in the performance of all my duties. I will support colleagues and co-workers in following this code of ethics.

approved by the SAGE Executive Committee and by the Ethics Working Group.

Rename a list in Mailman

Before read any further, make sure that you have administrator privileges for system-wide operations. Because you need to have access to the Mailman installation to perform the operations which I’m gonna tell you shortly. If you don’t have this access, you need to create a new list and configure like the old one. Hm.. think of a mailing list with a few hundred users and hell a lot of configuration

If you have access to the command line tools, you can simplify the whole process by doing the following after creating the new list:

 [root]# cd <mailman_installation> ; usually at /var/lib/mailman
 [root]# bin/config_list -o file oldlist
 [root]# bin/config_list -i file newlist
 [root]# bin/list_members -o digest_file -p -d -f oldlist
 [root]# bin/list_members -o regular_file -p -r -f oldlist
 [root]# bin/add_members -d digest_file -r regular_file [other options] newlist

Use the –help options on the above commands for more information.

The drawback of the above methods is member information other than name and regular/digest is not transferred and the archives are not transferred. The following procedure will preserve everything. Before doing this, consider whether you want to stop your incoming MTA during the process to prevent possible message loss.

 [root]# mv lists/oldlist lists/newlist
 [root]# mv archives/private/oldlist archives/private/newlist
 [root]# mv archives/private/oldlist.mbox archives/private/newlist.mbox
 [root]# mv archives/private/newlist.mbox/oldlist.mbox archives/private/newlist.mbox/newlist.mbox

Then go to the web admin interface for the ‘newlist’ list and change the real_name from ‘OldList’ to ‘NewList’.

You might be tempted to change the names of any data/heldmsg-oldlist-nnn.pck (or .txt) files. Don’t do this. The old names are in the lists request.pck files, and changing the name will result in the file not being found.

Also, don’t be too concerned about symlinks in the archives/public/ directory. Mailman will automatically create them as needed for the new name. If you leave the old symlinks, they will point to non-existant directories, and trying to visit old public archive URLs will probably give a 403 – forbidden error, so removing them is a good idea, but then visiting the old URLs will still produce a 404 – not found error.

At this point, everything is OK except the ‘more information about this list’ links on some archive pages will still point to the ‘oldlist’ list which doesn’t exist. You can fix those manually or rebuild the archive with:

 [root]# bin/arch --wipe newlist

Finally, if your MTA uses aliases for mailman lists rather than some process that automatically understands list names from the contents of the lists/ directory, you will have to update your aliases. If you do this manually, edit your aliases to change ‘oldlist’ to ‘newlist’ (a total of 2 changes per alias for 10 aliases), and then run ‘newaliases’ or whatever command rebuilds your alias database. It you have Postfix/Mailman integration, just run bin/genaliases.

Depending upon your version of Mailman, you may find that after performing the second type of renaming procedure, above, that when you log into the admin screen for ‘NewList,’ the list name in the first box still shows ‘OldList’ and needs changing to ‘NewList.’ Upon trying to submit that change, you may be presented with:

Error: real_name attribute not changed! It must differ from the list’s name by case only.

In that case, use the following commands to change the real_name:

[root]# bin/withlist -l newname
 Loading list newname (locked)
 The variable `m' is the newname MailList instance
 >>> m.real_name
 'OldName'
 >>> m.real_name = 'NewName'
 >>> m.Save()
 >>>
 Unlocking (but not saving) list: newname
 Finalizing

You type the withlist command, and within withlist, you type the responses to the >>> prompts. On the >>> last line, your entry is control-D (end of file). This procedure was found in the archives of the Mailman-Users mailing list:

http://www.mail-archive.com/mailman-users@python.org/msg50278.html

Enjoy!

Special Thanks to: David Topping and Mark Sapiro

Configure Apache2 with SSL

I recently had a need to setup a private directory on my web server that could only be accessed by a handful of selected people. The content also needed to be encrypted in transit. This article details how I did this on a Debian GNU/Linux system running Apache (2.0.40) server using mod_ssl and OpenSSL (0.9.6b and higher). Here are the goals of this project:

• Require HIGH or MEDIUM level SSL/TLS encryption at the transport (TCP) layer
• Browser must use SSLv3 or TLSv1, not SSLv2
• Require username/password authentication for some subdirectories
• Be a mini-CA (Certificate Authority)
• Use a non-standard port to keep most of the port-scanning riffraff away

The key to this whole system is the SSL/TLS protocol. SSL stands for Secure Sockets Layer, and it was developed by Netscape to enable secure transactions over the Web. It operates between the TCP layer and the HTTP application layer. TLSv1 is the IETF standard implementation, based on SSLv3. TLS stands for Transport Layer Security.

Assumptions

First and foremost, this document assumes that you are using some flavor of Linux, Apache 2.0.x and that you have OpenSSL installed. Other assumptions:

• This will be used over the Internet
• Your DNS configuration is correct (hostname=FQDN, PTR records O.K., etc.)
• Your firewall is setup to allow connections on the chosen https:// port
• You have a second machine with a modern web browser for testing purposes
• In these examples, my FQDN and hostname is: mycompany.com

Step 1: Setup your own CA (Certificate Authority)

In order to run a secure (SSL/TLS encrypted) web server, you have to have a private key and a certificate for the server. For a commercial web site, you will probably want to purchase a certificate signed by a well-known root CA. For Intranet or special-purpose uses like this, you can be your own CA. This is done with the OpenSSL tools.

Here, we will make a private CA key and a private CA X.509 certificate. We will also make a directory for the certs and keys:

[root]# mkdir /root/CA
[root]# chmod 0770 /root/CA
[root]# cd /root/CA

[root]# openssl genrsa -des3 -out my-ca.key 2048
Generating RSA private key, 2048 bit long modulus
.....................................................+++
...................................................+++
e is 65537 (0x10001)
Enter PEM pass phrase:
Verifying password - Enter PEM pass phrase:

[root]# openssl req -new -x509 -days 3650 -key my-ca.key -out my-ca.crt
Using configuration from /usr/share/ssl/openssl.cnf
Enter PEM pass phrase:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:LK
State or Province Name (full name) []:Western
Locality Name (eg, city) []:Colombo
Organization Name (eg, company) [My Company Ltd]:MyCompany.Com
Organizational Unit Name (eg, section) []:Certificate Authority
Common Name (eg, your name or your server's hostname) []:mycompany.com CA
Email Address []:user@mycompany.com

[root]# openssl x509 -in my-ca.crt -text -noout

Notes: The first OpenSSL command makes the key. The second command makes the X.509 certificate with a 10-year lifetime. The third command lets you view the completed certificate. Make sure that you keep the password in a safe place, you will need this every time you sign another certificate! You will probably also want to make backups of the cert and key and lock them in a safe place.

Step 2: Make a key and a certificate for the web server:

Now, we have to make an X.509 certificate and corresponding private key for the web server. Rather than creating a certificate directly, we will create a key and a certificate request, then “sign” the certificate request with the CA key we made in Step 1. You can make keys for multiple web servers this way. One thing to note is that SSL/TLS private keys for web servers need to be either 512 or 1024 bits. Any other key size may be incompatible with certain browsers.

[root]# openssl genrsa -des3 -out server.key 1024
Generating RSA private key, 1024 bit long modulus
....++++++
.++++++
e is 65537 (0x10001)
Enter PEM pass phrase:
Verifying password - Enter PEM pass phrase:

[root]# openssl req -new -key server.key -out server.csr
Using configuration from /usr/share/ssl/openssl.cnf
Enter PEM pass phrase:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:LK
State or Province Name (full name) []:Western
Locality Name (eg, city) []:Colombo
Organization Name (eg, company) [My Company Ltd]:mycompany.Com
Organizational Unit Name (eg, section) []:TechStaff
Common Name (eg, your name or your server's hostname) []:mycompany.com <=== This must be the real FQDN of your server!!!
Email Address []:user@mycompany.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

# openssl x509 -req -in server.csr -out server.crt -sha1 -CA my-ca.crt -CAkey my-ca.key -CAcreateserial -days 3650
Signature ok
subject=/C=LK/ST=Western/L=Colombo/O=mycompany.Com/OU=TechStaff/CN=mycompany.com/Email=user@mycompany.com
Getting CA Private Key
Enter PEM pass phrase:

[root]# openssl x509 -in server.crt -text -noout

Make sure that your server name is the same as the FQDN that your clients will use when connecting to your site. Also, let’s get in the habit of protecting our keys with appropriate permissions:

[root]# chmod 0400 *.key

Now, we need to move the new keys and certs into the proper directories in the /etc/apache2 hierarchy:

[root]# cp server.crt /etc/apache2/ssl.crt
[root]# cp server.key /etc/apache2/ssl.key
[root]# cp my-ca.crt /etc/apache2/ssl.crt

Step 3: Create directories and files for the secure web service

I do not want the secure branch of my webserver directory tree to be part of my “insecure” branch that serves unencrypted files. My normal web root directory is /var/www/ . The document root for the secure web server will be located at /var/www/SSL.

[root]# mkdir /var/www/SSL
[root]# chmod 0775 /var/www/SSL
[root]# cd /var/www/SSL
[root]# mkdir Passneeded

For testing purposes, create a simple HTML file in /var/www/SSL to print “Apache rocks with SSL”

Step 4: Configure the Apache web server

Create a file, let’s say https.mycompany.com in /etc/apache2/sites-enabled/ to define your HTTPS virtualhost and include the following lines.

NameVirtualHost mycompany.com:443
<VirtualHost mycompany.com:443>

DocumentRoot "/var/www/SSL"

# Note that the FQDN and server hostname must go here - clients will not be able to connect, otherwise!
ServerName mars.mycompany.com:443
ServerAdmin webmaster@mycompany.com

#Turning-on the SSL engine
SSLEngine On

# Here, I am allowing only "high" and "medium" security key lengths.
SSLCipherSuite HIGH:MEDIUM

# Here I am allowing SSLv3 and TLSv1, I am NOT allowing the old SSLv2.
SSLProtocol all -SSLv2

#   Server Certificate:
SSLCertificateFile /etc/apache2/ssl.crt/server.crt

#   Server Private Key:
SSLCertificateKeyFile /etc/apache2/ssl.key/server.key

#   Server Certificate Chain:
SSLCertificateChainFile /etc/apache2/ssl.crt/my-ca.crt

#   Certificate Authority (CA):
SSLCACertificateFile /etc/apache2/ssl.crt/my-ca.crt

# This is needed so that you can use auto-indexing for some directories in the
# /var/www/SSL directory branch.  This can be handy if you would like to have
# a list of sensitive files for people to download.
<Directory "/var/www/SSL">
        Options Indexes
        AllowOverride None
        Allow from from all
        Order allow,deny
</Directory>

Also you have to tell Apache, for all HTTPS requests use the port 443. For that append the following lines to /etc/apache2/ports.conf

<IfModule mod_ssl.c>
    Listen "443"
</IfModule>

Step 5: Start the web server and test

Run the following commands to start the the Apache web server:

[root]# /etc/init.d/apache2 start
Starting web server: apache2.
Server mycompany.com:443 (RSA)
Enter pass phrase:

Note that you will have to enter the password for your server key in order to start the server. You will also have to do this during boot if you have httpd configured to start automatically.

Make sure that the web server is now listening on the SSL/TLS port, TCP port 443:

[root]# netstat -tna
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN

In order to test that your SSL/TLS web server is running, you will now need to connect to it with a browser. The URL you use should be https://mycompany.com. You will probably get a warning prompt about the Certificate Authority (CA) being unknown. You can view the certificate properties, which will look familiar because you created the cert yourself. You can save the cert in your browser, or import the my-ca.crt file into your browser as a new CA. How you do this will depend on which browser you are using.

Step 6: Require simple username/password auth for one of the directories:

We want to require a valid username and password for the /var/www/SSL/Passneeded directory. The username and password will be encrypted in transit as part of the TCP stream. We will need to setup the access control directives, as well as use the htpasswd command to add the username/password pairs.

[root]# htpasswd -c -m /etc/apache2/.htpasswd test_user1
New password:
Re-type new password:
Adding password for user test_user1
[root]# htpasswd -m /etc/apache2/.htpasswd test_user2
New password:
Re-type new password:
Adding password for user test_user2

[root]# chown apache.root /etc/apache2/.htpasswd
[root]# chmod 0460 /etc/apache2/.htpasswd

Now, we need to tell Apache to require a username/password to access the Passneeded directory. Here is what we will add to /etc/apache2/sites-enabled/https.mycompany.com file:

<Directory "/var/www/SSL/Passneeded">
	AuthType Basic
	AuthName "Username and Password Required"
	AuthUserFile /etc/apache2/.htpasswd
	Require valid-user
</Directory>

Now, restart the webserver with /etc/init.d/apache2 restart. When you try to access the Passneeded directory from a web browser, you should be prompted for a username and password. If you enter incorrect information, you should be denied access.

Step 7: Change the TCP port that Apache SSL/TLS listens on:

Since this is a private, special-purpose secure web server, you may want to change the TCP port from 443 to something else. This will make it just a little more difficult for crackers to locate via automated network scans. For this excercise, we will change the port to TCP 444 by editing the ports.conf configuration file and /etc/apache2/sites-enabled/https.mycompany.com. Make the following changes to the ports.conf:

<IfModule mod_ssl.c>
    Listen "444"
</IfModule>

And make the following changes in /etc/apache2/sites-enabled/https.mycompany.com:

NameVirtualHost mycompany.com:444
<VirtualHost mycompany.com:444>

Now, restart Apache and look at the listening ports:

[root]# /etc/init.d/apache2 restart

[root]# netstat -tna
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:444             0.0.0.0:*               LISTEN

Now, you should be able to connect to the server with this URL:

https://mycompany.com:444

Web Server Key Password:

You have probably noticed by now that every time you restart Apache or boot your server, you are forced to enter the password for the server key. This is a security measure, but it can be inconvenient. If you would like to make an insecure server key that will allow Apache to start automatically at boot time, then there is a way to do this. The choice is yours…

Here is how you do it:

[root]# cd /etc/apache2/ssl.key
[root]# cp server.key server.key.orig

[root]# openssl rsa -in server.key.orig -out server.key

[root]# chmod 0400 server*

Now, you should be able to restart Apache or boot your server without having to input the password. This may also be a very good time to copy all the keys and certificates that you made to floppy or CD. You can imagine what a pain it would be if you lost all of your keys and certs due to a disk failure. You may even want to make paper copies of the PEM encoded certificates and keys, which use ASCII text. Lock them in a secure place, along with any passwords.

Conclusion/Final Comments

As you can see, setting up a secure web server for some specific function is not that difficult. All the tools are included with a standard GNU/Linux distribution. OpenSSL is a fantastic Open Source toolkit that can be used in a number of applications. For example, you can use it to run files through different hashing functions, handle S/MIME encrypted mail, or encrypt & decrypt files.

In order to use Apache as a high-volume e-commerce server with SSL/TLS, you will probably need to do more configuration and hardware tuning. You may need to buy and configure a hardware crypto accelerator card. You will almost certainly want to purchase a “real” server certificate signed by Entrust, Thawte, or one of the other root-level CAs.

In any event, you now have a good feel for all the pieces, parts, and protocols that make it work!

Resources

• Apache 2.0 Documentation
• mod_ssl home page
• OpenSSL home page
• IETF TLS Charter
• The following man pages:
  • man openssl
  • man genrsa
  • man req
  • man x509
  • man pkcs12
  • man htpasswd