Procmail mail delivery agent (MDA)

This chapter introduces the Procmail mail delivery agent (MDA). The MDA is called by the mail transport agent (or MTA, such as Sendmail) as the final step in the mail delivery process. A call to the MDA delivers email to each end user's home directory for Maildir-style mailboxes or /var/spool/mail for other MTAs such as Sendmail. The MDA can also perform other checks such as anti-spam, anti-virus, filtering, and many other functions too numerous to list here.

Under the default Sendmail MTA installation, the default MDA is mail.local. mail.local is very limited in its capabilities. For example, many mail.local implementations in use today cannot perform a disk usage check (quota) prior to delivery or filter messages based on From:, To:, headers etc. This lack of functionality in mail.local led to Procmail's development and widespread use.

Procmail can be configured to perform actions based on patterns matched in header items as well as the body of a mail message. Actions include filing, forwarding, and further processing. Many anti-spam mechanisms covered in this book utilize Procmail from either the delivery agent or the Unix mail client side. It is usually preferable to invoke Procmail at MDA time, rather than for each user individually. In this manner, Procmail setup is performed once, rather than having to go into each user's home directory to set it up.

Typically, Procmail is used when the desired anti-spam program requires it or when no other mechanisms are available. As with many Unix utilities, Procmail is lean and mean, making it difficult to understand and use. For every spam-fighting utility covered in the book that requires the use of Procmail, a canned Procmail recipe is provided. For newcomers to Procmail, the availability of preexisting recipes makes learning the tool much easier.

Installation

Procmail is included by default in most Linux distributions. However, many commercial Unix implementations do not include it, requiring you to install it if needed. We cover Procmail version 3.22 here, but you should be able to use any recent version of the tool with the recipes provided in this book.

The installation of Procmail is required only on certain versions of Unix. Most distributions of Linux contain Procmail. However, Solaris doesn't contain Procmail, so it must be built (or the binary package must be downloaded).

To install Procmail, perform the following steps. First, download the current sources by transferring them from ftp://ftp.procmail.net and downloading procmail-3.22.tar.gz into a directory such as /usr/local/src. Un-tar the package by executing the following commands:

bash% gunzip procmail-3.22.tar.gz
bash% tar xf procmail-3.22.tar
bash% cd procmail-3.22

After changing directories into the Procmail source directory, you will need to edit the Makefile variable BASENAME to install the package to the base location, /usr/local. (All of the examples assume Procmail is installed here.) A few other options can be specified in the config.h file; check the INSTALL for additional installation options if desired.

After the Makefile and config.h settings have been changed appropriately as outlined previously, compile and install Procmail as follows:

bash% make
bash% sudo su
# make install

Once installed, procmail must be hooked into the mail transfer agent (MTA). These steps are the subjects of the following sections.

Invoking procmail

There are two different ways procmail can be invoked or called. The most efficient way is for the MTA to invoke it directly. This is more difficult to set up, but it uses fewer system resources. The MTA invocation topic is covered in the section titled "Delivery Agent Invocation" and requires MTA setup. The second way is to call it via .forward (or in qmail's case, .qmail) files in each user's home directory. Although easier to set up, the per-user forward method uses additional system resources.

There are two different ways Procmail recipes can be called, either system-wide in /etc/procmailrc or per-user (by default, a file called .procmailrc located in each user's home directory). The system-wide file is called first, and then the per-user recipes are called second.

`.forward` File

Both Postfix and Sendmail support .forward files, and procmail can be invoked from them. Create a file called .forward in your home directory and place the following line in it:

| /usr/local/bin/procmail

This will tell Sendmail or Postfix to process all messages through procmail for this user.

`.qmail` File

Unlike Sendmail and Postfix, it is not easy to use Procmail as a delivery agent in a qmail installation. However, Procmail can be used by the Unix user as a filtering or processing agent. procmail can be invoked by the user by placing appropriate lines in each user's .qmail file. For example, the following line in a user's .qmail-default file will cause messages for that user to be processed by procmail:

| /usr/local/bin/procmail

This line in the .qmail-default file will cause qmail to process messages through Procmail by default.

Delivery Agent Invocation

When built from source as outlined previously, the procmail command itself is installed in /usr/local/bin. Most Linux distributions install the procmail binary in /usr/bin/procmail. In the next sections, we show how to invoke procmail under the covered MTAs, which include Sendmail, Postfix, and qmail.

Sendmail

Sendmail can be configured to use procmail as the delivery agent. Procmail is enabled by a FEATURE setting in the Sendmail configuration file, such as $SRC/cf/sendmail.mc.

FEATURE(`local_procmail')

If procmail is not in the standard place, you have two choices. Perhaps the simplest way to tell Sendmail where to find procmail is use a second argument to FEATURE to tell it where it is:

FEATURE(`local_procmail', `/usr/bin/procmail')

Alternatively, you can define the macro PROCMAIL_MAILER_PATH before the FEATURE statement:

define(`PROCMAIL_MAILER_PATH', `/usr/bin/procmail')
FEATURE(`local_procmail')

After the appropriate changes are made to the sendmail.mc file, remake the sendmail.cf file from the m4 sources and restart Sendmail like this:

# m4 ../m4/cf.m4 sendmail.mc > sendmail.cf
# cp sendmail.cf /etc/mail
# /etc/init.d/sendmail restart

After restarting Sendmail, Procmail recipes can be tested and deployed.

Postfix

Postfix can be configured to use procmail as the delivery agent. Enable it inside the main Postfix configuration file, typically /etc/postfix/main.cf, like this:

mailbox_command = /usr/local/bin/procmail

After the setting is changed, the Postfix system is restarted to enable changes:

# postfix reload

When Postfix is reloaded, you are ready to install procmail filters as necessary.

qmail

Unfortunately, qmail doesn't hand off messages to delivery in a standard way because the "envelope From:" line is stripped. It also doesn't handle standard exit codes. Instead, qmail expects a certain set of exit codes to indicate either a soft or hard failure. A soft failure indicates that delivery should be attempted again in the future. A hard failure indicates that delivery attempts should be abandoned. As a result, it is not possible to easily use Procmail as a delivery agent. In order to integrate qmail with Procmail, you are stuck doing it on a per-user basis and must invoke procmail via .qmail files, as outlined previously in the ".qmail File" section.

Configuration

The configuration of procmail is independent of the MTA used, so everything in the rest of this chapter applies to Sendmail, Postfix, and qmail (if using .qmail files for each user). By default, Procmail can use a common configuration file, /etc/procmailrc, as well as a user-supplied one, ~/.procmailrc. The file is made up of variable assignments and recipes. Recipes consist of patterns or conditions that are matched against mail header items and actions to take in the event of a match.

There are two kinds of recipes: delivering and nondelivering. Delivering recipes write the mail message to a file, pipe it to a program, or forward it in the event of a match, and further processing stops. Nondelivering recipes continue processing the email message after performing the action associated with a particular successful match.

A nondelivering recipe would be one recipe in a chain of recipes. They can be called in series and processed that way. Delivering recipes would be called at the end of a string of recipes.

Delivering recipes can be configured to act as nondelivering with the c flag. This causes a copy of the mail message to be delivered while processing continues as in a non-delivering recipe. Using this option is useful for sending a message to a second user, to log the message before processing or similar sorts of functions.

Recipes begin with :0 followed by optional flags. Zero or more conditions qualify as a match and indicate that one action is to be performed.

Option Flags

Here you will find some commonly used procmail options. Additional flags are documented on the procmailrc man page.

c A copy of the message is used for continued processing following this delivering recipe
e This recipe only executes if the previous one failed, indicated by a non-zero exit status
f The pipe in the action of this recipe is to be treated as a filter whose output will be used for further processing
w procmail waits for the filter or program to finish and checks its error status before proceeding

Conditions

The conditions are expressed as message header items and extended regular expression similar to egrep. Additional special conditions include

^TO_ Matches destination specifications including To:, Cc:, Bcc:, and Apparently-to: containing a specific address
^TO Matches destination specifications including To:, Cc:, Bcc:, and Apparently-to: containing a specific word

Actions

An action is a path to the mailbox (or mbox) formatted folder in which the message is to be filed. If the path ends with a / character, then the folder is assumed to be a maildir format folder.

If the action is prefixed with a special character, it is treated differently:

! The action is treated as a mail address to which the message will be forwarded
| The action is treated as a program to which the message will be piped
{ The action is treated as a nested block of further Procmail recipes

Examples

This assignment identifies $HOME/Maildir as the root directory that will be used as the starting point for paths not beginning with /:

MAILDIR=$HOME/Maildir

This identifies a file in which Procmail's actions will be recorded. It grows without bound, so be sure to rotate it appropriately:

LOGFILE=$MAILDIR/procmail.log

A recipe with no condition always matches. This example filters all messages through the program called /usr/local/bin/bogofilter, checking the exit status:

:0 fw
| /usr/local/bin/bogofilter

In this recipe, subjects with the word SPAM are placed in an mbox-formatted file called spam:

:0
* ^Subject:.*SPAM
spam

Mail originally sent to This e-mail address is being protected from spambots. You need JavaScript enabled to view it is filed in a maildir-style folder (individual mail files located in the user's mail folder) named $MAILDIR/SecurityJobs:

:0
* ^TO_.*
 This e-mail address is being protected from spambots. You need JavaScript enabled to view it
 
SecurityJobs/

A copy of mail sent to This e-mail address is being protected from spambots. You need JavaScript enabled to view it is forwarded to This e-mail address is being protected from spambots. You need JavaScript enabled to view it:

:0 c
* ^TO_.*
 This e-mail address is being protected from spambots. You need JavaScript enabled to view it
 
! 
 This e-mail address is being protected from spambots. You need JavaScript enabled to view it

Blacklisting and Filtering Example

In this example, we want to filter messages with certain attributes to a folder called Spam (see Figure 2.1). We receive a lot of spam with subject lines containing ADV, along with messages that may be filtered by another anti-spam mechanism. For example, some anti-spam software places a word or phrase in the subject line such as Possible UCE. Figure 9.18 contains a list of subject line modifications made by software programs covered in this book.

Figure 2.1. Blacklist/subject line filtering script.

[View full width]

MAILDIR=$HOME/Maildir
DEFAULT=$MAILDIR/
ORGMAIL=$MAILDIR/
LOGFILE=$MAILDIR/procmail.log
# file emails with subjects matching undesired regexes in the Spam folder
:0
* ? formail -xSubject: | egrep -i -f spamSubjects > /dev/null 2>&1 .Spam/
# file emails with from addresses matching undesired domains in the Spam folder
:0
* ? formail -xFrom -xFrom: -xReply-To: -xReturn-Path: -xSender: -xTo: | egrep -i -f spammers > /dev/null 2>&1
.Spam/

In addition, we receive messages from people we never want to see, so we want to implement a blacklist based upon this list. A single Procmail recipe can be written to automatically file these messages for you, without any effort on your part.

The Procmail recipe invokes the procmail program called formail. The two lines that begin with formail are the basis for this Procmail recipe. The options used in the example echo out the contents of the header specified on the formail command line. The header contents are piped through egrep, and the return value of egrep (success or fail depending on whether a match was found) is used to conditionally trigger the recipe. That's the meaning of the ? in front of the command line. The first formail matches any subject that is listed in the spamSubject file, located in the user's home directory. If matched, the message is filed into the Spam folder. The second formail matches any header listed on the line with the addresses listed in the spammers file. Any message matched ends up in the user's Spam folder. The headers that are checked for addresses are as follows:

From
From:
Reply-To:
Return-Path:
Sender:
To:

From (without the trailing colon) is the envelope from, which always exists. From: (with the colon) is not always specified by the sending SMTP server and therefore may or may not exist. Spammers often omit the From: line, but they cannot omit the envelope From, however.

We could put the headers for the other anti-spam utilities we use into the spamSubject file, and all spam messages would end up in our Spam folder.

Conclusion

Procmail is a mail delivery agent that is used by a number of applications in this book to filter messages according to various attributes. The utility comes with most distributions of Linux, and it is easy to install and configure for use with Sendmail, Postfix, and qmail. procmail can be called directly by the mail delivery agent for Postfix and Sendmail or indirectly by .forward files on a per-user basis. Under qmail, procmail must be invoked via .qmail files because qmail has no direct integration with Procmail (besides per-user .qmail files).

Procmail recipes are terse in nature, which also makes them somewhat cryptic and difficult to grasp. Recipes can live in /etc/procmailrc for all users or .procmailrc in each user's directory. All spam-fighting utilities in this book that utilize Procmail have cookbook recipes that can be used to filter spam messages caught by those spam-identifying programs into your spam folder. This makes learning and implementing Procmail-based solutions much easier.

www.Tutorialsforu.info