Maildrop tips and tricks

Last update: 10/11/1998.

	The include statement
	The `$LINES` and `$SIZE` environment
	Partial matches using the ! operator
	Short-circuit evaluation using logical operators
	The `foreach` statement
	The `$HOME/.tmp` directory

Here's a list of tips and tricks to write efficient mail filters using maildrop. Maildrop - just like any programming language - can do certain things better than others.

The include statement

When maildrop is started, it reads $HOME/.mailfilter, or some other file containing mail filtering instructions. Before doing anything else, this entire file is read and semi-compiled. Any grammatical or syntax errors will result in maildrop terminating with a temporary error code, without doing anything with the message.

The entire file must be 'semi-compiled". The 'semi-compilation' process consists of building an logical representation of the filtering statements in memory.

If you have a long and complicated set of instructions, it will be semi-compiled whether or not it will actually be executed. For example, consider the following code:

if ( /^Subject: rosebud/ )
{
    ...
    some
    really
    long
    set
    of
    instructions
    ...
}

If the contents of the "if" construct are large, maildrop may spend considerable amount of time and memory semi-compiling filtering instructions that may never be used. Consider putting all these instructions into a separate file, and using the include statement to execute them.

The include statement is processed only when it is actually executed by maildrop. That carries certain side effects. One feature of maildrop is the semi-compilation which weeds out errors in the filtering recipe file, holding all mail and not doing anything until the errors are fixed. The include statement is not processed until it is actually executed. If there are errors in the file referenced by the include statement, maildrop will not know about them until it executes the include statement. When maildrop detects the error, it will terminate with a temporary error code, holding all mail. However, any filtering instructions before the include statement would've already been executed.

The `$LINES` and `$SIZE` environment variables.

These variables are initialized to the number of lines in the message, and the size of the message in bytes. They may not be exact, due to the presence of the From_ line, or -A arguments on the command line, but they will always be a reasonable approximation.

Scanning headers for patterns will take the same amount of time whether the body of the message has 100 or 1,000 lines. However, the "b" option causes the message body to be searched. If someone sends you a 5 megabyte binary file, each pattern matched against the message body will take a significant time to complete.

Whenever possible, the first thing you should do is check both variables to see if the message is excessively large. If so, divert the message to a separate mailbox, or reject it.

Weighted scoring is especially sensitive to the size of the message being delivered. Normally, as soon as a pattern match is found, maildrop terminates the pattern scan. If weighted scoring is used then the pattern scan is immediately restarted, until the end of the message is reached.

A fixed overhead is required to start a pattern scan going, so that weighted scoring that matches a very short pattern, like /[:lower:]/:Dbw,1,1 will take a significant amount of CPU time to complete. Maildrop is simply not optimized for these kinds of situations.

Nevertheless, on a Pentium 200, /[:lower:]/:Dbw,1,1 perform reasonably well for messages less than 60K long.

Here's an example of using $LINES and $SIZE to divert large messages to a separate mailbox:

 
    if ($LINES > 1000 || $SIZE > 60000)
    {
        to "mail/IN.large"
    }

    .
    .
    .

Partial matches using the ! operator

Maildrop implements the ! operator in patterns by breaking up the pattern into two separate patterns, which are compiled and executed separately.

Whenever the first pattern is succesfully matched, maildrop runs the second pattern. If the second pattern does not match, maildrop resumes matching the first pattern until it matches again.

Since starting a pattern match involves some fixed overhead, minimizing the number of times the first pattern will match will reduce the amount of time it takes to match the entire pattern.

For example: you're trying to extract an IP address from the first Received header. Here's your first attempt at doing so:

 
    /^Received:.*![0-9\.]+/
    IP=$MATCH2

This will not work at all. From the manual page for maildropfilter:

 
       When  there  is  more  than  one  way  to  match a string,
       maildrop favors matching as much  as  possible  initially.

So this fragment will set IP to the last digit, or period found in the first Received: header, because the first half of the pattern will match as much as possible.

Here's your second attempt, then:

 
    /^Received:.*[^0-9\.]![0-9\.]+/
    IP=$MATCH2

This will work, except the first half of the pattern will end up matching every character in the Received: header that's not a digit or a period. Maildrop will then stop and try to run the second half of the pattern.

Your mail server will probably put a left bracket before the IP address of the relay. Noting that, the following code picks up the IP address as efficiently as possible:

 
    /^Received:.*\[![0-9\.]+/
    IP=$MATCH2

Short-circuit evaluation using logical operators

The logical operators || and && work in maildrop just like they work in C. Consider the following expression:

 
    if (/ ... some pattern ... / || / ... some other pattern ... /)
    {
        .
        .
        .
    }

If the first pattern is found, maildrop will not execute the second pattern, since the logical expression will be true no matter what. You can use that to your advantage. If you have two patterns, and one of them takes more time to process, put the other first. Even if both patterns are relatively quick, if you expect one of them to be found almost every time, put that one first so that it will not be necessary to run the second pattern most of the time.

The same concept works for the && operator:

 
    if (/ ... some pattern ... / && / ... some other pattern ... /)
    {
        .
        .
        .
    }

If the first pattern is not found, maildrop will not execute the second pattern, since the logical expression will be false no matter what.

The `foreach` statement

The foreach statement is implemented as follows. Maildrop will execute the regular expression, and compile a list of all the patterns in the message matched by the regular expression. Afterwards, maildrop will execute the foreach statement, once for each matched pattern.

It is important to note that maildrop will create a list in memory containing every pattern matched by the regular expression. if the regular expression is found many times in the message, this will require a lot of memory. For example, the following is a Very Bad Thing[tm]:

 
    foreach /[:alpha:]/
    {
        .
        .
        .
    }

Maildrop does not include any built-in limits on resource usage, except for the watchdog timer. Therefore, as a system administrator, it is your responsibility to limit resources used by the maildrop process.

This implementation of the foreach statement is the one that yields the least number of surprises. Observe that the contents of the foreach construct can modify the message using the xfilter statement. Furthermore, the regular expression may include variable substitution, and those variables could be modified by the foreach statement. If the foreach statement were to be executed "on the fly", these factors would yield unpredictable - and rather messy - results.

The `$HOME/.tmp` directory

Under certain conditions, maildrop will need to use a temporary file. Temporary files are created in the $HOME/.tmp directory. Temporary files are used to store large messages (instead of storing them in memory). Temporary files may also be used by the xfilter command, or in other situations.

Maildrop will automatically delete all temporary files before terminating. However, if the maildrop process is killed, or terminates abnormally, a temporary file may remain and take up disk space. To have those leftover files automatically cleaned up, put the following code in /etc/profile, or some other file that all users run after logging in:

 
    find $HOME/.tmp -depth -mtime +7 \( \( \
        -type d -exec rmdir {} \; \) -o -exec rm -f {} \; \) 2>/dev/null