How my email setup works

Email is pervasive, and yet I routinely find people who have done an atrocious job of setting up mail software. I am no mail guru, I'm just a demanding user. I thought it'd be interesting for others to see how I went about my stuff.

My goals

Therefore, what is not acceptable

Once I stated these goals, I straight away knew that a few possible solutions were out of the window:

How mail comes into my notebook

I use fetchmail. In fetchmailrc, I have specified the multiple hosts which supply email. fetchmail runs as a daemon, constantly picking up email. Using procmail, mail is delivered into $HOME/Mail in "maildir" format, where a mail folder is spread across a large number of files, a few for each message.

fetchmail runs as a daemon, checking every minute for new mail. If a host is inaccessible, it goes away and tries again later. This is fully compatible with intermittently available connectivity and nomadic usage. I use fetchmail-ssl, so passwords do not flow in the clear.

There are many useful documents on the net on configuring fetchmail and procmail, so I will not go over that here. My /etc/fetchmailrc is:

poll 66.216.80.49 with proto pop3
      username ajayshah password pw1
      is ajayshah here
      with mda "procmail -d %T"
      ssl

poll finance.nic.in with proto pop3
      username ajayshah password pw2
      is ajayshah here
      with mda "procmail -d %T"

I have found that putting two stanzas in the /etc/fetchmailrc is a bit of a problem because a lot of time is lost in the timeout when I'm not able to reach one or the other.

How mail is delivered out of my notebook

Local mail applications (whatever they be) deliver email locally. I have a mail transport agent (MTA) running locally. I use exim as the MTA. It is taught who my "smarthost" is, and delivers mail to the smarthost.

Once again, exim runs as a daemon. It keeps trying to find the smarthost, and when the smarthost is found, mail is delivered.

There are many useful documents on the net on configuring exim, so I will not go over that here. My smarthost uses SMTP authentication, so that a name and password is required for each interaction, and exim knows how to do this.

Mail user application -- mutt

After long exploration, I settled on mutt as my mail user application (MUA).

You may find my .muttrc useful.

Prettyprinting email - method 1 (muttprint)

muttprint is a good package which hooks into mutt and captures the processing of the key 'p'. It is enabled in my .muttrc (which is available above). So in the menu, I say 'p' and printing is done. In my case, I have configured my .muttprintrc to generate a file /tmp/printedmail.ps for me when I say 'p'.

Prettyprinting email - method 2 (a2ps)

Save the file and say --

  $ a2ps -1 -g myfile -o myfile.ps

and you get a nice email hardcopy. The command here assumes you wrote the email from mutt into `myfile', and it generates `myfile.ps' that is ready to print. a2ps is a Debian package.

How to make `myfile'? While looking at an email in mutt, say 'v'. This gives you a menu. Put the cursor on the 1st line, and say 's' to save, and give him the name of file that you want to use.

Running a web browser with URLs in incoming mail

mutt displays mail in plain text. In that case, how might one easily fire a web browser on URLs contained in an incoming email? The answer is to run mutt inside gnome-terminal. When text is displayed inside gnome-terminal, if a URL is seen, then gnome-terminal shows this as underlined when the mouse cursor is over the URL. Right button brings up a menu where the last option is to run a web browser with this URL. It works nicely.

In my .xsession file, I have --

gnome-terminal login -e mutt &

There is an alternative -- mutt can be taught to run urlview when you press a hotkey Ctrl-B. This program extracts all URLs in the incoming email, and gives it to you as a menu. When you hit Enter, it runs a browser of your choice. This is nice and works well for simple emails where there are just one or two URLs. But if you get a newsletter email where there are a dozen URLs, it gets confusing to see the URL without context. So I like the gnome-terminal solution better.

Incoming HTML email

You need this line in the $HOME/.mailcap file -- 
     text/html; lynx -force_html %s; needsterminal;

Commandline, attachments

When the mail user application is command-line friendly, there are all sorts of lovely opportunities to marry the Unix commandline with mail-related tasks.

You can say --

  $ mutt -a file1 -a file2 person@fqdn

In this case, mutt attaches file1 and file2, and starts you into the editor to compose an email to this person. This is faster because file completion can be used to get file1 and file2 onto the commandline, whereas if you did attachments inside mutt, you'd have to do them through a 'file manager' interface, which is much slower.

You can go the whole hog with --

  $ echo "See attached files paper1.pdf and paper2.pdf" \
    | mutt -s "The papers I'd promised" -a paper1.pdf -a paper2.pdf person@fqdn

which is a complete, standalone command that sends off two papers to the person with a few lines of a "Subject". It's incredibly efficient and convenient, when compared with the waste of time involved in giving attachments to mail programs.

I have a small program manyattach which takes a list of files and generates a string "-a file1 -a file2 -a file3", etc. It is a simple shell script --

  #!/bin/sh

  s="";
  for f in $* ; do
    s="$s -a $f";
  done
  echo $s

With this in place, I can do wonderful things like --

  $ cat /dev/null | mutt `manyattach *.jpg` -s "Pictures from last night" person@fqdn

Ooops - that'd give a huge single email! If that's a bottleneck, you can do --

  $ for f in *jpg ; do
      cat /dev/null | mutt -a $f -s "Pictures from last night ($f)" person@fqdn
    done

Ooops - even this clogs your line since a dozen exims contend for the line at once. If you do this when offline, then exim has no concept of exponential backoff, so some hours later, suddenly your line goes dead when you do get on and all those exims retry. So instead I say -- 

  $ for f in *jpg ; do
      cat /dev/null | mutt -a $f -s "Pictures from last night ($f)" person@fqdn
      sleep 120
    done

so that each of these emails get spaced out by two minutes.

What is my point? The Unix command line, coupled with mutt, is a lovely way to get a lot of efficiency on normal email activities.

Integration with palm pilot

mutt can be taught to run an external program when the user attempts file completion for email addresses (Ctrl-T). In my .muttrc I have the line

set query_command="lbdbq %s"

which tells him that when the user says xxx followed by Ctrl-T, mutt is to run a program lbdbq with the argument xxx.

lbdbq is part of the package lbdb. It can be taught to query the palm pilot AddressDB.pdb file in order to access the required information. For example, if I say:

  $ lbdbq Subra
  lbdbq: 2 matches
  subra@anderson.ucla.edu Avanidhar Subrahmanyam  (Palm)
  msubrahm@stern.nyu.edu  Marti Subrahmanyam      (Palm)

You may find my .lbdbrc file useful. In particular, I have enabled the method m_inmail as a way to harvest email addresses from all incoming mail. Read on.

Automated incoming mail processing using procmail

My ~/.procmailrc reads as follows --

:0:
* ^From.*ajayshah
/home/ajayshah/Mail/myarchive/

:0hc
| lbdb-fetchaddr -a

:0
/home/ajayshah/Mail/Inbox/

There are three pattern-actions here:

  1. The first pattern looks for any mail from me, and puts it into myarchive. So whenever I want to keep a copy of mail I'm writing, I cc or bcc it to ajayshah, and it gets tucked away.
  2. The second pattern feeds every incoming email to lbdb-fetchaddr(1), who extracts all email addresses and keeps them in ~/.lbdb/m_inmail.list. Hence, every email address of everyone who ever sent me mail becomes available to me when doing file completion using mutt.
  3. The third pattern delivers all incoming mail into Mail/Inbox.

Backup-friendly mailbox

I have real problems with backup infrastructure. We don't have dedicated backup facilities, so we hack up homebrewn backups by running rsync to mirror the file system. Sometimes this is done over slow links.

If one uses a file format where a single large file is used for an entire mail folder, then every time the file changes, a huge file has to move on the network when rsync clones it. I find maildir format is a real godsend, because each piece of email is kept in separate files, and that generates very little network traffic when backing up.

What I do not like about my mail setup

Multiple smarthosts
With nomadic usage, it's particularly important to be able to intelligently choose between multiple alternative smarthosts.
I have not yet figured out how to make exim deal with multiple smarthosts, and try one after another until he finds one which will do the job for him. Currently, I'm nomadic, but I am forced to have only one smarthost. Wherever I might be, exim tries to reach this same smarthost. It'd be nice if we could give the MTA a table of smarthosts. A naive implementation of this is already in exim. But this feature actually requires work:
  1. Instead of trying one smarthost after another in linear order, waiting for timeouts, the MTA should measure the distance to all the smarthosts and only use the closest one.
  2. Sometimes, we need to modify the outgoing email address to be compatible with a certain smarthost. That intelligence will have to be in the MTA.
File completion for email addresses inside emacs
Ctrl-T is nice, but that's at the mutt prompts. I would like it if while I was inside emacs editing email, Ctrl-T would do file completion of email addresses.
Email appears as attachments
Some DOS people complain that my email to them comes as an attachment and not as the main body of email. There isn't anything to be done about this; they are using obsolete and broken software. The mutt FAQ points out that PGP/MIME is the only way to use PGP with email that is actually specified (RFC 2015), and this has been around since 1996, so there is no excuse to not use it.
Need for file completion when saving files
When saving files, mutt is nice in having file completion in part. That is, if I say "W[tab]", it will complete this with the only directory I have which starts from W. But if two or more directories exist, starting from W, mutt does nothing. Saying "tab tab" does not help, he just sits there. In short: completion in mutt is partial, and it needs to be done fully.
Need access to mailbox while writing email!
The most irritating thing about mutt is that mail composition is single-threaded. When you say "m" to compose a new mail, a new emacs pops up. While the emacs is running, the mail program is out of bounds. I find this much inferior to netscape's model, where punching "New message" gives you a new thread and a new window, and all other email access continues to be available.
As a saving grace, mutt does allow you to run multiple incovations of mutt at the same time. So when I get stuck, I just fire up a new xterm, and run one more mutt inside it. Now I have an emacs (owing to the old mutt), and a new mutt using which I can look at other email messages which are useful while composing the new one.
Example -- Just now, I wanted to compose an email to a few people which was to be done by cutting and pasting pieces from five recent emails. Because mutt is not multithreaded, I made the mistake of saying "emacs &" and starting to write inside it. I went over each of the messages in the thread, cutting out pieces, and creating a new mail in this buffer. I must have been thinking that when I was done, I would say "m" to mutt, and cut-and-paste from my emacs. By mistake, I said C-x C-c and emacs was stupid enough to exit without saving. I lost 20 minutes of work.
I think an MUA must allow you to read any email while composing a new email, and mutt is lacking on this score.

Switching smarthosts dynamically

I got email from Bernard Blackham where he shows a solution to my problem :-).

From: Bernard Blackham 
To: ajayshah@igidr.ac.in
Date: Tue, 16 Apr 2002 15:15:59 +0800
Subject: your mail setup webpage

Hi Ajay,

I was looking around at making exim use multiple smarthosts on my
machine and I stumbled across your page on your email setup (and now
considering using Maildirs). But I've some-what solved the problem
of multiple smarthosts. Perhaps this would appeal to you? Basically
I have:
  a) package called laptop-net installed which auto detects the
     network you are connected to (works beautifully with both wired
     and wireless!)
  b) tell it update a file /etc/smarthost and enter the name of the
     current mail server. This is easily done with laptop-net
  c) alter exim's configuration file to read:
    smarthost:
       driver = domainlist
       transport = remote_smtp
       route_list = * ${lookup{smart}lsearch{/etc/smarthost}{$value}} byname

    (this was pulled from http://www.exim.org/FAQ.html#SEC93)

And so now each time my laptop connects to a network, exim will
figure out which smarthost to use. This might suit your purposes
well. I'd just got this running on my machine so I thought I'd let
you know. Your webpage was very informative.

Cheers,

Bernard.

-- 
 Bernard Blackham
 bernard@blackham.com.au

Wishlist for email setup functionality that mutt lacks

This is by Viral Shah.

  1. Use imaps to check mail - Keep mailboxes synchronized independent of the mail client used. Mutt's imap sucks.
  2. Never delete mail from the server, and always have a local cache.
  3. Svd based searching.
  4. Bayesian spam filtering + honouring spamassassin flags.
  5. Inlining of non-program attachments (html,pdf etc).
  6. Changing my From/Reply-To based on the account I was using.
  7. Present a unified view of multiple mail accounts.
  8. A multithreaded client (compose several emails at a time, read incoming emails while writing one)
  9. Good address guessing / addressbook integration.
  10. Not crash to a halt if one of the imap servers fails to respond - Keep up with the responding ones.
  11. Easy way to change the smtp server (when changing networks).
  12. Plugins provide mail previews on matching criteria and full gpg compatibility.
  13. Fully replaceable in the future with a separate mail client - because of storing email on the server with imap.

Credits

I got help from Mohit Agarwal (mohit at foc.demonhosting.co.uk), Vikram Aggarwal (vikram at mayin dot org).

Ajay Shah, 2002