On some broken Mail Transport Agents

You have been referred to this document for one of a number of reasons.
  1. You are a "postmaster" or administrator for a site which someone thinks is broken in the way described here.
  2. You are a list manager whose list isn't quite working right, because some addresses on your list go to sites broken in the way described in this document.
  3. You are a user at a site who may be excluded from participation in some Internet mailing list or discussion group because your site (or some site that handles mail for you) is broken in the way described here.
  4. You are ultimately responsible for some site which has been accused of misbehaving in the way described here. (That is, you are the boss of the "postmaster".)
Although this document should be read by all of those people listed above, its language is directed to the the postmaster or site administrator of the broken site.

This document is being written because I, Jeffrey Goldberg, have become fed up with the trouble that some misconfigured or broken Mail Transport Agents (MTAs) are causing to users and managers of Internet mailing lists. The tone of this document may often seem harsh, but it is often better to be clear and direct, then to paper over serious problems.

In this document, I am speaking for myself, although it was written while I worked at the Computer Centre of Cranfield University as part of the email management team, and specifically for managing mailing lists. A version of this document was hosted there for some time.

It ain't what you send but the way how you send it (that's what gets me so)

You have not been referred to because you issued a non-delivery report (henceforth "bounce") for some message. The problem is that you sent the bounce to the wrong address. The bounce may have been fully justified and correct, but your mail system is systematically sending them to the wrong place in ways that can and do interfere with the operation of discussion groups and other bits of Internet email.

So please read this document seriously. Email list managers may choose to exclude your site from certain email services as a consequence of misconfiguration. There has been some discussion of creating a "black list" of such sites.

RFC821, RFC822 and their successors

RFC821 and RFC822 along with successor documents describe what it means to be running an Internet SMTP service. There are many updates do them adding additional features in SMTP for ESMTP updating RFC821, and a very large number of updates to RFC822, many having to do with MIME and character encodings, none of the updates change the features being discussed.

Responsibilities of being on the net

If you are running a service on the standard SMTP port (25) on a public network (one that can be reached on that port from the network at large) then you have a responsibility to run it according to the standards. The standards specify what other sites have the right to assume about how your site behaves. You have probably been referred to this document because mailing list managers and mailing list software make certain assumptions - justified by these standards - which your site is not living up to.

Many sites violate some minor aspects of these standards, but the aspect of the standards that has led to this document being written are things that software systems since 1982 (the date of the standards) have relied on. These are not obscure points of the standards that no one has cared about. Violating the standards does not add or enable some new (non-standard) feature either.

So by connecting your machine to the network you have taken on a responsibility to behave correctly. Passing Email from one host to another is a cooperative activity. Failing to behave correctly makes life harder for other sites. One very important thing to keep in mind is that installing mail transport software from some big and popular software companies does not abdicate you of your responsibility if that software behaves badly on the net. You have the responsibility to get your system acting as a good network citizen or removing yourself from the net. Don't assume that just because a software company is big, that its software behaves correctly. You can't pass the buck.

Contemporary systems make it seem that one can manage a site on the Internet with little more expertise then to be able to drive a graphical interface to configure a mail system. That view is misleading. The software may seem easy to configure, but it appears that it is easy to configure wrongly. You, or someone at your site, still has to understand the basics of the standards of mail transport.

Initial Summary

A brief summary is that your site is sending delivery error reports to the header From or Reply-To addresses instead of sending to the envelope From or header Sender addresses. That behavior is not only a violation of the standards, but it adversely effects the management of Internet mailing lists.

As a site potentially sending out error reports in response to Internet mail, you have a responsibility to either configure your software to work correctly or to replace your software with something that does work correctly. What you use for your local system is your business, but if it talks to the rest of the network, then failure to do so properly is destructive to the operation of the network as a whole.

What the standards say

Anyone managing an email switch for a site should be broadly familiar with the standards that define Internet email transport. While it is true that "the software is supposed to worry about the details", it is still the mail administrator's responsibility to select, install, and configure software which does it correctly. Some off the shelf software from very large software suppliers is broken in the way described in this document. Do not just assume that you can defend your set-up by saying that you are using a popular system.

Mail messages come with many "return addresses". In the actual header of a message, there may be things like "From:", "Reply-To:" and "Sender:". Additionally when two mail systems talk to each other, the sending one provides what is sometimes called a "reverse-path", but is more commonly called an "envelope from". Note that the envelope from is not the same thing as the from line in the mail header. In this document, I will refer to one as the "envelope from" and the other as the "header from".

RFC821 says in section 4.1.1 on the "MAIL" command (which is the one that provides the envelope from)

The [address given by the MAIL command] consists of [...] the sender mailbox. [...] This [address] is used [...] to return non-delivery notices to the sender.

The original wording of the RFC spoke about "lists" and "source routes". In current practice, those can safely be replaced by "address" and "return address" etc, as I have done.

So, if an MTA is going to generate a bounce, it should bounce it to the address in the envelope from if it exists.

What happens if an MTA gets confused and forgets that its an MTA and tries to work from the header information instead of the envelope information.

This is from RFC822, section 4.4.4.

4.4.4. AUTOMATIC USE OF FROM / SENDER / REPLY-TO

For systems which automatically generate address lists for replies to messages, the following recommendations are made: Sometimes, a recipient may actually wish to communicate with the person that initiated the message transfer. In such cases, it is reasonable to use the "Sender" address.
The first point is very clear. Bounces should be sent to the address listed in the "Sender:" field if it exists. Only if it doesn't exist should it defer to the information in the header "From:" field. It should not use the "Reply-To:" field.

Note also that normal user level automatic responses should use the "Reply-To:" field, then "From" if Reply-To isn't there, and should avoid "Sender:" for most purposes. But delivery errors should go to the information in "Sender:" if not to the envelope from.

Why does this matter for mailing lists

Internet mailing lists or discussion groups specifically set make use of these features of the standards. Since mailing lists/discussion groups may have thousands of members, it is important that errors about addresses go the the managers of the list and not to the particular person you posted to the list and most certainly not to the whole list.

Mail list systems have been explicitly designed since 1982 (and before, since that standards actually codified what was common practice back then) to ensure that both the Sender field and the envelope from contain the address of the list manager. It is the address of the list manager that error reports for addresses on a list should be sent to.

Unfortunately, some MTA software is either completely broken or is misconfigured so that bounces get treated like user-level auto responses going first to a Reply-To and then to a From. Depending on how the mailing list is set up, this can lead to bounce reports going to the entire list or to the person who posted to the list. The situation can persist for quite some time (and leading to a cascading discussion of the bounces) before the list manager even becomes aware that there is a bad address on the list.

So what should you do?

There are several options are open to you
  1. Reconfigure your MTA to behave properly. If the supplier of the software doesn't provide the assistance that you need, then you may need to hire someone who can do it.
  2. Replace your MTA software with an MTA that does behave properly. There are many suppliers of good MTAs out there. Some of the best are entirely free.
  3. Inform your users that they are not to join any Internet mailing lists because your system doesn't respect the relevant Internet standards for allowing the smooth function of mailing lists.
In all cases, learn about the standards and systems that are part of participating in the community of networked sites

Version: $Revision: 1.9 $
Last Modified: $Date: 2001/06/04 23:14:25 $ GMT
First put under revision numbering: March 18, 2001
First established at prior location: Some time in 1997--1998
Author: Jeffrey Goldberg