Variable envelope return path
Encyclopedia
Variable envelope return path (VERP) is a technique used by some electronic mailing list
software to enable automatic detection and removal of undeliverable e-mail address
es. It works by using a different return path (also called "envelope sender") for each recipient of a message.
. In another scenario, the address may still exist but be abandoned, with unread mail accumulating until there is not enough room left to accept any more.
When a message is sent to a mailing list, the mailing list software re-sends it to all of the addresses on the list. The presence of invalid addresses in the list results in bounce message
s being sent to the owner of the list. If the mailing list is small, the owner can read the bounce messages and manually remove the invalid addresses from the list. With a larger mailing list, this is a tedious, unpleasant job, so it is desirable to automate the process.
Unfortunately, most bounce messages have historically been designed to be read by human users, not automatically handled by software. They all convey the same basic idea (the message from X to Y could not be delivered because of reason Z) but with so many variations that it would be nearly impossible to write a program to reliably interpret the meaning of every bounce message. RFC 1894 (obsoleted by RFC 3464) defines a standard format to fix this problem, but support for the standard is far from universal. Having said that, there are several common formats (e.g., RFC 3464, qmail
's qsbmf, and Microsoft's DSN format for Exchange
) that cover large proportion of bounces.
Microsoft Exchange can sometimes bounce a message without providing any indication of the address to which the original message was sent. When Exchange knows the intended recipient, but is not willing to accept email for him, it omits his address. If a message is sent to
While bounce message formats in general vary wildly, there is one aspect of a bounce message that is highly predictable: the address to which it will be sent. VERP takes full advantage of this. In a mailing list that uses VERP, a different sender address is used for each recipient.
The mailing list manager knows that it sent a message from X to Y, so if a bounce message is received at address X, it can only be because address Y was undeliverable, because nothing was sent from X to any other destination. Thus the important information has been extracted from the bounce message, without any need to understand its contents, which means the person in charge of the list does not need to deal with it manually.
, who first put the idea into practice in his qmail
MTA
and ezmlm
mailing list manager. See his original explanation, dated February 1, 1997.
This would result in a bounce, generated by the MTA of either example.net or example.org, with the following characteristics:
The mailing list manager can't be expected to understand the contents of this bounce, and can't deduce anything from the recipient address because hundreds of other people besides Bob were also sent messages from
The bounce, then, will be more useful:
From this bounce message the mailing list manager can deduce that a message to
This example shows the simplest possible method of matching a VERP to a list subscriber: the entire recipient address is included within the return path, with the at sign replaced by an equals sign because a return path with two at signs would be invalid. Other encoding schemes are possible.
, a mailing list that is not using VERP can combine multiple deliveries into a single transaction. It connects to the appropriate server
for the domain, gives the single sender address, the recipient addresses, and then sends the message contents only once.
A mailing list using VERP, on the other hand, must send the entire message body repeatedly, which leads to an overall increase in bandwidth
usage. This inefficiency is usually not considered a big problem, especially by qmail
users, since qmail always sends messages once per recipient, even when VERP is not being used. Some packages mitigate the impact of VERP by applying it selectively, for example a mailing list manager might only use VERP on 1 in 10 mailings. This way you can gain much of VERP's tight bounce control and accurate feedback without incurring the processing and network overhead every time.
Another problem with VERP (and with any automatic bounce handling scheme) is that there will always be some recalcitrant MTA
s on the Internet that fail to follow even the simplest standards. VERP depends on the recipients' MTAs following the rule that bounces are sent to the envelope sender. This has been a standard requirement since the dawn of SMTP in 1982 (see RFC 821), but still there are MTAs that get it wrong, usually by bouncing to the address in the
(See Imail for an example).
Systems that implement greylisting
work fine with VERP if the envelope sender follows the above mentioned format. However, some VERP implementations use message number or random key as part of VERP, which causes each post to the mailing list to be delayed unless the greylisting system treats "similar" sender addresses as being equivalent.
Electronic mailing list
An electronic mailing list is a special usage of email that allows for widespread distribution of information to many Internet users. It is similar to a traditional mailing list — a list of names and addresses — as might be kept by an organization for sending publications to...
software to enable automatic detection and removal of undeliverable e-mail address
E-mail address
An email address identifies an email box to which email messages are delivered. An example format of an email address is lewis@example.net which is read as lewis at example dot net...
es. It works by using a different return path (also called "envelope sender") for each recipient of a message.
Motivation
Any long-lived mailing list is going to eventually contain addresses that can't be reached. Addresses that were once valid can become unusable because the person receiving the mail there has switched to a different providerInternet service provider
An Internet service provider is a company that provides access to the Internet. Access ISPs directly connect customers to the Internet using copper wires, wireless or fiber-optic connections. Hosting ISPs lease server space for smaller businesses and host other people servers...
. In another scenario, the address may still exist but be abandoned, with unread mail accumulating until there is not enough room left to accept any more.
When a message is sent to a mailing list, the mailing list software re-sends it to all of the addresses on the list. The presence of invalid addresses in the list results in bounce message
Bounce message
In the Internet's standard e-mail protocol SMTP, a bounce message, also called a Non-Delivery Report/Receipt , a Delivery Status Notification message, a Non-Delivery Notification or simply a bounce, is an automated electronic mail message from a mail system informing the sender of another...
s being sent to the owner of the list. If the mailing list is small, the owner can read the bounce messages and manually remove the invalid addresses from the list. With a larger mailing list, this is a tedious, unpleasant job, so it is desirable to automate the process.
Unfortunately, most bounce messages have historically been designed to be read by human users, not automatically handled by software. They all convey the same basic idea (the message from X to Y could not be delivered because of reason Z) but with so many variations that it would be nearly impossible to write a program to reliably interpret the meaning of every bounce message. RFC 1894 (obsoleted by RFC 3464) defines a standard format to fix this problem, but support for the standard is far from universal. Having said that, there are several common formats (e.g., RFC 3464, qmail
Qmail
qmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
's qsbmf, and Microsoft's DSN format for Exchange
Microsoft Exchange Server
Microsoft Exchange Server is the server side of a client–server, collaborative application product developed by Microsoft. It is part of the Microsoft Servers line of server products and is used by enterprises using Microsoft infrastructure products...
) that cover large proportion of bounces.
Microsoft Exchange can sometimes bounce a message without providing any indication of the address to which the original message was sent. When Exchange knows the intended recipient, but is not willing to accept email for him, it omits his address. If a message is sent to
joe@example.com and the server knows that this is Joe User, it will bounce the message saying that the message to Joe User could not be delivered, leaving out the joe@example.com address altogether. VERP is the only viable way to handle such bounces correctly.How VERP solves the bounce handling problem
The hard part of bounce handling is matching up a bounce message with the undeliverable address that caused the bounce. If the mailing list software can see that a bounce resulted from an attempt to send a message to user@example.com then it doesn't need to understand the rest of the information in the bounce. It can simply count how many messages were recently sent to user@example.com, and how many bounces resulted, and if the proportion of bounced messages is too high, the address is removed from the list.While bounce message formats in general vary wildly, there is one aspect of a bounce message that is highly predictable: the address to which it will be sent. VERP takes full advantage of this. In a mailing list that uses VERP, a different sender address is used for each recipient.
The mailing list manager knows that it sent a message from X to Y, so if a bounce message is received at address X, it can only be because address Y was undeliverable, because nothing was sent from X to any other destination. Thus the important information has been extracted from the bounce message, without any need to understand its contents, which means the person in charge of the list does not need to deal with it manually.
Origin
The first serious advocate of this solution, and the originator of the term VERP to describe it, was Daniel J. BernsteinDaniel J. Bernstein
Daniel Julius Bernstein is a mathematician, cryptologist, programmer, and professor of mathematics at the University of Illinois at Chicago...
, who first put the idea into practice in his qmail
Qmail
qmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
MTA
Mail transfer agent
Within Internet message handling services , a message transfer agent or mail transfer agent or mail relay is software that transfers electronic mail messages from one computer to another using a client–server application architecture...
and ezmlm
Ezmlm
ezmlm is a software package for managing electronic mailing lists by Daniel J. Bernstein. It is similar to GNU Mailman and Majordomo but only works with the qmail mail transfer agent. It is released into the public domain....
mailing list manager. See his original explanation, dated February 1, 1997.
Example
Assume there is mailing list calledwikipedians@example.net and that an individual, bob@example.org has subscribed to it, but later on, Bob has left example.org, so his address is no longer valid. Consider what happens when someone sends a message to the list.Without VERP
Without VERP, the mailing list manager might send a message with the following characteristics:- envelope sender:
wikipedians-owner@example.net - recipient:
bob@example.org
This would result in a bounce, generated by the MTA of either example.net or example.org, with the following characteristics:
- envelope sender: empty
- recipient:
wikipedians-owner@example.net - contents: example.org was unable to deliver the following message to bob: ...
The mailing list manager can't be expected to understand the contents of this bounce, and can't deduce anything from the recipient address because hundreds of other people besides Bob were also sent messages from
wikipedians-owner@example.net.With VERP
With VERP, the original message would be different:- envelope sender:
wikipedians-owner+bob=example.org@example.net - recipient:
bob@example.org
The bounce, then, will be more useful:
- envelope sender: empty
- recipient:
wikipedians-owner+bob=example.org@example.net - contents: example.org was unable to deliver the following message to bob: ...
From this bounce message the mailing list manager can deduce that a message to
bob@example.org must have failed.This example shows the simplest possible method of matching a VERP to a list subscriber: the entire recipient address is included within the return path, with the at sign replaced by an equals sign because a return path with two at signs would be invalid. Other encoding schemes are possible.
Software that supports VERP
- ezmlmEzmlmezmlm is a software package for managing electronic mailing lists by Daniel J. Bernstein. It is similar to GNU Mailman and Majordomo but only works with the qmail mail transfer agent. It is released into the public domain....
- GNU MailmanGNU MailmanGNU Mailman is a computer software application from the GNU project for managing electronic mailing lists.Mailman is coded primarily in Python and currently maintained by Barry Warsaw...
- eximEximExim is a mail transfer agent used on Unix-like operating systems. Exim is free software distributed under the terms of the GNU General Public License, and it aims to be a general and flexible mailer with extensive facilities for checking incoming e-mail....
, using a specialized Router/Transport combination - MoodleMoodleMoodle is a free source e-learning software platform, also known as a Course Management System, Learning Management System, or Virtual Learning Environment...
- postfixPostfix (software)In computing, Postfix is a free and open-source mail transfer agent that routes and delivers electronic mail. It is intended as a fast, easier-to-administer, and secure alternative to the widely-used Sendmail MTA....
- qmailQmailqmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
- SendmailSendmailSendmail is a general purpose internetwork email routing facility that supports many kinds of mail-transfer and -delivery methods, including the Simple Mail Transfer Protocol used for email transport over the Internet....
, with a ruleset. - Mercury Mail Transport SystemMercury Mail Transport SystemMercury Mail Transport System is a standards-compliant donationware mail server developed by David Harris, who also develops the Pegasus Mail client....
- ZimbraZimbraZimbra Collaboration Suite is a groupware product created by Zimbra, Inc., located in Palo Alto, California, USA. The company was purchased by Yahoo! in September 2007, and subsequently purchased by VMware on Tuesday, January 12, 2010. The software consists of both client and server components...
- Strongmail
- STEdb
- Courier Mail ServerCourier Mail ServerThe Courier mail server is a mail transfer agent server that provides ESMTP, IMAP, POP3, SMAP, webmail, and mailing list services with individual components. It is best known for its IMAP server component....
- SympaSympaSympa is an open source mailing list management software. Its name, which is an acronym for Systeme de Multi-Postage Automatique Sympa is an open source (GNU GPL) mailing list management (MLM) software. Its name, which is an acronym for Systeme de Multi-Postage Automatique Sympa is an open source...
Disadvantages
The use of VERP requires each message to be sent once for every recipient, instead of once to each receiving SMTP server. This is because of a limitation of SMTP, which allows multiple recipient addresses to be specified in a single transaction, but only one sender address. When there are many subscribers in the same domainDomain name
A domain name is an identification string that defines a realm of administrative autonomy, authority, or control in the Internet. Domain names are formed by the rules and procedures of the Domain Name System ....
, a mailing list that is not using VERP can combine multiple deliveries into a single transaction. It connects to the appropriate server
Server (computing)
In the context of client-server architecture, a server is a computer program running to serve the requests of other programs, the "clients". Thus, the "server" performs some computational task on behalf of "clients"...
for the domain, gives the single sender address, the recipient addresses, and then sends the message contents only once.
A mailing list using VERP, on the other hand, must send the entire message body repeatedly, which leads to an overall increase in bandwidth
Bandwidth (computing)
In computer networking and computer science, bandwidth, network bandwidth, data bandwidth, or digital bandwidth is a measure of available or consumed data communication resources expressed in bits/second or multiples of it .Note that in textbooks on wireless communications, modem data transmission,...
usage. This inefficiency is usually not considered a big problem, especially by qmail
Qmail
qmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
users, since qmail always sends messages once per recipient, even when VERP is not being used. Some packages mitigate the impact of VERP by applying it selectively, for example a mailing list manager might only use VERP on 1 in 10 mailings. This way you can gain much of VERP's tight bounce control and accurate feedback without incurring the processing and network overhead every time.
Another problem with VERP (and with any automatic bounce handling scheme) is that there will always be some recalcitrant MTA
Mail transfer agent
Within Internet message handling services , a message transfer agent or mail transfer agent or mail relay is software that transfers electronic mail messages from one computer to another using a client–server application architecture...
s on the Internet that fail to follow even the simplest standards. VERP depends on the recipients' MTAs following the rule that bounces are sent to the envelope sender. This has been a standard requirement since the dawn of SMTP in 1982 (see RFC 821), but still there are MTAs that get it wrong, usually by bouncing to the address in the
From: headerHeader (information technology)
In information technology, header refers to supplemental data placed at the beginning of a block of data being stored or transmitted. In data transmission, the data following the header are sometimes called the payload or body....
(See Imail for an example).
Systems that implement greylisting
Greylisting
Greylisting is a method of defending e-mail users against spam. A mail transfer agent using greylisting will "temporarily reject" any email from a sender it does not recognize. If the mail is legitimate the originating server will, after a delay, try again and, if sufficient time has elapsed, the...
work fine with VERP if the envelope sender follows the above mentioned format. However, some VERP implementations use message number or random key as part of VERP, which causes each post to the mailing list to be delayed unless the greylisting system treats "similar" sender addresses as being equivalent.
See also
- Bounce messageBounce messageIn the Internet's standard e-mail protocol SMTP, a bounce message, also called a Non-Delivery Report/Receipt , a Delivery Status Notification message, a Non-Delivery Notification or simply a bounce, is an automated electronic mail message from a mail system informing the sender of another...
- Bounce Address Tag ValidationBounce Address Tag ValidationIn computing, Bounce Address Tag Validation is a method, defined in an Internet Draft, for determining whether the bounce address specified in an E-mail message is valid...
(BATV) - for bounces from backscatterBackscatter (e-mail)Backscatter is incorrect automated bounce messages sent by mail servers, typically as a side effect of incoming spam.... - Sender Rewriting SchemeSender Rewriting SchemeSender Rewriting Scheme is a technique to re-mail an email message so that eventual Delivery Status Notifications can reach the original message sender...
(SRS) - for bounces from e-mail forwarding and SPFSender Policy FrameworkSender Policy Framework is an email validation system designed to prevent email spam by detecting email spoofing, a common vulnerability, by verifying sender IP addresses. SPF allows administrators to specify which hosts are allowed to send mail from a given domain by creating a specific SPF... - Simple Mail Transfer ProtocolSimple Mail Transfer ProtocolSimple Mail Transfer Protocol is an Internet standard for electronic mail transmission across Internet Protocol networks. SMTP was first defined by RFC 821 , and last updated by RFC 5321 which includes the extended SMTP additions, and is the protocol in widespread use today...
(SMTP)