Simple mail transfer protocol is one of the first text based protocols that software engineers play with. You can easily fire up a netcat or telnet session,talk to a mail server and deliver email.
It is very similar to HTTP and it is stateful protocol, a transaction starts once you issue the DATA command.
The excitement of seeing a mail sent by using low level semantics is not only very gratifying for a smart programmer it also used to serve a useful purpose.
And it is no longer possible to live so simply because we no longer live in a wholly trusted world where we can be believed to be who we claim to be.
E-mail spam filters started putting all sorts of rules and regulations on how emails are to be sent, who can send it, from which IP addresses, how many in a minute and so on.
The long list of rules are worrisome to many but people always find ways to fight them.
Or else by today spam would be a solved problem.
The SMTP standard was developed around the same time as Usenet, a one-to-many communication network with some similarities.
The original SMTP protocol supported only unauthenticated unencrypted 7-bit ASCII text communications, susceptible to trivial man-in-the-middle attack, spoofing, and spamming, and requiring any binary data to be encoded to readable text before transmission.
Popularity of the Internet meant that SMTP had to include specific rules and methods for relaying mail alongside authenticating users to prevent abuses such as relaying of unsolicited email (spam).
As this protocol started out purely ASCII text-based, it did not deal well with binary files, or characters in many non-English languages. Standards such as Multipurpose Internet Mail Extensions (MIME) were developed to encode binary files for transfer through SMTP. Mail transfer agents (MTAs) developed after Sendmail also tended to be implemented 8-bit-clean, so that the alternate “just send eight” strategy could be used to transmit arbitrary text data (in any 8-bit ASCII-like character encoding) via SMTP.
The whole world of E-mail regardless of the details works the same way. Whether or not it is a simple text mail, or html mail or with signature or with an audio or image or video attachment the way an e-mail flows through the wire or if wireless is the same.
The simple text protocol SMTP either in its plaintext form or encrypted form carries all the e-mail traffic this world sees in 2022.
The internal details of how this all works behind the scenes may differ but at a bare level, the protocol is text based and is SMTP as described in this rfc.
SMTP has been discussed in detail with some practical examples here as well.
The way SMTP works is fully described in RFC5321 and for those not familiar with reading RFC documents it could be very intimidating.
In this blog however we shall examine in enough detail what the SMTP protocol means and how it moves E-mail from point A to point B on the Internet.
The world of SMTP has changed little ever since the advent in the 70s and though a lot of improvements have been seen to combat spam , malware and phishing by way of signatures, DKIM, SPF and the like SMTP as a transport for e-mail has not been modified.
You can send e-mail over an encrypted tunnel, a VPN, a PGP channel, SMIME or GPG, but ultimately it is still SMTP only.
This is very likely not going to change in future either.
But the way mail flows changes slightly when you insert a secure E-mail gateway like SpamCheetah in between. Or you apply authentication or proxying but they all must not mess with SMTP semantics or mail flow will be disturbed.
Now let us discuss the topic of MIME or multipurpose Internet Mail Extensions. This is something that has taken the E-mail experience to a totally new level since without MIME we cannot even have a logo in an e-mail or display HTML emails properly.
Now with MIME you can send videos, multiple audios , PPT files, Excel files and what not.
The idea of MIME can be thought of as some kind of packaging like the Matryoshka doll in which layer after layer of MIME bodies can be packed together and sent as one single message stream.
It is the job of the MIME builder or MIME parser to interpret this packaging and the intermediate SMTP implementations need not bother at all, for them it is just an ordinary text mail message.
All SMTP implementations allow 8bit mime or allow binary data transcoded as text in order to abide by the rules of the SMTP text protocol semantics.
And in my experience a typical E-mail is of size 5 kilobytes or something. And even with multiple instant messaging communication tools like Whatsapp, slack messenger, people still predominantly use e-mail for business and personal purposes.
Mailing lists are public forums where SMTP is being used to obtain support and ask questions in an open source environment. This was also popular for commercial software but nowadays not very common.
In a mailing list, one single SMTP transaction results in a to e-mail address that translates the mail to be forwarded to 1000s of E-mail addresses. A mailing list manager like mailman can do that.
Any e-mail with attachment can now be despatched using a combination of MIME and tools that process them. Some secure E-mail gateways frown upon attachments since they can contain active content that can carry viruses or other malware.
Things have gotten so bad that some mail providers do not even allow zip files to be sent as E-mail attachments.
People come up with online uploading sites like dropbox to deal with this problem.
The SMTP protocol itself on the wire has had some minor adjustments over the years as is understandable. But at a basic level it has retained its 5 or 6 commands and the transactional semantics of sending and receiving mail.
A typical mail environment involves the following parties.
This in short captures the entire E-mail story. Of how an E-mail starts at point A on the Internet and is read by point B on the Internet across the world, in outer space or across the street.
Regardless of how many hops it goes through, the way it works at a broad level are exactly same.
This fact has not changed with all these years, but the E-mail threat control methods have evolved and are changing even today with most E-mail security vendors preferring the API model in lieu of the secure email gateway deployment architecture.
But it is unlikely SEG s shall go away since the convenience and effectiveness of SEGs in the long run can’t be matched by API based E-mail threat control solutions.
SMTP in itself has changed little and if past is a good predictor of the future then very likely it is going to continue that way.
Perhaps things will get more and more stringent as time goes on to protect users from abuse like spam of BEC or UCE/ UBE.
The world of SMTP is very exciting with newer and more robust mail server implementations like OpenSMTPD entering the scene and gaining quick adoption.
But due to the commercial players like Gmail and Office365, the end user E-mail server deployments have gone down and people no longer prefer running their own mail servers in 2022 like they used to do in 2010.
Hopefully this shall change once people realize the dangers in trusting one’s critical infrastructure with big companies.
Moreover this leads to a harmful monoculture and crappy end user experiences.