Overview of this blog
Here are some basic ideas we wish to cover.
SPF allows senders to define which IP addresses are allowed to send mail for a particular domain.
DKIM provides an encryption key and digital signature that verifies that an email message was not faked or altered.
DMARC unifies the SPF and DKIM authentication mechanisms into a common framework and allows domain owners to declare how they would like email from that domain to be handled if it fails an authorization test.
RBL establishes that the sending mail server is not belonging to a compromised IP address or bogon or open relay
Other tests SpamCheetah performs for ensuring email deliverability like senderscore and so on
What is sender policy framework?
A sender policy framework is a mechanism to mitigate sender domain spoofing. If the standard SPF is followed by all Internet mail systems, then nobody will be able to impersonate as belonging to a sending domain and represent the sending domain from a rogue IP address.
By rogue IP address is meant the IP address not owned by the sending domain. In other words the mail sender is not authorized to send mails on their behalf.
A mail receiver can perform a set of SPF checks for each mail message it receives. An SPF check tests the authorization of a client host to emit mail with a given identity. Typically, such checks are done by a receiving MTA, but can be performed elsewhere in the mail processing chain so long as the required information is available and reliable. The “MAIL FROM” and “HELO” identities are checked for SPF.
How does SPF employ DNS?
SPF is usually checked very early in the SMTP conversation, well before the body of the message has been transmitted.
When a mail attempt is made, a TCP connection is opened between the sender and the receiving mail server process.
Once the connection is established, a HELO command is issued, which essentially tells the receiving server which domain is trying to send.
$ nc spamcheetah.com 587 220 mail.spamcheetah.com ESMTP OpenSMTPD ehlo yahoo.com 250-mail.spamcheetah.com Hello yahoo.com [220.127.116.11], pleased to meet you 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-SIZE 36700160 250-DSN 250-STARTTLS 250 HELP
As you can see from above, the IP address of sender is printed by the receiving mail server.
This is followed by a MAIL FROM command that tells the receiving server what email address the message is coming from. The domain found in the MAIL FROM (also known as the envelope from and return path) command is the domain used for looking up SPF .
250-mail.spamcheetah.com Hello india.com [127.0.0.1], pleased to meet you 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-SIZE 36700160 250-DSN 250-STARTTLS 250 HELP mail from: <firstname.lastname@example.org> 250 2.0.0 Ok
So suppose a message has been received, and the MAIL FROM address is email@example.com. The receiving server will check the public DNS records for party.com and look for a TXT record that begins with v=spf1. If there is no TXT record that begins with v=spf1, authentication will pass. If there is more than one TXT record that beings with v=spf1, an error may occur.
Assume one is found, and it looks like our example from before:
“v=spf1 ip4:18.104.22.168 include:example.com -all”
The receiving server will now check to see if the IP address of the SMTP client trying to send the message is included in the SPF record. If the IP address is listed, the message will pass SPF authentication.
Here is a sample SPF DNS record
“v=spf1 ip4:22.214.171.124 include:spamcheetah.com -all”
SPF record is a short line of text that the administrator of a domain adds to their DNS txt record type.
The txt record is stored in the DNS (domain name system) alongside their
SPF record looks something like this:
“v=spf1 ip4:126.96.36.199 include:spamcheetah.com -all”
But let us look at a complex SPF record.
dig +short mxtoolbox.com txt "ZOOM_verify_5KjRz3J9SrirwTiYL3fOkw" "google-site-verification=P-RT1CxoGVlDlIrGF-m72jC3S_O6yoMRQ2azVsagoG8" "status-page-domain-verification=c4509l4110vh" "v=spf1 redirect=mxtoolbox.com.hosted.spf-report.com"
Obviously this makes no sense. Let us try again.
$ echo mxtoolbox.com|smtpctl spf walk 188.8.131.52/32 184.108.40.206/32 220.127.116.11/19 18.104.22.168/20 22.214.171.124/19 126.96.36.199/20 188.8.131.52/19 184.108.40.206/21 220.127.116.11/20 18.104.22.168/19 22.214.171.124/16 126.96.36.199/22 2001:4860:4000::/36 2404:6800:4000::/36 2607:f8b0:4000::/36 2800:3f0:4000::/36 2a00:1450:4000::/36 2c0f:fb50:4000::/36 188.8.131.52/24 184.108.40.206/22 220.127.116.11/23 18.104.22.168/23 22.214.171.124/23 126.96.36.199/28 188.8.131.52/24 184.108.40.206/24 220.127.116.11/20 18.104.22.168/20 22.214.171.124/23 126.96.36.199/26 188.8.131.52/23 184.108.40.206/20 220.127.116.11/20 18.104.22.168/24 22.214.171.124/19 126.96.36.199/20 188.8.131.52/20 184.108.40.206/18 220.127.116.11/16 18.104.22.168/21 22.214.171.124/16 126.96.36.199/17 188.8.131.52/19 184.108.40.206/19 220.127.116.11/29 18.104.22.168/25 22.214.171.124/22 126.96.36.199/21 188.8.131.52/22 184.108.40.206/22 220.127.116.11/22 18.104.22.168/24 22.214.171.124/24 126.96.36.199/25 188.8.131.52/24 184.108.40.206/30 220.127.116.11/24 18.104.22.168/24 22.214.171.124/23 126.96.36.199/24 188.8.131.52/24 184.108.40.206/20 220.127.116.11/20
As you can see, querying SPF records is not easy at all.
For that you must know a thing or two about the domain naming system or DNS in short.
Anti-spam software like SpamAssassin implements SPF.
Many mail transfer agents (MTAs) support SPF directly such as
- CommuniGate Pro
- Microsoft Exchange
According to wikipedia,
In 2017 more than eight million domains publish SPF FAIL -all policies.
In a survey published in 2007, 5% of the .com and .net domains had some kind of SPF policy. In 2009, a continuous survey run at Nokia Research reports that 51% of the tested domains specify an SPF policy.
Spammers can send email with an SPF PASS result if they have an account in a domain with a sender policy, or abuse a compromised system in this domain. However, doing so makes the spammer easier to trace.
The main benefit of SPF is to the owners of email addresses that are forged in the Return-Path. They receive large numbers of unsolicited error messages and other auto-replies. If such receivers use SPF to specify their legitimate source IP addresses and indicate FAIL result for all other addresses, receivers checking SPF can reject forgeries, thus reducing or eliminating the amount of backscatter.
SPF has potential advantages beyond helping identify unwanted mail. In particular, if a sender provides SPF information, then receivers can use SPF PASS results in combination with an allow list to identify known reliable senders. Scenarios like compromised systems and shared sending mailers limit this use.
How are netblocks useful for SPF?
Now, let us get to how netblocks or specifically CIDR netblocks are useful for SPF.
Since large mail providers like Gmail require plenty of mail servers to do its job, specifying each IP address in SPF is very difficult and verbose. To tackle this issue, entire network blocks are purchased by such companies and assigned to mail sending servers, thereby making it easy for us mail receivers to verify by doing a simple network block match (SpamCheetah does it in C language) to validate an SPF.
What is DKIM?
DomainKeys Identified Mail (DKIM) is an email authentication method designed to detect forged sender addresses in email (email spoofing).
DKIM allows the MTA to check that an email claiming to have come from a specific domain was indeed authorized by checking the hash in the mail header. Hashes of body and select headers are computed and signed.
It achieves this by appending a digital signature, linked to a DKIM selector , to each outgoing email message.
The receiving MTA can verify this by looking up the sender’s public key published in the DNS.
A valid signature also guarantees that some parts of the email (possibly including attachments in mail envelope) have not been modified since the signature was affixed.
Normally, DKIM signatures are not visible to end-users, and are affixed or verified by the infrastructure rather than the message’s authors and recipients.
DKIM is an Internet Standard. It is defined in [RFC 6376], dated September 2011.
How to query for DKIM?
Example of a DKIM selector in DNS
$ dig 1620537750.spamcheetah._domainkey.spamcheetah.com txt ; <<>> DiG 9.16.8-Ubuntu <<>> 1620537750.spamcheetah._domainkey.spamcheetah.com txt ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51551 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;1620537750.spamcheetah._domainkey.spamcheetah.com. IN TXT ;; ANSWER SECTION: 1620537750.spamcheetah._domainkey.spamcheetah.com. 1798 IN TXT "v=DKIM1;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCv4l8pTIJzD50h/I/fgoT8GIpYCJqy8okEfmYk3L9c0CNDZNEqWEvK7owXawCYxL4F3syiu723hF/01Y/DSRU0+h6/0o0viyDpSDKzxnXNSsaoGymFjqxQMJmeriK9Ad2TeQ7o9XqF/zjyvwlV1WF/dbLqtp22kO6ZiCzMM3qFcwIDAQAB" ;; Query time: 256 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Mon Jun 28 11:23:00 IST 2021 ;; MSG SIZE rcvd: 317
The procedure of querying DKIM is definitely not straight forward as obtaining access to the exact selector without background information is impossible.
The email header contains the DKIM selector using which the mail server or spam control software executes a DNS query like above to gain access to the public key.
Once the public key is obtained the hash can be freshly computed from e-mail headers and body and verified against the one mentioned in the header.
If it matches, the DKIM signature is valid. If not it is invalid and mail rejected.
However if DKIM signature is not present at all, SpamCheetah considers it a pass since that does not quality for signature failure.
Let me explain once again how this happens.
The receiver decrypts the header signature using the public key and the hash now obtained is verified against the hash that is computed afresh.
The body hash is not signed.
In the context of email it is the headers that matter more.
DNS blocklists, spamhaus and RBL
In addition to the above, many mail providers also public DMARC policies, do ARC signing, SRS and so on.
But let us get to the main idea of spam control that interests us and how relay blackhole lists or spamhaus lists help us in validating that the mail sender IP address is not a known spammer source.
There are plenty of RBL lookup tools/APIs.
Here is one such.
$ curl -s https://rbl-check.org/rbl_api.php?ipaddress="18.104.22.168" Sorbs;zombie.dnsbl.sorbs.net;nowebsite;notlisted nixspam;ix.dnsbl.manitu.net;nowebsite;notlisted Spamcop;bl.spamcop.net;nowebsite;notlisted rfc-ignorant;dsn.rfc-ignorant.org;nowebsite;notlisted surbl;multi.surbl.org;nowebsite;notlisted anti-spam;cblless.anti-spam.org.cn;nowebsite;notlisted cblplus.anti-spam.org.cn;cblplus.anti-spam.org.cn;nowebsite;notlisted cbl.anti-spam.org.cn;cbl.anti-spam.org.cn;nowebsite;notlisted five-ten-sg;blackholes.five-ten-sg.com;nowebsite;notlisted sorbs.dnsbl.net.au;sorbs.dnsbl.net.au;nowebsite;notlisted dnsbl.sorbs.net;dnsbl.sorbs.net;nowebsite;notlisted spamhaus;zen.spamhaus.org;nowebsite;notlisted wpbl;db.wpbl.info;nowebsite;notlisted rmst.dnsbl.net.au;rmst.dnsbl.net.au;http://dnsbl.net.au/rmst/;notlisted KEMPT;dnsbl.kempt.net;http://www.kempt.net/dnsbl/;notlisted Woody;blacklist.woody.ch;http://blacklist.woody.ch/rblcheck.php3;notlisted surriel;psbl.surriel.com;http://psbl.surriel.com/remove;notlisted virbl.bit.nl;virbl.bit.nl;http://virbl.bit.nl/faq.php;notlisted rot.blackhole.cantv.net;rot.blackhole.cantv.net;http://abuso.cantv.net/bl/rot;notlisted RBL.JP-VIRUS;virus.rbl.jp;http://www.rbl.jp/allrbl-e.html;notlisted IMP-WORMS;wormrbl.imp.ch;http://antispam.imp.ch/03-wormlist.html;notlisted IMP-SPAM;spamrbl.imp.ch;http://antispam.imp.ch/04-spamlist.html;notlisted INTERSERVER;rbl.interserver.net;http://rbl.interserver.net/;notlisted KISARBL;spamlist.or.kr;https://www.kisarbl.or.kr/english/;notlisted RATS;dyna.spamrats.com;http://www.spamrats.com/;notlisted ABUSE-CH;dnsbl.abuse.ch;http://dnsbl.abuse.ch/;notlisted INPS;dnsbl.inps.de;http://dnsbl.inps.de/;notlisted DRONEBL;dnsbl.dronebl.org;http://dronebl.org/;notlisted DEADBEEF;bl.deadbeef.com;http://spam.deadbeef.com/bl/;notlisted ICN;ricn.dnsbl.net.au;http://dnsbl.net.au/ricn/;notlisted ICM.EDU.PL;forbidden.icm.edu.pl;http://sunsite.icm.edu.pl/spam/bh.html;notlisted PROBES;probes.dnsbl.net.au;http://dnsbl.net.au/probes/;notlisted LASHBACK;ubl.unsubscore.com;http://blacklist.lashback.com/;notlisted BarracudaCentral;b.barracudacentral.org;http://www.barracudacentral.org/;notlisted KSI;ksi.dnsbl.net.au;http://dnsbl.net.au/ksi/;notlisted IMP-URIBL;uribl.swinog.ch;http://antispam.imp.ch/05-uribl.php;notlisted SPAMLOOKUP;bsb.spamlookup.net;http://bradchoate.com/projects/spamlookup;notlisted DOB;dob.sibl.support-intelligence.net;http://www.support-intelligence.com/dob/;notlisted RBL.JP-URL;url.rbl.jp;http://www.rbl.jp/url-e.html;notlisted RBL.JP-DYN;dyndns.rbl.jp;http://www.rbl.jp/url-e.html;notlisted CYMRU-BOGON;bogons.cymru.com;http://www.team-cymru.org/Services/Bogons/;notlisted MAPS-RSS;relays.mail-abuse.org;http://www.mail-abuse.com/nominats_rss.html;notlisted OMRS;omrs.dnsbl.net.au;http://dnsbl.net.au/omrs/;notlisted OSRS;osrs.dnsbl.net.au;http://dnsbl.net.au/osrs/;notlisted ORVEDB;orvedb.aupads.org;http://www.aupads.org/ordb.html;notlisted NETHER-OR;relays.nether.net;http://puck.nether.net/or/;notlisted GWEEP-RELAYS;relays.bl.gweep.ca;nowebsite;notlisted SORBS-RELAYS;smtp.dnsbl.sorbs.net;http://www.sorbs.net/;notlisted KUNDENSERVER.DE;relays.bl.kundenserver.de;http://relaytest.kundenserver.de/faq/admin/5.html;notlisted MAPS-DUL;dialups.mail-abuse.org;http://www.mail-abuse.com/nominats_dul.html;notlisted RDTS;rdts.dnsbl.net.au;http://dnsbl.net.au/rdts/;notlisted SORBS-SPAM;spam.dnsbl.sorbs.net;http://www.sorbs.net/;notlisted AUPADS-DUINV;duinv.aupads.org;http://www.aupads.org/duinv.html;notlisted SORBS-DYNA;dynablock.sorbs.net;http://www.dnsbl.us.sorbs.net/faq/dul.shtml;notlisted DYNIP;dynip.rothen.com;http://antispam.rothen.com/;notlisted CANTV DUL;dul.blackhole.cantv.net;http://abuso.cantv.net/bl/dul;notlisted CDL;cdl.anti-spam.org.cn;http://anti-spam.org.cn/AID/711;notlisted RBL.JP-SHORT;short.rbl.jp;http://www.rbl.jp/allrbl-e.html;notlisted KOREA;korea.services.net;http://korea.services.net/;notlisted PEOPLE.IT;mail.people.it;nowebsite;notlisted SCI.KUN.NL;blacklist.sci.kun.nl;http://wiki.science.ru.nl/cncz/Email_spam;notlisted SPAMBLOCK;all.spamblock.unit.liu.se;http://www.unit.liu.se/irt/epost/spamblocked.html;notlisted
RBLs have been in vogues in spam control landscape for a really really long time now. They are also known as DNSBLs.
Another important idea is SURBL or Spam URL RBL is a form of URL validator for ensuring that the spam mails do not cause phishing attacks.
SpamCheetah uses the abuse.ch SURBL API to validate the links in each email message.
RBLs act at a level further high in the email pipeline.
Once an IP address contacts us to send mail, we are immediately checking the RBL records to look for a match and we drop the connection immediately without looking any further.
The SURBL check happens after the
DATA command and mail is delivered
What other methods exist?
The arena of email validation is very huge and this article only scratches the surface. Some of these concepts are deep and take long time to sink in.
But email systems are getting more and more secure with anti spam systems also getting better and better.
In addition there is the big business of email marketing and campaign mails.
Should spam filters allow them or pass them?