What is greylisting ?

What is greylisting in SMTP?

E-mail Spam has always been a problem that has been around ever since email came into being.variables

Greylisting schematic

From the above diagram it must be somewhat clear what greylisting is.

This video will help you as well.

Greylisting video

What you can infer from above video is that the email is not accepted and a reject message is sent by the fake SMTP server process even before sending the E-mail body(the DATA command).

The result being that the spam sender if caught in this mess will simply go away and not retry.

According to RFC 5321 every standards compliant SMTP implementation or mail server is supposed to retry when a 451 temporary error is returned by the other side of the SMTP session.

However spammers often operate with a volume business and this standards conformance and retrying logic will mean more resources/money and that will not suit them.

Most spam, not all is sent by automated bots or programs and not humans. They take up temporary IP address blocks known as bogons and pump spam to as many email addresses they can and then go attack someone else from some other network block.

Theirs is a volume game. If they target million recipients and in their sales funnel if they get a minuscule conversion percentage their purpose is met. Unforunately this annoys the rest of us.

This method of pumping unwanted junk mail is called as botnet spew.

This is not the only way spam mails are sent but the vast majority falls in this category.

There have been several techniques to arrest this from hitting your inbox.

  • content filtering
  • IP reputation of sending mail server
  • Sender score based blocking
  • checking content hash against known hashes
  • Several variations of content analysis
  • CRM114 Markov chains
  • Bayesian filtering

And much more.

But then the most effective of all is greylisting and we shall presently analyze why that is so.

Already the mail sending IP address is being used in RBL lists and DNSBL lists and so on. But the IP addresses change all the time. That is why we must depend on SPF and keep the lists of current known spammer IPs up-to-date.

Greylisting as an effective method to combat spam has been around for decades but with a key problem.

The e-mail hitting your inbox from the sending IP address is delayed by few minutes or hours depending on greylisting configuration Once the IP address is whitelisted, then emails are no longer delayed

This can be an issue and there is another problem as well.

More and more SMTP senders are using a network of IP addresses to retry the SMTP failure. And this wreaks havoc on greylisting. But there are fixes to address this behavior.

One obvious solution is to whitelist all the popular E-mail services that are known to have a huge bank of IP addresses to send out email.

Since greylisting never accepts any mail that is not standards compliant the major benefit we get is we save huge bandwidth and lessen network load.

Most of the core routers are even today in 2021 stressed by various trojans, malwares and junk mail traffic and greylisting if used widely can lessen their load in a big way.

In my experience however, content filtering is a very lame way of addressing the spam problem. And in today’s scheme of things that has a place but deep down in the hierarchy of methods. And Spamcheetah has content filtering too but it does not play a big role in its marking a message as spam.

Content filtering must be augemented with training and there can be several subjective interpretations, the less said about it the better.

How effective is greylisting in spam control?

In terms of raw effectiveness , the load on SpamCheetah as well as the target mail server are both cut down dratically by greylisting.

Without greylisting every email attempt reaches the mail server. But with greylisting, only the standards compliant mail servers are able to deliver mail.

Even after greylisting, the filtering of e-mail header body and attachmet takes place and none of those are bypassed.

The idea is that the greylisting system has to settle down and understand your frequent sender IP addresses. Greylisting is only concerned about whether your mail sender is genuine or not.

Once a meaningful conclusion is arrived through subjecting your network’s mail senders to the greylisting filtering process, then the procedure is very straight forward.

Most of the mails now flow without being subjected to greylisting logic. This not only means instant inbox delivery but also ensures that new mail senders must still comply by the rules to be considered genuine.

Also the IP addresses that bypass greylisting once proven to be genuine do not stay in the trust list forever. They expire and you must prove once again. This is done due to the churn in Internet in which good IP addresses become bad and vice versa.

Blacklisting and greytrapping

In addition to the above, SpamCheetah also does something called as greytrapping.

This works in conjunction with blacklisting of known spammer networks/botnets.

What this does is that, the email communication is deliberately slowed down as you would seen in this video.

This is called as stuttering. And you have the option of running SpamCheetah in blacklisting only mode in which the known spammer networks obtained through RBL or something can be subjected to deliberate fake SMTP sessions whereby they never get a chance to talk to the real mail server.

Does greylisting cause false positive or false negatives?

Well no technique in spam control is 100% foolproof. Whenever you combat spam using a method,there will always be the danger of false positive and some spam getting through.

Greylisting however does not lead to false positives since all standards compliant mail servers are whitelited eventually and succeed in the greylisting process.

However that said, there is an increasing number of cloud based mail sending networks that keep retrying mail from different IP addresses. Many greyliting implementations get confused and sometime legitimate mails can get lost or delayed much.

Can you turn off greylisting in SpamCheetah?

Greylisting used to be the only trick in town for SpamCheetah 15 years ago when the product first came into being. But today, it is only one of the several other methods employed to combat junk mail.

You can enable proxy mode in SpamCheetah UI and bypass greylisting completely. Or you can run it in blacklist only mode whereby also greylisting is turned off.

The sky is the limit when it comes to how you decide to configure the spam control engines within SpamCheetah.

One of the most attractive features of greylisting continues to be its effectiveness in spam control. What with its drawbacks.

Where is the world headed as regards greylisting?

After the advent of big mail companies in which a very large percentage of mail traffic is between Gmail and Office365, the interest in running individual email servers and spam control to protect them have been coming down over the years.

But even if the big companies keep on attracting customers there will still be people that have a really valid reason to not trust the big player or do things on their own for a justifiable reason.

It is also a myth that running a mail server is a hassle. Like so many myths like flat earth theory, this has no basis whatsoever.

The only way to truly hurt spammers which is what greytrapping attempts.

And in order for it to do that well, you must run more and more mail servers where this happens. This not only combats spam for us effectively and meaningfully it also reduces the spam volume despatched to other unsuspecting user/networks.

The intention to do greylisting may or may not be altruistic but the fact is that greylisting does have an overall impact on reducing e-mail spam and malware spread.

In fact most of the malware traffic that leads to infections on the Internet are carried in spam payloads only. Without e-mail even today, no malware would spreas.

This is not strictly true as normally government backed attacks are more sophisticated and do not depend on e-mail.

The way e-mail standards created 3 decades ago have evolved very little, the spammer business model and their methods have also evolved very little and in the spam control arsenal the very same methods that existed 10 years ago are in vogue today as well.

And greylisting as a technique to fight spam is relevant today as it always was.

How is greylisting different from other approaches?

If you want to kill spam there is always a mix of methods to be employed. Instead of relying only on few time tested methods we must constantly evolve with the times and attack vectors.

The fact that greylisting works very well with 90% of the spam traffic is good news. But let us not stop there. We must do whatever we can to identify a given message as spam or ham.

It could be content filtering, it could be using DCC or Razor or whatever else you have up your sleeve.

Spam as a problem must be fixed and fixed well. Since your network and uers are at attack and you as an MSP is answerable to several SLAs you sign and promises you make to customers.

As long as the expectations are clear that no product in this universe can stop 100% of all spam, we must keep upping our game.

Don’t you get spam in your gmail INBOX? You do.

And with all our capabilities we still can’t deal with targeted Spearphishing without doing sophisticated things that are geared towards addressing the issue of Business E-mail Compromise or BEC.

The world is not yet done with spam. And mail.

We must keep fighting even if we are tired. No other go.

So how to address shortcomings of greylisting?

Greylisting is now without its own set of issues. When you send an email you expect to see the mail land in INBOX instantly don’t you?

And if it lands in spam foler then you wish to see your next mail in INBOX once the recipient marks the mail as not spam.

In the case of greylisting, the mail is simply high up in air.

You have delays and this is thankfully only for the mail server. So if your mail server is whitelisted then any number of email from your mail server gets delivered instantly to the recipients despite presence of greylisting.

There is also the issue of cloud based mail server groups that wreak havoc with greylisting. This problem has only gotten worse over last 10 years. There are methods to combat this, mostly by SPF whitelisting and doing greylisting by keying sender domain instead of sender IP address.

But despite of all that, greylisting still works and works well to fight spam.

Every spam control product must employ it in some form or other.