SpamCheetah internals

What problems do MSPs face today?

In 2021, Managed service providers (MSPs) face an email spam threat that can be broadly categorized as falling into several heads

  • phishing
  • time waste
  • business email compromise
  • viruses
  • malware
  • losing important mails when deleting spam

and so on.

Nobody likes to see email from unwanted sender or with unwanted content.

But email being what it is, this will never be the case.

So we must learn to deal with spam.

How?

Why do we need a spam filter?

How does a spam control product fit in?

Nearly all of the malware, trojans, trapdoors and other unwanted junk that gets into our network has its origins in email spam.

This is something that has been true from 30 years ago.

No amount of countermeasures has been able to fix this core issue.

Perhaps the case for spam control products is shouting through the rooftops but the problem is still there.

In addition to several other problems faced by MSPs in today’s world like managing scale and moving to cloud and so on, the email spam problem remains a pesky issue to be addressed effectively.

What is the ROI(Return on investment)?

Any dime spent by the MSP must be recovered from the customers and no price is too high to pay for peace of mind.

And spam control products attempt to deliver peace of mind at varying price ranges.

In order to achieve real happiness and freedom from support calls and so on, we must deploy and manage our e-mail assets meaningfully and effectively.

Even today, despite of Whatsapp and phone and instant messaging, email still continues to be most important method of contacting someone and the deluge of unwanted junk messages in a given INBOX annoys end users.

And in order to get a clean INBOX experience spam filters are purchased deployed and licenses renewed.

Or the entire mail service outsourced.

Cloud hosted vis-a-vis on prem

Cloud hosting of spam control products mean less hassle/less work. You get a clean inbox by offloading all spam control to the cloud.

There is no hassle of managing your spam control assets within your network. And the deployment problems are also minimized though not avoided totally.

On premises spam control appliance implies more burden and more effort on the part of system admin but in most cases this burden is worth it in peace of mind.

Also some jurisdictions do not allow data that is as sensitive as email to be with a third party cloud provider.

These are some of the considerations for choosing cloud vis-a-vis on premises deployment of email security gateway products.

SpamCheetah’s terminology

In the world of email spam , SpamCheetah performs its tasks by making use of other well known successful third party plugins and tools and by using a wide variety of programming languages, tools and frameworks and uses Angular material based web interface to manage and monitor the appliance. In this blog , we shall explore how most of these complex subsystems work with one another to give a seamless experience to you, the consumer.

What all techniques does SpamCheetah use to combat spam?

E-mail Spam has always been a problem that has been around ever since email came into being.

Techniques to combat spam

Internal components

Internal components

This is a high level, 35,000 feet high overview of the various components inside SpamCheetah that go towards making SpamCheetah tick.

You can observe that topics like URLScan API check and malware checks do not figure high in this document.

Reason is that they are third party plugins that SpamCheetah invokes and the design does not interact with the overall workings of the product.

Whereas this document is more focused on the design elements that make the whole product work not only from the spam control angle.

From the perspective of effectiveness of spam control, there are several things that matter more than the internal design discussed here. For a blog article on that, please see here

Clustering of nodes within LAN

Clustering of nodes within a LAN

SpamCheetah supports CARP clustering of SpamCheetah nodes within the same network LAN segment since clustering relies on multicast and at this time the clustering subsystem is not fully functional. It is code that is not well tested yet. So more details will be available in a future release.

The clustering of SpamCheetah follows an active passive model where each node of SpamCheetah is identical in all respects/configuration and internal states that when one node is found down the other will take over within milliseconds.

This is a very interesting feature for MSPs but this requires a lot of testing and synchronization and our company has not been able to invest in this so far. There shall be a future update in this blog article once this feature is ready for production use.

Daily Cron jobs

Daily Cron jobs

No serious product in today’s world can work well without some housekeeping and other scheduled tasks that happen on a daily basis.

SpamCheetah is no different.

Quarantine mailer invocations, clearing some DB entries and file system junk and such are done at this time.

All cron jobs are very vital. For instance, the freshclam script used by ClamAV virus scanner is very important and so is the update of several other API data.

Quarantine and licensing are the most important aspects of the cron job today but that will change as more features pile on.

Greylisting internals

Greylisting internals

Greylisting is one of the key subsystems in SpamCheetah and this schematic gives a detailed overview of how it all fits together.

Although greylisting is a very critical component of spam control you can turn it off should you feel uncomfortable with its delays or other side effects.

SpamCheetah will still function and stop spam using other methods described in this article

Greylisting design

The design of greylisting in SpamCheetah is very simple. Store all mail attempts that we reject with a 401 response code in a Berkeley DB database.

Then look for retries within a specified time window. Whilst the mail sending IP address is in greylisting queue we can inspect the time window , how many attempts it made after rejection and so on.

Greylisting design

Usually the whitelisting that follows greylisting also has an expiry date but that is usually too high from time window of greylisting that it is for all practical purposes permanently whitelisted.

This is to avoid situations where mail sending IP addresses churn from being good to being bad and vice versa.

IP address assets keep changing hands in today’s Internet.

Licensing

Licensing

The licensing subsystem of SpamCheetah uses the typical network model of most software licenses in which it incorporates a license server that tracks expiry and renewal and the purchase and key activation are all facilitated with the help of the website and web interface of the product.

License server not being reachable is an offense and SpamCheetah will be taken down. So please open your firewall to allow this traffic.

Mail alerts system

SpamCheetah supports a fairly sophisticated email alerts subsystem by which licensing expiry, mail server down and other serious events are communicated.

Mail alerts system

If the target mail server has submission port open, we use that. Else we use the SMTP port in the target mail server to deliver alerts.

E-mail alerting is very critical for many operations and SpamCheetah hopes the admin takes swift action in response since usually alerts indicate something serious that requires immediate corrective action.

Concurrent SMTP proxy invocation

Concurrent SMTP proxy invocation

The concurrent processing of SMTP proxy processes is limited only by the RAM and swap usage assuming there are no memory leaks or crashes in the code.

The ability to process each mail independently and still listen to fresh mail and communicate the global counters for graphing and analytics involved a great deal of complex coding and Spamcheetah uses shared memory for achieving inter process communication between child processes and parent that responds to the UNIX domain socket command that lists all the counters.

For each passing mail or virus or spam that is blocked a counter value is incremented and accounted for.

Proxy flow of email

This is the most fascinating aspect of SpamCheetah. The C SMTP proxy code that receives e-mail on behalf of the mail server and acts as an intermediary ferrying traffic back and forth and interfering only when needed and as needed.

Proxy flow of email

This design requires a great deal of in-depth knowledge of the UNIX/POSIX process design and so on.

In our tests we found SpamCheetah to be processing a great deal of mail traffic in sub second times even on a machine with very low resources like CPU and RAM.

Quarantine flow

The quarantine system in SpamCheetah involves two steps:

  1. First quarantine mail by stopping and not delivering to INBOX

  2. Next step is to send all quarantined traffic for user to delete or deliver

The ability to quarantine viruses or spam messages or rejected attachments is given in the web interface that the admin can setup.

You can also schedule the quarantine mailers to clear

  • weekly
  • daily
  • monthly

All quarantine mails are however deleted after an expiry date since otherwise the database will overflow.

Quarantine flow

Temporary internal mail queueing

SpamCheetah does support email queueing as a stopgap arrangement in case the target mail server is down for some reason. It also supports alerting the sys admin when the target mail server is found down or unreachable.

Temporary internal mail queueing

The internal mail Queue of SpamCheetah can be viewed from the web interface and as soon as the target mail server is found up, the queued up mails are delivered automatically.

However the spam control features of SpamCheetah other than greylisting will not be available when the target mail server is down.

We hope that such situations are the exception rather than the norm and that the target mail server will be up in all situations.

Other components not discussed here

SpamCheetah being a really complex product with roughly 15 years of development means that a lot of subsystems are not being discussed here, but the UNIX domain sockets and the text messages supported internally are very important for the statistics counters the mail tester feature and many other niceties offered by the product.

Usually commercial vendors are too leery of discussing internal design matters publicly but we as a company believe in transparency and being upfront.

The more we talk about something the more we learn and the more we all benefit as a community.