background

Outbound Spam Monitoring

The filters are very effective at blocking a large percentage of outbound spam/viruses, to prevent issues with your network reputation. It is very important however to pro-actively suspend any spamming customers/accounts, to stop the abuse at its source. If such accounts are not suspended/blocked, eventually there will always be a spam run which our engines could miss. You can prevent any such spam escalations (or other type of attacks from abusive customer accounts), by ensuring the account is locked down before it starts to cause real issues. Our systems allow you to quickly and easily identify such abusive accounts, before any third-party issues occur.There are a number of ways that spammers can be monitored via our systems, the best method depends per-customer

Best practise for smarthost users

Managing Outbound Spam

Control Panel Quarantine (Local Cloud Only)

If you have super administrator access to the control panel,  you can review the blocked outgoing spam emails on the "Spam quarantine" page in the Outgoing section as Super Administrator. Although the number of daily spam emails you find there can be overwhelming at start, simply spending 15 minutes a day to analyze/block the source of the most frequent messages you can find there will quickly result in a significant drop of overall spam traffic. 

This is strongly recommended when starting to use our filtering, so your administrators can easily pinpoint the top spam causes and get more familiar with tracking down/blocking the spam sources.


Outgoing Log search

To view the outbound blocked messages via the outgoing log search you can do the following:

  1. Login to interface
  2. Select the outgoing authenticating domain from the overview
  3. Click outgoing log search
  4. Set "Classification" to "Rejected"
  5. Set status to "Quarantined" (or other statuses if you want to see other non quarantined but rejected messages)
  6. Click search

It is possible to lock senders based on their Identity header directly from this page. To do this you simply

  1. Locate sender with identity you wish to block
  2. Click the lock icon next to the identity

The lock will turn blue, this indicates the identity is now locked. to unlock, simply re-click the lock icon.


Outgoing Reports page

To view senders/Identities in grouped format you can do the following:

  1. Login to cluster
  2. Click "Outgoing Reports"
  3. Add the outbound authenticating domain
  4. Select the period you wish to check
  5. Classification - Rejected (or Accepted if you wish to see accepted emails and not quarantined ones)
  6. Group by - Identity
  7. Click Show

It's possible to lock identities directly from this page. To do this you need to do the following:

  1. To lock the Identity, click the "lock icon" next to the identity. This will then turn blue for locked.
  2. To unlock the identity, click the blue "lock icon".

The lock will turn blue, this indicates the identity is now locked. to unlock, simply re-click the lock icon.


Automatic and Manual Locking

Auto-Locking senders based on the Identity header within the Control Panel

It is possible to auto-lock senders based on their Identity header. For this to work, there must first be a configured Identity. 

To start autolocking senders based on this you need to make sure the option Lock Identities Automatically:" is set to "Yes"  in the outgoing user settings page. 

The Identity will be locked when a certain amount of spam, phishing or virsues are seen in a short time frame. The Locked identities can continue to be seen via the log search and outgoing reports page. 


Manual locking via the Outgoing Log Search and Report page. 

Manual identity locking can be done directly via the log search or outgoing reports page by clicking the lock icon next to the given identity.  To unlock, simply click the blue locked icon. 

As this is a manual lock, this will not auto unlock. 


Manual locking via the Manage Identities page

The manage Identities page gives you a comprehensive overview of the current locked and unlocked identities. To lock or unlock either on mass or individual identities you can do the following:

  1. use the drop down menu next to the Identity
  2. Click lock
  3. Enter a reason for the lock
  4. Click Execute.

To unlock, do the reverse.


Manual Locking senders based on the Identity header via the API

When using IP authentication, it's often needed to be able to lock specific senders without locking the whole IP.  This can be done by  locking senders via their Identification method (Envelope-sender, Authenticating User, Identification Header). To do this you need to execute the following API:

https://APIHOSTNAME/cgi-bin/api?call=api_lock_outgoing_identity&domain=DOMAIN&identity=bob@example.com&username=USERNAME

To list the current locked users:

https://APIHOSTNAME/cgi-bin/api?call=api_list_locked_identities&domain=DOMAIN&username=USERNAME

Please note, to be able to use this method, an Identification header must first be set as mentioned above.


Lock & Unlock user identity script

An example user identity locking script can be found here

python lock_identity.py -h
Usage: lock_identity.py [options] hostname api_username [api_password]

Lock outgoing identities that are sending bad mail.

Options:
  -h, --help            show this help message and exit
  -n NICE, --nice=NICE  'nice' level [default: 10]
  --unattended          run unattended (always answer 'yes')
  -l LIMIT, --limit=LIMIT
                        lock users over this limit [default: 50]
  --hours-ago=HOURS     check behaviour over the last n hours [default: 2]
  -q, --quiet           don't output anything in a successful run.
  --client-username=CLIENT_USERNAME
                        client username for API logging [default:
                        lock_identity]

An example user identity unlocking script can be found here

python unlock_identity.py -h
Usage: unlock_identity.py [options] hostname api_username [api_password]

Review and unlock outgoing identities.

Options:
  -h, --help            show this help message and exit
  -n NICE, --nice=NICE  'nice' level
  -s SEARCH, --search=SEARCH
                        Match only these identities
  --client-username=CLIENT_USERNAME
                        client username for API logging [default:
                        lock_identity]

The following example, shows how this can be run manually to check and block identities that have had 25 rejected messages in the last 1 hour:

~ % python lock_identity.py master.hostname.tld apiusername apipassword --hours-ago=1 -l 25 XXX spam/virus/phish messages were sent by users with no identity. bob@example.com (10.0.0.1@smtp.example.com): sent 29 bad messages - Do you wish to lock this user? y

To run this automatically via a CronJob for example, you can do the following:

python lock_identity.py master.hostname.tld apiusername apipassword --hours-ago=1 -l 25 --unattended -q

This will answer yes automatically to the locking question and not output any results.


Alternative reporting

Outbound Spam Reports via CSV

There is  an option to have a daily CSV report for outbound spam per outgoing user:The API call to activate the daily report with spamming accounts is:api_set_outgoing_report_recipient(domain, recipient='', username='') -> "".Set the address where the outgoing filter report should be sent to. If the 'recipient' argument is omitted then disable this feature. Please note that this feature is in development and the format and content of the report are subject to change without our usual deprecation procedures.

https://MASTERHOSTNAME/cgi-bin/api?call=api_set_outgoing_report_recipient&domain=DOMAIN&recipient=RECIPIENT&username=USERNAME" 

So this should be good for you in regards to the monitoring of outbound spam, and be able to see overall information. (please note , subject and body will not be shown here)

Alternative Outbound Spam Reports via CSV based on "Identification header"

There is another option to have a  CSV report for outbound spam per identification header sender that is sent every 2 hours to a specified email address. The API call to activate the daily report with spamming accounts is:api_set_outgoing_report2_recipient(domain, recipient='', username='') -> "".Set the address where the outgoing filter report should be sent to. If the 'recipient' argument is omitted then disable this feature. Please note that this feature is in development and the format and content of the report are subject to change without our usual deprecation procedures.

https://master.hostname/cgi-bin/api?call=api_set_outgoing_report2_recipient&domain=DOMAIN&username=USERNAME&recipient=RECIPEINT

The report will contain counts of blocked spam per Identification and counts of invalid senders.  For example:

"Authentication Domain","Authentication User","User Identification","Spam Count","Invalid Sender Count"
example.com,,bob@example.com,100,0
example.net,,example.net,235,301
example.org,,example.org,0,2000

IMAP quarantine access

Rather than using any of the scripts, or the Spampanel webinterface, you can simply authenticate with your "global" administrator account (Local Cloud only) using any IMAP compatible email client for real-time access to the spam quarantine. Please contact our support in case you do not have the "global" credentials yet.


Global quarantine reporting script

Rather than using Spampanel or direct IMAP quarantine access to review the quarantine, this is a simple script that will parse the outbound IMAP quarantine. (Local Cloud Only)You can download this here:http://download.seinternal.com/tools/retrieve_quarantine_info.py

Usage: retrieve_quarantine_info.py [options]
 
 Output a list of quarantined outgoing messages.
 
 Options:
 -h, --help show this help message and exit
 -c, --csv saves output to csv file
 -d, --display displays the loglines as they pass by
 -i, --incoming search the incoming quarantine
 -o, --outgoing search the outgoing quarantine
 -t, --today load results from today (otherwise yesterday)
 -s IMAPHOST, --imaphost=IMAPHOST
 The hostname of the imap server
 -u IMAPUSER, --username=IMAPUSER
 The username to check, usually 'global'
 -p IMAPPASS, --password=IMAPPASS
 The password for the 'global' user
 -n, --no-bounce filters out mail originating from 'mailer-daemon'

Please make sure you run this from a NON FILTERING server only. This will retrieve the quarantined messages either in an on screen format or saved to a .csv file. For example you could do something like this:$./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAMEThis will display on screen the messages that have been quarantined outbound in the last hour. You will be prompted for a "Global" password.  This is given out only on request. If this is required please contact support@spamexperts.com for more details.This will show you something like this:

$./retrieve_quarantine_info.py -d -o -n -s MASTERHOSTNAME (or quarantine server if applicable)
 Please enter the password for the IMAP account 'global'.
 Password:
 #,From,To,Reply-To,Qmail UID,Invoked for,IP/Username,Evidence,PHP script,Auth.sender,Auth-User,Auth-Email
 1,--DATA WILL BE HERE--
 2,--DATA WILL BE HERE--
 3,--DATA WILL BE HERE--
 4,--DATA WILL BE HERE--
 5,--DATA WILL BE HERE--

Here you can see the information on the blocked messages and some relevant details.Alternatively it may be easier for you not to display the data and save it to a csv file, then you can open it in any excel like program and sort it on the specific field to group the data.This should then give you a better idea of some of the clients that you can close down for spamming.


Using the Log Search API

It's possible to use the api_find_outgoing_messages to be able to get for example a list of the top 50 spammers in X amount of time. A simple bash example can be seen below

curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&classification=oversize%2Cblacklisted%2Clocked%2Cphish%2Cvirus%2Cspam%2Cdeferred&include_in_progress=False&id=&subject_header=&api_language=en&columns=sender" | sort | uniq -c | sort -nr | head -50

It's also then possible to start automating actions. For example, you can use 2 API's 'api_find_outgoing_messages' and 'api_blacklist_outgoing_sender' to be able to query the log API to find senders that have sent X amount of messages in X amount of time, and then take a further action. The example below shows a simple way to check the outgoing logs for the last 24 hours, and if the sender has sent more than 2000 messages then blacklist the sender.

curl -s -k "https://user:pass@master.hostname/cgi-bin/api?call=api_find_outgoing_messages&domain=DOMAIN&from_date=`date -d '12 hours ago' +'%s'`&to_date=`date +%s`&predicate=and&partial=False&sort_field=datestamp&include_in_progress=False&columns=sender" | sort | uniq -c | grep "@" | sort | awk '{if($1==$1+0 && $1>2000)print $2}' | xargs -I{} curl -k -s "https://user:pass@master.hostname/cgi-bin/api?call=api_blacklist_outgoing_sender&domain=DOMAIN&sender={}'

While these are very basic examples, our API is very versatile, so these can be amended or changed to suit your exact needs, for example, by locking specific senders based on the identification headers or more.

ARF reports

A report that will be sent each time an outgoing spam message is blocked, and will contain a copy of the original message including headers.Information on this and how to set up the ARF reports can be found here.

Many larger companies already process such ARF reports originating from external sources such as AOL. You can simply set your administrator address to point to your existing ARF parsing infrastructure, so your existing abuse handling systems automatically receive and process our datafeeds.


ARF parser

If you do not have an ARF parser yet, we definitely recommend to setup a system to handle your incoming ARF reports. We can recommend the free opensource software Abuse.IO for this. Alternatively you can e.g. use a simple python file that can parse the contents of the ARF reports. Your sysadmins will know how best they can utilize this and parse the data that they need. This can be found here: http://download.spambrand.com/arf_parser.py

Using ARF automation also allows you to accept ARF feed from third-parties, to further improve your abuse handling and to deal with abuse that does not (yet) use our outgoing filter.

Mass outbound message removal from outbound delivery queue

Often it's needed to mass remove messages from the outgoing delivery queue (for example when ARF reports). This can already be done using the Interface and API, however we also have an open source script that can help facilitate this. This can be downloaded here. This script must NOT be run from a SpamExperts filtering server, and should be run externally using the API credentialsUsage:

python queue_remove_emails.py

Please specify the API server hostname and your API username!

Usage: queue_remove_emails.py [options]

Options:
  -h, --help            show this help message and exit
  -f SENDER, --from=SENDER
                        find and remove messages from the specified sender
  -t RECIPIENT, --to=RECIPIENT
                        find and remove messages from the specified recipient
  -s SERVER, --server=SERVER
                        API server hostname
  -u USERNAME, --username=USERNAME
                        API username

An example, would be:

python queue_remove_emails.py -f bob@example.com -s master.hostname -u admin

Blacklisting outbound senders via the API

To be able to blacklist an outgoing sender, you need to use the API :

https://APIHOSTNAME/cgi-bin/api?call=api_blacklist_outgoing_sender&domain=DOMAIN&sender=SENDER&username=USERNAME

To list the current blacklisted senders:

https://APIHOSTNAME/cgi-bin/api?call=api_get_outgoing_sender_blacklist&domain=DOMAIN&username=USERNAME

Please note, to be able to use this method, you will always need to set the domain and username. This should be the outgoing authenticating user.


Was this article helpful?

Related articles

Search result for :