HSL code examples

From Halon, SMTP software for hosting providers
Revision as of 08:31, 27 November 2015 by Erik (talk | contribs) (Array intersect)
Jump to: navigation, search

Below is a smorgasbord with some script examples you may find useful for your IP Policy (connection level) and Mail Content flows.

Flow

These scripts may be used in all flows.

Notification limit

On certain events it may be useful to eg. send a mail notification, however if these messages aren't rate-limited you may end up with huge amounts of notifications. Here are two ways of limiting notifications by time, non of which is better but serves different purposes.

if (true) {
     
// this example sends one mail per hour for every unique function arguments.
     
cache "ttl" => 3600 ]
          
mail("""[email protected]""Event occurred""An event occurred"); 
if (true) {
     
// this example sends one mail per hour (based on the event1 rate).
     
if (rate("notification""event1"13600))
          
mail("""[email protected]""Event occurred""An event occurred"); 

Array sorting and shuffling

Two example functions how to do non-recursive sorting and shuffling of an array.

// Insert sort (O(n^2))
function array_sort($l)
{
    foreach (
range(1count($l) - 1) as $i)
    {
        
$k $l[$i];
        
$j $i 1;
        foreach (
array_reverse(range(-1$j)) as $j)
        {
            if (
$j or $l[$j] <= $k)
                break;
            
$l[$j 1] = $l[$j];
        }
        
$l[$j 1] = $k;
    }
    return 
$l;
}

// Knuth shuffle (O(n))
function array_shuffle($l)
{
    foreach (
range(1count($l) - 1) as $i)
    {
        
$r rand(0$i);
        if (
$r != $i) {
            
$t $l[$r];
            
$l[$r] = $l[$i];
            
$l[$i] = $t;
        }
    }
    return 
$l;
}

echo 
array_sort(array_shuffle([01234])); 

Array intersect

Find and return elements which exists in both arrays ($list1 and $list2).

function array_intersect($list1$list2)
{
    
$list3 = [];
    foreach (
$list1 as $k => $i) {
        if (
in_array(string($i), $list2))
            
$list3[$k] = $i;
    }
    return 
$list3;
}

$r1 array_intersect(["1","2","3","4"], ["2","3"]);

// new method using lambda functions

$r2 array_filter(function ($v) { return in_array($v, ["2","3"]); }, ["1","2","3","4"]); 

Clustered lookup cache

It's possible to query the rate function for the number of entries (implementing a lookup cache), which could be used to implement eg. a SASL authentication cache shared in a cluster.

$ttl 3600;
if (
rate("sasl-cache"$saslusername ":" sha1($saslpassword), 0$ttl))
    
Accept();
if (
$saslusername == "username" and $saslpassword == "secret") {
    
rate("sasl-cache"$saslusername ":" sha1($saslpassword), 1$ttl);
    
Accept();

IP flow

These scripts may be used in IP flows.

Black and white list

// White list: check if $senderip is in any of the networks..
$network = ["10.0.0.0/8""192.168.0.0/16"];
foreach (
$network as $net) {
 if (
in_network($senderip$net)) {
  
Allow();
 }
}

// Black list: check if $senderip is in any of the networks..
$network = ["172.16.0.0/12"];
foreach (
$network as $net) {
 if (
in_network($senderip$net)) {
  
Block();
 }

DNSBL

if (count(dnsbl($senderip,"zen.spamhaus.org")))
   
Block("$senderip blocked by Spamhaus");
if (
count(dnsbl($senderip,"bl.spamcop.net")))
   
Block("$senderip blocked by SpamCop"); 

GeoIP

It's possible to parse the MaxMind GeoLite CSV database using HSL and implement a lookup function. Download the GeoLite database GeoIPCountryCSV.zip, extract and upload the GeoIPCountryWhois.csv file to the FTP.

function ip2long($ip)
{
    
$ip explode("."$ip);
    return (
number($ip[0]) * 256 ** 3) +
           (
number($ip[1]) * 256 ** 2) +
           (
number($ip[2]) * 256 ** 1) +
           (
number($ip[3]) * 256 ** 0);
}

function 
geoip_object()
{
    
$object file("file://GeoIPCountryWhois.csv");
    foreach (
$object as $k => $x) {
        
$x explode(","$x);
        
$object[$k] = [number($x[2][1:-1]), number($x[3][1:-1]), $x[5][1:-1]];
    }
    return 
$object;
}

function 
geoip($senderip)
{
    
$ip ip2long($senderip);
    
$lst cache [] geoip_object();
    
$min 0;
    
$max count($lst);
    
$log log($max2);
    foreach (
range(0$log) as $i) {
        
$mid floor(($min $max) / 2);
        if (
$ip $lst[$mid][1]) {
            
$min $mid;
            continue;
        }
        if (
$ip $lst[$mid][0]) {
            
$max $mid;
            continue;
        }
        return 
$lst[$mid][2];
    }
    return 
"";

RCPT TO flow

These examples can be used in the flows executing MAIL FROM/RCPT TO combinations.

External black/whitelist using API

This example shows how to implement a black and whitelist as a web service in PHP and use that data in the SP as a black/whitelist passing the data using JSON.

bwlist.php:

<?php
 $blacklist 
= [];
 
$whitelist = ["[email protected]""example.org"];
 echo 
json_encode(["blacklist" => $blacklist"whitelist" => $whitelist]); 

json data:

{"blacklist":[], "whitelist":["[email protected]", "example.org"]}

rcptflow:

$list json_decode(http("http://api.example.com/bwlist.php", ["timeout" => 10])); // TODO: use cache

if (is_array($list) and is_array($list["whitelist"])) {
    foreach (
$list["whitelist"] as $item) {
        if (
$senderip == $itemAccept();
        if (
$senderdomain == $itemAccept();
        if (
$sender == $itemAccept();
    }
}

if (
is_array($list) and is_array($list["blacklist"])) {
    foreach (
$list["blacklist"] as $item) {
        if (
$senderip == $itemReject("Blacklisted");
        if (
$senderdomain == $itemReject("Blacklisted");
        if (
$sender == $itemReject("Blacklisted");
    }

Greylisting

Greylisting is a technique to prevent custom/non-compliant spam software/bots to send messages on the first try, thus requiring a server implementations with proper queuing.

function greylist($triplet$time) {
    return 
$time;
}

// $sendernet (network/24) can be replaced with $senderip
$sendernet implode("."explode("."$senderip)[0:3]);
$triplet "$sendernet:$sender:$recipient";

// time recommendations from http://en.wikipedia.org/wiki/Greylisting
$windowopen 24;                   // Greylist time (24 minutes)
$windowclose 60 4;              // Greylist time end (4 hours)
$windowtime 60 24 14;         // Skip greylist time (14 days)
    
$slot cache ["size" => 1000000,
        
"ttl" => ($windowclose 60),
        
"argv_filter" => [1] ]
            
greylist(md5($triplet), uptime() + ($windowopen 60));
    
if (
$slot == 0)
    echo 
"Greylist, ok";
else if (
$slot <= uptime()) {
    echo 
"Greylist, pass";
    
cache ["size" => 1000000,
        
"ttl" => ($windowtime 60),
        
"argv_filter" => [1],
        
"force" => true]
            
greylist(md5($triplet), 0);
} else
    
Defer("Greylisted, please try again in ".round($slot-uptime(), 0)." seconds"); 

Outbound rate control based on failed deliveries

One indication of outbound messages being spam, is that other server reject them. Put the following code in the post-delivery (transport) flow

if ($transportid == "mailtransport:2"// whichever is the outbound (lookup-mx) transport
   
if ($errorcode >= 500)
      
rate("delivery-fail"$sender10000003600); 

and the following in the outbound RCPT TO flow:

if (rate("delivery-fail"$sender03600) > 100)
   
Defer("You have more than 100 failed deliveries the last hour, try again later"); 

Concurrency control for smtp_lookup_rcpt

function smtp_lookup_rcpt(...$argv) {
    
barrier "smtp_lookup_rcpt" => $var {
        if (!isset(
$var)) {
            
$var = [];
        }
        if (
is_string($argv[0]))
            
$unique $argv[0];
        else
            
$unique $argv[0]["host"];
        if (!isset(
$var[$unique])) {
            
$var[$unique] = 0;
        }
        if (
$var[$unique] >= 3) {
            if (
$argv[3]["error_code"] == true) {
                return [
                    
"error_code" => 400,
                    
"error_message" => "Too many concurrent lookups"
                
];
            }
            return -
1;
        }
        
$var[$unique] += 1;
    }
    
$result builtin smtp_lookup_rcpt(...$argv);
    
barrier "smtp_lookup_rcpt" => $var {
        
$var[$unique] -= 1;
    }
    return 
$result;

DATA flow

These scripts may be used in DATA flows; processing the actual mail message.

Filter anti-virus engine results

This example filter various heuristics detection from the ClamAV antivirus engine, which may be useful if these apply to you on regular basis preventing unwanted false-positives.

// ClamAV check
$skip = ["Heuristics.Safebrowsing.Suspected-phishing_safebrowsing.clamav.net"];
$clam = [];
foreach (
ScanCLAM() as $v)
   if (!
in_array($v$skip))
       
$clam[] = $v;
if (
count($clam))
   
Reject("Virus found by CLAM"); 

Distribution list

This example shows how to create a simple mail distribution list with JSON API integration. Add a custom/fictional domain (eg. notify). Add a custom flow

<?php
if ($_GET['key'] === 'secretkey') {
    
$mail = array();
    
$mail[] = '[email protected]'// fetch from database
    
$mail[] = '[email protected]'// fetch from database
    
die(json_encode($mail));
if ($senderip == "192.168.0.2" and $sender == "[email protected]") {
    
$mail json_decode(http("https://api.example.com/notifications/?key=secretkey"));
    
SetSender("[email protected]");
    
SetDelayedDeliver(300); // "whops"-protection
    
foreach ($mail as $e) {
        
DiscardMailDataChanges();
        
DelHeader("Received");
        
SetHeader("From""Example company <[email protected]>");
        
SetHeader("To"string($e));
        
CopyMail(string($e));
    }
    
Reject("Message sent to ".count($mail)." users!");
}
Reject("Not allowed"); 

Prevent getting blacklisted by using a dedicated "spam" source IP

  1. Add an extra IP address (for each cluster node)
  2. Add an outbound (lookup-mx) transport with that source address
  3. Add the script below to the outbound DATA flow
if (ScanRPD() > or ScanSA() > 2)
    
SetMailTransport(“mailtransport:X”); // with X being the transport added above 

DSN bypass (rate)

Deliver (possible without scanning) DSN messages (in this example, if receiving more than 10 DSN / 5 minutes, only do a Deliver). Good if you do regularly large send out and are expecting bounces.

if ($sender == "" and rate("dsn-bypass"$recipient10300) == false) {
    echo 
"DSN bypass";
    
Deliver();

Manually review high-volume email

Place this script in the very end of the DATA flow, to delay the 20th+ delivery (during one hour) of messages with identical subject lines. These can be listed on the Activity > Rate limit page, and reviewed by searching for them using the filter

rawsubject~"TEXT_FROM_RATE_PAGE"

on the Activity > Tracking page. In some occasions, this can be spam, and thus deleted.

function is_news($senderdomain) {
    
$news = [
        
"epmail.se",
        
"anpdm.com",
        
"carmamail.com",
        
"exacttarget.com",
        
"anppub1.com",
        
"clickredirect.com"
    
];
    foreach (
$news as $domain)
        if (
is_subdomain($senderdomain$domain))
            return 
true;
    return 
false;
}

$subject GetHeader("subject"false);
if (
$subject and
    !
cache ["ttl" => 3600"size" => 20000is_news($senderdomain) and 
    
rate("delayed-review"$subject[0:40], 20003600) and
    
rate("delayed-review"$subject[0:40], 03600) > 20) {
    if (
rate("delayed-review-report""junk"13600))
        
mail("""[email protected]""Review delayed messages""Please do it");
    
SetDelayedDeliver(3600);

Check total process time of a message

executiontime() only tells the current recipients process time, to check total process time of a message (for all recipients). The following script may be used.

$beginprocess cache ["per_message" => trueuptime();
// do processing...
echo (uptime() - $beginprocess) . " total process time"

Check if sender domain exists

Check if Sender Domain exists; if not delete the mail.

if ($senderdomain != "" and !dns($senderdomain) and !dnsmx($senderdomain))
    
Reject("$senderdomain doesn't exist"); 

Dynamic routing to several mail servers

This will lookup, cache and deliver messages to the correct server dynamically. It's very important that you control all those servers, so that your users cannot re-route traffic themselves by adding a domain.

$servers = ["mailtransport:1""mailtransport:2"];
function 
choose_server($domain$email) {
    foreach (
$servers as $t)
        if (
smtp_lookup_rcpt($t""$email) == 1)
            return 
$t;
    return 
false;
}

$server cache ["argv_filter" => [1], "size" => 1000"ttl" => 86400choose_server($recipientdomain$recipient);

if (
$server == false)
    
Reject("Unknown user");
SetMailTransport($server); 

Use a magic keyword for attachment passthrough

This script will block messages with .zip attachments if they do not have the keyword "sendzip" in the subject.

// This is the magic keyword aka. passphrase
$magickeyword "sendzip";

// Test if magickeyword is not found anywhere in the subject
if (!(GetHeader("Subject") =~ $magickeyword))
{
 
// Find all attachments of type zip
 
if (count($illegalattachments GetAttachmentsByName("\\.zip$")))
 {
  
// Remove the attachment, tag the subject and wrap the message...
  
RemoveAttachments($illegalattachments);
  
WrapMessage("[Attachments Removed] "+GetHeader("Subject"),
    
"The following attachments were removed, if you want to obtain these; ".
    
"please, ask the sender to resend them and type $magickeyword in the ".
    
"subject of the message.<br />");

  
// Print out the file names
  
foreach (GetAttachmentName($illegalattachments) as $name)
  {
   
WrapMessage(""$name);
  }
 }

White list sender domain

This is how a white list could be written. Below follows two examples, the first using string compare and second using regular expressions.

$whitelist = ["halonsecurity.com""halon.se""example.org"];
if (
in_array($senderdomain$whitelist))
   
Deliver(); 

Example using regular expressions against $sender.

$whitelist = ["@halonsecurity\\.com$""@halon\\.se$""@example\\.org$"];
foreach (
$whitelist as $host)
   if (
$sender =~ $host)
      
Deliver(); 

Be aware of that you have to escape the dots in the host with \\., or else the dot will match any character.

Add headers

AddHeader("X-Halon-ID"$messageid);
AddHeader("X-Halon-RPD"ScanRPD());
AddHeader("X-Halon-RefID"ScanRPD(["refid"=>true])); 

Limit number of recipient addresses allowed per domain

We generally recommend using the http() function to query external data sources when implementing business logic such as this, but in case you want to design a pure HSL script which limits the number of users to scan on a domain basis, the following script should work.

$maxusers in_file($recipientdomain"file:1");
if (
$maxusers)
    
$maxusers $maxusers[1];
else
    
$maxusers 5;
echo 
$maxusers;

barrier "user-count" => $var {
    if (!isset(
$var))
        
$var = [];
    if (!isset(
$var[$recipientdomain]))
        
$var[$recipientdomain] = [];
    
$mbox explode("@"$recipient)[0];
    echo 
$mbox;
    if (!
in_array($mbox$var[$recipientdomain]))
        
$var[$recipientdomain][] = $mbox;
    if (
count($var[$recipientdomain]) > number($maxusers))
        echo 
"Exceeding user count";
    echo 
$var;

Support/collaboration script

This script makes sure support or sales addresses always reaches everyone in the distribution group.

// Inbound
function DoInCC($address) {
    global 
$recipient;
    global 
$messageid;
    if (
$recipient == $address and GetHeader("References") == "")
        
SetHeader("References""<$messageid-cc-$address>");
}
DoInCC("[email protected]");
DoInCC("[email protected]");

// Outbound
function DoOutCC($address$name$transport) {
    if (
GetHeader("References") =~ "-cc-$address>") {
        
SetHeader("from""$name  <$address>");
        
SetSender($address);
           
CopyMail($address$transport);
    }
}
DoOutCC("[email protected]""Example Sales""mailtransport:X");
DoOutCC("[email protected]""Example Sales""mailtransport:X"); 

Verify sender "HELO" hostname

This example will check if the senders HELO message does resolve back to his IP.

if (!in_array($senderipdns($senderhelo)))
   echo 
"Provided HELO message does not resolve to sender IP, this is suspicious"

Log all mail messages sent out not during office hours

Since you don't expect anyone to send out mail during the night, you may want to explicit log all sent mail. This of course only makes sense when using the flow for outgoing traffic.

// Code executed between 17:00 and 6:59
$time number(strftime("%H"));
if (
$time 16 or $time 7)
    echo 
"$senderip tried to send a mail to $recipient (from $sender)"

Scan message, delete viruses and encapsulate message

This example will scan all MIME parts having a name, that said it will not scan the entire message only files appearing as attachments.

foreach (GetAttachmentsByName(".+") as $id) {
 
$virus ScanKAV($id);
 if (
count($virus)) {
  
RemoveAttachments($id);
  
WrapMessage("Virus Alert""Virus " $virus " found in attachment " GetAttachmentName(string($id)));
 }

This example will scan the message and remove and notify is a virus was found.

$virus ScanKAV();
if (
count($virus)) {
 
RemoveAttachments();
 
WrapMessage("Virus Alert""Virus " $virus " found in message");
 
Deliver();

DSN Spam Protection

Delete the message if it is a DSN (delivery status notification) but not sent directly from your outgoing mail server (*.example.com).

$dsn GetDSN();
if (
$dsn["route"][1] =~ "\\.example\\.com") {
  echo 
"DSN Spam; Deleted Message from $sender to $recipient";
  
Delete();

Bitwise operators

HSL doesn't have explicit bitwise operators, instead they can be implemented using these custom functions. The functions operates on non-negative 32bit integers.

AND

function bitand($a$b) {
    
$result 0;
    foreach (
range(031) as $x) {
        if (
$a and $b 2)
            
$result += ** $x;
        
$a floor($a 2);
        
$b floor($b 2);
    }
    return 
$result;

OR

function bitor($a$b) {
    
$result 0;
    foreach (
range(031) as $x) {
        if (
$a or $b 2)
            
$result += ** $x;
        
$a floor($a 2);
        
$b floor($b 2);
    }
    return 
$result;

XOR

function bitxor($a$b) {
    
$result 0;
    foreach (
range(031) as $x) {
        if (
$a $b == 1)
            
$result += ** $x;
        
$a floor($a 2);
        
$b floor($b 2);
    }
    return 
$result;

NOT

function bitnot($a) {
    
$result 0;
    foreach (
range(031) as $x) {
        if (
$a == 0)
            
$result += ** $x;
        
$a floor($a 2);
    }
    return 
$result;