Community Forum

Resolved
0 votes
This how to will configure a public folder for spam so that Zarafa users can "train" the SpamAssassin spam filter to be more accurate. This folder is accessible via the Webmail or Outlook. I assume you already have Zarafa installed and running as per the forum thread - you will also need the Zarafa-Gateway service running

Big thanks to the SME thread here, the steps below are derived from it
http://wiki.contribs.org/Zarafa_Bayesian_Learning
[ol]
  • Configure ClearOS LDAP to use Zarafa LDAP schema
    cp /usr/share/zarafa/zarafa.schema /etc/openldap/schema/
    vi /etc/openldap/slapd.conf
    ## add the following line after the top section
    include /etc/openldap/schema/zarafa.schema
    ## restart LDAP
    service ldap restart


  • Modify webconfig php so that it recognises the Zarafa LDAP parameters
    Edit /var/webconfig/api/User.class.php, add the "zarafa-user" to the core class list. Around line 210:
    $this->coreclasses = array(
    'top',
    'posixAccount',
    'shadowAccount',
    'inetOrgPerson',
    'kolabInetOrgPerson',
    'hordePerson',
    'pcnAccount',
    'zarafa-user',
    );


  • Create the Zarafa Public store (if you haven't already)
    zarafa-admin -s


  • Create a user account via the webconfig with mail permissions only specifically for collecting the mail. I've used 'SpamAdmin' account below


  • Use LdapAdmin or similar tool to assign a user Zarafa admin account, and add zarafa-user to object class. See screenshots below.
    a. First set your LDAP publish policy to enabled within the webconfig
    b. Use LdapAdmin to login using your LDAP credentials Bind DN, Base, and Bind Password
    c. Right click on your user, and add the Zarafa-user object class to the drop down list on the left hand side
    d. Then change the Zarafa-Admin attribute to 1
    http://timburgess.net/files/ldapadmin-zarafa.jpg


  • Create ham/spam folders and assign permissions
    Login to Zarafa with an account that has admin rights as assigned above and make two new folders LearnAsSpam and LearnAsHam under: Public folder > Public folders. Set the permissions (right-click folder > Properties > Permission-tab) on both these new folders to:

    Spam administration account
    * Folder visible
    * Read items
    * Edit items: all
    * Delete items: all

    Everyone (and/or other users/groups you've added at least need:)
    * Folder visible
    * Create items
    * Edit items: none
    * Delete items: none


  • Create a script that will login via IMAP check the folders periodically and pull mail into the ClearOS spam training system. Note that you will need the Zarafa Gateway service running for IMAP access. You can run this on a seperate port if you are still using the ClearOS IMAP. I've assumed that Zarafa Gateway is listening on port 143.

    Create /usr/bin/Zarafa-sa-learn.pl
    #!/usr/bin/perl
    #
    # Extract mail from imap server shared folder 'Public folders/LearnAsSpam' & 'Public folders/LearnAsHam'
    # Orig by dmz@dmzs.com - March 19, 2004
    # http://www.dmzs.com/tools/files/spam.phtml
    # Modified for compatibility with ClearOS spam training Jan 14, 2011, spam training handled by external script
    # LGPL

    use Mail::IMAPClient;

    my $debug=0;
    my $salearn;

    my $filterdir="/var/spool/filter/training";
    my $spamwebclient="$filterdir/spam-web";
    my $notspamwebclient="$filterdir/notspam-web";

    # # # # # # # # # # EDIT USER AND PASSWORD (CHECK PORT) # # # # # # # # # #
    my $imap = Mail::IMAPClient->new( Server=> '127.0.0.1:143',
    User => 'spamadmin',
    Password => 'yourpasswordhere',
    Debug => $debug);

    if (!defined($imap)) { die "IMAP Login Failed"; }

    # If debugging, print out the total counts for each mailbox
    if ($debug) {
    my $spamcount = $imap->message_count('Public folders/LearnAsSpam');
    print $spamcount, " Spam to process\n";

    my $nonspamcount = $imap->message_count('Public folders/LearnAsHam');
    print $nonspamcount, " Notspam to process\n" if $debug;
    }

    # Process the spam mailbox
    $imap->select('Public Folders/LearnAsSpam');
    my @msgs = $imap->search("ALL");
    for (my $i=0;$i <= $#msgs; $i++)
    {
    # export message to file
    $imap->message_to_file($spamwebclient.".zarafa.".time.".".$i,$msgs[$i]);

    # delete processed message
    $imap->delete_message($msgs[$i]);
    open("placeholder",'>'.$spamwebclient) or die "Can't create placeholder $!";
    close("placeholder");
    }

    $imap->expunge();
    $imap->close();

    # Process the not-spam mailbox
    $imap->select('Public Folders/LearnAsHam');
    my @msgs = $imap->search("ALL");
    for (my $i=0;$i <= $#msgs; $i++)
    {
    $imap->message_to_file($notspamwebclient.".zarafa.".time.".".$i,$msgs[$i]);

    # delete processed message
    $imap->delete_message($msgs[$i]);
    open("placeholder",'>'.$notspamwebclient) or die "Can't create placeholder $!";
    close("placeholder");
    }
    $imap->expunge();
    $imap->close();

    $imap->logout();

    Make sure it's executable
    chmod a+x /usr/bin/Zarafa-sa-learn.pl


  • Create cron script to regularly pull mail from the Public folder into the Spam training system
    Create /etc/cron.d/zarafa-spamassassin
    # Extract mail from spam folders for training every 10minutes from 8:00 to 22:00 during weekdays
    */10 8-22 * * 1-5 root /usr/bin/Zarafa-sa-learn.pl


  • Test! you should now be able to login via webmail or Outlook, and drag (move) mail that isnt' being correctly recognised as SPAM into the LearnAsSpam folder and ClearOS will 'auto-learn' it. The same applies for HAM (mail incorreclty marked as SPAM). Note that mail is deleted after it is moved into the Public Folders so keep a copy if you need it.

  • [/ol]
    Enjoy!
    Monday, January 17 2011, 10:20 AM
    Share this post:
    Responses (33)
    • Accepted Answer

      Al Catoe
      Al Catoe
      Offline
      Wednesday, August 06 2014, 03:12 PM - #Permalink
      Resolved
      0 votes
      Never mind.... guess the instructions need the following step:

      Create the folder /var/spool/filter/training

      Sure hope this helps with all the spam I am dealing with.

      Thanks for your help Tim!
      The reply is currently minimized Show
    • Accepted Answer

      Al Catoe
      Al Catoe
      Offline
      Wednesday, August 06 2014, 02:31 PM - #Permalink
      Resolved
      0 votes
      Thanks Tim.

      I am trying to figure out why I am getting this error when the cron job runs:
      binmode() on closed filehandle $fh at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 926.
      print() on closed filehandle $fh at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 1736.
      print() on closed filehandle $fh at /usr/share/perl5/vendor_perl/Mail/IMAPClient.pm line 1767.
      Can't create placeholder No such file or directory at /usr/bin/Zarafa-sa-learn.pl line 45.


      Any advice?

      Thanks,
      Al
      The reply is currently minimized Show
    • Accepted Answer

      Wednesday, August 06 2014, 01:50 PM - #Permalink
      Resolved
      0 votes
      Hi Al, this script still works with ClearOS 6.5, and it got simpler :)

      Just skip the bit about adding the Zarafa LDAP schema, and the webconfig API changes (steps 1 and 2) as these are incorporated already
      The reply is currently minimized Show
    • Accepted Answer

      Al Catoe
      Al Catoe
      Offline
      Wednesday, August 06 2014, 01:44 PM - #Permalink
      Resolved
      0 votes
      How do you modify webconfig php so that it recognises the Zarafa LDAP parameters as instructed by "Edit /var/webconfig/api/User.class.php, add the "zarafa-user" to the core class list. Around line 210:"? I know this path is legacy now, but how can you do this in 6.5 with Zarafa 7.1.7?

      Thanks!
      The reply is currently minimized Show
    • Accepted Answer

      ingkram
      ingkram
      Offline
      Saturday, January 25 2014, 01:34 AM - #Permalink
      Resolved
      0 votes
      I have waiting this fiture too for Zarafa ClearOS 6x, because there are some email in the junk folder already and should be marking as not junk (as not spam)

      I see it have been add to the roadmap and confirmed
      " 0000662: [app-zarafa - Zarafa Engine] Add spam training from timb80 (pbaldwin) - confirmed."

      Thanks,
      Ingkram
      The reply is currently minimized Show
    • Accepted Answer

      bsleys
      bsleys
      Offline
      Friday, September 27 2013, 05:14 PM - #Permalink
      Resolved
      0 votes
      I added to following to the bottom of the script in the first post to force sa-learn to run after the mail has been exported. I don't know if its 100% needed but I'll see if spam gets flagged now or not.

      print `sa-learn --no-sync --spam ${spamwebclient}.".zarafa.*"`;
      unlink glob $spamwebclient.".zarafa.*";
      print `sa-learn --no-sync --ham ${notspamwebclient}.".zarafa.*"`;
      unlink glob $notspamwebclient.".zarafa.*";
      print `sa-learn --sync`;
      print `sa-learn --dump magic`;

      Bob
      The reply is currently minimized Show
    • Accepted Answer

      bsleys
      bsleys
      Offline
      Friday, September 27 2013, 04:00 PM - #Permalink
      Resolved
      0 votes
      I got this all working a couple of weeks a go but users aren't noticing any reduction in spam. If I look at mail headers I see that spam assassin is running but it doesn't appear that it's "learning". How can I confirm that the main being dumped out to /var/spool/filter/training is actually being used for training by assassin?

      Bob
      The reply is currently minimized Show
    • Accepted Answer

      bsleys
      bsleys
      Offline
      Friday, August 30 2013, 06:23 PM - #Permalink
      Resolved
      0 votes
      I've been trying to get this working on 6.4 but I'm stuck where the IMAP login of the script isn't working. I get the following error if I turn debug on.

      /Zarafa-sa-learn.pl
      Started at Fri Aug 30 14:18:22 2013
      Using Mail::IMAPClient version 3.33 on perl 5.010001
      Connecting via IO::Socket::INET to localhost:143:143 Timeout 600
      Connected to localhost:143
      Read: * OK [CAPABILITY IMAP4rev1 LITERAL+ STARTTLS AUTH=PLAIN] Zarafa IMAP gateway ready
      Sending: 1 LOGIN bsleys *********
      Sent 24 bytes
      Read: 1 NO LOGIN imap feature disabled

      I've tried using the spamadmin account and my own personal account for testing. I know both accounts work in Zarafa webmail but don't know why it thinks imap is disabled.

      Any hints would be appreciated.

      Thanks

      Bob
      The reply is currently minimized Show
    • Accepted Answer

      Monday, May 06 2013, 05:07 AM - #Permalink
      Resolved
      0 votes
      The Skript works with 6.4!

      |quote] I just had to create the folder /var/spool/filter/training...and find the old filter training script. I'm can't work out why the script you are running is using a file at ./tmp.pl? are you running the right script?[/quote]

      Thanks Team,
      that was the only problem: Just make a directory /var/spool/filter/training. Now it works.

      And sorry, tmp.pl was just the name for your skript when I tried out again-just for my post to see the errormessage.

      Greetings Alex
      The reply is currently minimized Show
    • Accepted Answer

      Sunday, May 05 2013, 08:06 PM - #Permalink
      Resolved
      0 votes
      Hi Alex, thanks for the feedback - I did check this out for 6.3 but never got round to updating the script or how to. My modified script is tweaked so that it's works with existing ClearOS filtering scripts (/usr/bin/filtertraining as was in 5.2), where as the one linked above calls sa-learn direct.

      However, the script 'Zarafa-sa-learn.pl' still works fine here in 6.4 but the instructions need updating. I just had to create the folder /var/spool/filter/training...and find the old filter training script. I'm can't work out why the script you are running is using a file at ./tmp.pl? are you running the right script?
      The reply is currently minimized Show
    Your Reply