Forums

Resolved
0 votes
My server was recently on a black list. I found that there were a bunch of PHP files uploaded to the web site in various directories. I cleaned out all the bogus files and reset the permissions on the web server and I have stopped sending out spam, and the web server appears to be clean. I changed the web sql and server root passwords and updated the joomla config files and the web sites are working fine.

HOWEVER, my email server (prevshop.com) dos not appear to be working any more. It did work fine. Someitmes when I open the web admin panel in the browser and look at the SMTP menu, in the upper right corner I see beside "additional info" "connection failed" in red letters. when i run TOP from the command line the SQL process is pretty high (99-100%) still.

I also have roundcube web mail which use to work fine, but now when I run it it often tells me there are too many sql connections, and when the email list comes up, the date column is blank among several emails in my inbox.

My gut feeling is that something is wrong with the system mysql but I have no idea how to troubleshoot it.

Here is an output of my TOP

[root@prevshop ~]# top
top - 15:19:54 up 3 days, 22:05, 3 users, load average: 11.86, 12.15, 12.42
Tasks: 663 total, 1 running, 661 sleeping, 0 stopped, 1 zombie
Cpu(s): 12.3%us, 13.8%sy, 0.1%ni, 38.4%id, 35.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7539108k total, 6643372k used, 895736k free, 618688k buffers
Swap: 8384444k total, 286880k used, 8097564k free, 1351436k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20376 system-m 20 0 2766m 140m 4808 S 100.1 1.9 914:42.95 system-mysqld
20459 dansguar 30 10 123m 4044 440 S 2.0 0.1 3:04.72 dansguardian-av
1089 root 20 0 0 0 0 D 0.3 0.0 1:57.19 flush-9:2
1862 ldap 20 0 1975m 109m 2800 S 0.3 1.5 36:15.52 slapd
5300 root 20 0 156m 2796 2268 S 0.3 0.0 1:00.33 X
27819 amavis 20 0 317m 55m 3284 S 0.3 0.8 0:00.24 amavisd
27826 root 20 0 13536 1684 904 R 0.3 0.0 0:00.05 top
27835 amavis 20 0 317m 55m 3340 D 0.3 0.8 0:00.14 amavisd
30297 postfix 20 0 83652 4748 2996 S 0.3 0.1 0:07.06 trivial-rewrite
1 root 20 0 21448 1108 896 S 0.0 0.0 0:02.80 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:03.69 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:20.34 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.33 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:04.88 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1

Any help is appreciated,
Thanks,
George
Tuesday, June 02 2015, 07:22 PM
Share this post:
Responses (4)
  • Accepted Answer

    Friday, June 05 2015, 01:47 PM - #Permalink
    Resolved
    0 votes
    I missed your post. Remember there are three possible sources for spam e-mail - from your LAN, ClearOS itself and from the WAN. If it mail is coming from the LAN (infected PC) and you relay e-mail through your smtp server, look in /var/log/maillog for a typical mail sending sequence:
    Jun  4 19:42:38 server postfix/smtpd[26954]: connect from Black.howitts.co.uk[172.17.2.100]
    Jun 4 19:42:38 server postfix/smtpd[26954]: EB722E4C59: client=Black.howitts.co.uk[172.17.2.100]
    Jun 4 19:42:39 server postfix/cleanup[26956]: EB722E4C59: message-id=<55709C1E.50106@howitts.co.uk>
    Jun 4 19:42:39 server postfix/qmgr[30711]: EB722E4C59: from=<someone@howitts.co.uk>, size=31026, nrcpt=1 (queue active)
    Jun 4 19:42:39 server postfix/smtpd[26954]: disconnect from Black.howitts.co.uk[172.17.2.100]
    Jun 4 19:42:39 server postfix/smtpd[26959]: connect from localhost[127.0.0.1]
    Jun 4 19:42:39 server postfix/smtpd[26959]: 34417E6E9B: client=localhost[127.0.0.1]
    Jun 4 19:42:39 server postfix/cleanup[26956]: 34417E6E9B: message-id=<55709C1E.50106@howitts.co.uk>
    Jun 4 19:42:39 server postfix/qmgr[30711]: 34417E6E9B: from=<someone@howitts.co.uk>, size=31026, nrcpt=1 (queue active)
    Jun 4 19:42:39 server postfix/smtpd[26959]: disconnect from localhost[127.0.0.1]
    Jun 4 19:42:39 server postfix/pipe[26957]: EB722E4C59: to=<someone@somewhere.com>, relay=mailprefilter, delay=0.39, delays=0.16/0.01/0/0.22, dsn=2.0.0, status=sent (delivered via mailprefilter service)
    Jun 4 19:42:39 server postfix/qmgr[30711]: EB722E4C59: removed
    Jun 4 19:42:41 server postfix/smtpd[26963]: connect from localhost[127.0.0.1]
    Jun 4 19:42:41 server postfix/smtpd[26963]: 019AFE4C59: client=localhost[127.0.0.1]
    Jun 4 19:42:41 server postfix/cleanup[26956]: 019AFE4C59: message-id=<55709C1E.50106@howitts.co.uk>
    Jun 4 19:42:41 server postfix/qmgr[30711]: 019AFE4C59: from=<someone@howitts.co.uk>, size=31073, nrcpt=1 (queue active)
    Jun 4 19:42:41 server postfix/smtpd[26963]: disconnect from localhost[127.0.0.1]
    Jun 4 19:42:41 server amavis[14629]: (14629-06) Passed CLEAN, LOCAL [127.0.0.1] <someone@howitts.co.uk> -> <someone@somewhere.com>, Message-ID: <55709C1E.50106@howitts.co.uk>, mail_id: vYCBmBdn9+Ku, Hits: 1.105, size: 31025, queued_as: 019AFE4C59, 1666 ms
    Jun 4 19:42:41 server postfix/smtp[26960]: 34417E6E9B: to=<someone@somewhere.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.9, delays=0.1/0.08/0/1.7, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10026): 250 2.0.0 Ok: queued as 019AFE4C59)
    Jun 4 19:42:41 server postfix/qmgr[30711]: 34417E6E9B: removed
    Jun 4 19:42:41 server postfix/smtp[26964]: certificate verification failed for 127.0.0.1[127.0.0.1]:10465: untrusted issuer /C=US/O=Sample, Inc./OU=IT Team/CN=CA
    Jun 4 19:42:41 server postfix/smtp[26964]: 019AFE4C59: to=<someone@somewhere.com>, relay=127.0.0.1[127.0.0.1]:10465, delay=0.93, delays=0.05/0.06/0.51/0.31, dsn=2.0.0, status=sent (250 cJih1q00N49M2Xm01Jihgt mail accepted for delivery)
    Jun 4 19:42:41 server postfix/qmgr[30711]: 019AFE4C59: removed
    Your sequence will vary a bit at the end as I am relaying out via my ISP's server using stunnel. You'll probably have fewer lines if you are sending directly. From this sequence you can see the sending machine's IP address and name (client=Black.howitts.co.uk[172.17.2.100]). If you get a lot of these to destinations you don't recognise then perhaps that machine has an infection.

    If you only send mail from your LAN, can you turn off authentication and just set up a trusted network in the SMTP server? If that stops the spam then you need to change your user passwords. Someone will have guessed or cracked a user's password. Securing e-mail this way means anyone on your LAN can send e-mails as they don't need to authenticate, but no one else can. This is the way I run. If you then want to relay from your WAN, say from a smartphone on the road, by what could be a misconfiguration default, you can relay using SSL/TLS on port 465. This is already set up for user/pass authentication. All you have to do is to open port 465 as well. There is much less (virtually no?) cracking of passwords going on on port 465. If you do go down the SSL/TLS route also install fail2ban to block password cracking. Port 587 with STARTTLS can also be set up this way but post back for details.

    You'll need to inspect your maillog if the server is infected as I don't know the logging sequence for mail originating from the server.

    If a LAN client is sending mail directly you can try logging it with a firewall rule:
    iptables -I FORWARD -p tcp --dport 25 -j LOG
    This will log all port 25 traffic through your server but not to the server. To delete the rule change the -I to -D or restart the firewall. Logs will go into /var/log/messages. If needed the rule can be fine tuned just to pick up new connection attempts.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, June 04 2015, 02:20 PM - #Permalink
    Resolved
    0 votes
    I did a postsuper -d ALL and it took about 3 hours for it to clean out all the (1040981 total) messages.
    after an hour or two it filled back up to about 500 messages
    I turned off everything accept http and https on the firewall and now it does not seem to be filling back up.

    this is an educational thing for me... not a production email server, rather one I had set up for back office so i have some time.

    would really appreciate some guidance on how to troubleshoot and figure out how they are getting in to do this.

    I looked in the logs, but am not sure what to look for.

    I am using the paid version of roundcubemail. only two email users on the system....

    i reappeared on the black list, but disappeared again after closing up the firewall as described above.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, June 03 2015, 08:49 AM - #Permalink
    Resolved
    0 votes
    First question. Have you asked yourself how all those rogue files got there? Is that back door closed or have they left some other process running so they can always access your system? If you opened ssh or the Webconfig to the web, can I suggest you close them and use something like OpenVPN to connect to your server then you can use ssh/Webconfig as if you are a local user?

    Are you seeing anything in your logs (/var/log/maillog and /var/log/messages) to give you clues to the errors?

    Are you running the paid-for version of Roundcubemail or did you follow my instructions in a Roundcube thread? The paid-for version uses system-mysql and the epel version (my instructions) uses plain mysql. Do you have many users or data such as address books you's like to keep? If not, you may just be able to drop the databases and re-initialise them, but I think you'll need to get the smtp server working first.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, June 02 2015, 08:38 PM - #Permalink
    Resolved
    0 votes
    Note: clarification, I am now off of all blacklists. when I send an email to myself from gmail, I get nothing.. no return receipt, nothing, but I never get the email in my inbox of the webmail.
    The reply is currently minimized Show
Your Reply