Thanks, Nick. I just didn't see your follow up. Replaced the first box yesterday with new hardware.
We serve out the built in sites like Webconfig, Zarafa Webapp, and Z-Push. No other websites hosted. Root password was 14 characters. Found the external port was also shared as IMPI port on Supermicro box. Looking at that angle as well. SSH was blocked by firewall when not in use. Shell access component was installed for OpenLDAP and several users had it enabled after the fact but I'm unable to confirm it they did before on one of the boxes. The other box didn't have the component installed. It is not our practice to give shell access until needed and then custom rules to allow specific IP addresses to connect. No users needed it a either location.
I'm comparing the files that were left behind. Unfortunately, it covered its tracks pretty well. Most of the log files were removed along with bash history. Will be happy to share what I find.
Hi, Community. Just relating information at this time. Selected category as system security.
We have just over 25 ClearOS 7 boxes at various locations. Most are to for individual sites. We have had two boxes compromised in as many weeks. Not been able to ascertain the attack vector. We find a user named support with the same ID as root and indicates the same login times as our with root. It removes the bash history for root, removes many of the logs from /var/log and adds config.txt, cpu.txt, minerd, and monero to /etc/sbin. Cron for root is edited to start monero from /etc/sbin every 5 minutes. We have managed to recreate the logs and rename the additional files to prevent the miner from starting again and getting the boxes operational again (both run Zarafa Community). I removed the user via "vipw" and "vipw -s". Not going to leave the boxes in production.
Port 81 is open and accessible from outside, passwords are fairly complex, root access is allow from WAN but the port was closed (we only open when we to use for support). It is possible the threat came from LAN side but not been able to find anything from remaining information.
Posted queries about restoring mail archives due to compromised box but didn't hear anything from anyone. Was wondering if anyone had seen anything like this or if ClearOS would be interested in some telemetry. Going to replace the box compromised last week with fresh install tomorrow and try to replace the other by end of week.
Had a ClearOS 7 box compromised and running minerd. Was able to stop the mining and get the box settled but do not trust it. Will be moving mail to new box. Zarafa community should be pretty straight forward. I've done sqldump from old box and import database into the new one. My question is what is the recommended method for moving the mail archive so that the new box will be able to search the old mail archives. Never have figured this out but we've always went from different versions of OS and Zarafa. Anyone have some pointers?
Nick Howitt wrote:
The single threaded performance of your processor is quite low. Can you try stopping clamd then time its startup?
Tried three times. 1st time services failed to start, rebooted after 6 minutes. 2nd time 4min, 6sec. 3rd time 4 min, 5 sec.
Trying changing clamd.conf turning on verbose logging and increasing ReadTimeout to 400.
Appreciate you thinking through it with me, Nick.
From System Details.... It is a supermicro 1U box, Intel(R) Atom(TM) CPU C2550 @ 2.41GHz, Memory Size 15.5 GB, Load 0.19 0.35 0.29. There is also an 8GB swap but I don't remember seeing it used. Drives are Samsung 750 EVO SSDs in RAID1. Firewall duties, mail prefiltering, and one IPSEC VPN.
I had noted that as well the last time it happened on 10/8. It failed again this morning for a couple of minutes at 5:39am and at 7:49am when we were also experiencing bandwidth issues from ISP. Found unfinished transactions this morning when installing hdparm and cleared those with "yum-complete-transaction" which cleared the transactions. It found one transaction with four elements all dealing with app-antivirus-core, app-file-scan, clamav, app-antimalware, app-antiphishing, app-mail-antivirus, and app-content-filter. May be due to the uninstall/reinstall process yesterday.
I believe I had also checked to make sure we had no other repositories install other than the standard ClearOS ones last time this happened from a thread I had found.
ClearOS 7 Business Edition, Silver current to 9/2020
Happens periodically. Has gone several weeks between issues. Have seen it on a few other COS 7 boxes but it is consistently breaking on this one. Last two mornings are most recent. Mail comes in with [UNCHECKED] in subject line. Check maillog reports:
Oct 31 05:20:17 lgedge amavis: (04850-09) (!)ClamAV-clamd av-scanner FAILED: run_av error: ask_daemon_internal: Exceeded allowed time at (eval 128) line 611.\n
but clamd.log reports (looks normal to my untrained eye):
Thu Oct 31 05:15:22 2019 -> SelfCheck: Database modification detected. Forcing reload.
Thu Oct 31 05:15:24 2019 -> Reading databases from /var/lib/clamav
Thu Oct 31 05:20:23 2019 -> Database correctly reloaded (6608708 signatures)
Thu Oct 31 05:20:46 2019 -> Pid file removed.
Thu Oct 31 05:20:46 2019 -> --- Stopped at Thu Oct 31 05:20:46 2019
Thu Oct 31 05:20:46 2019 -> Socket file removed.
Thu Oct 31 05:20:46 2019 -> +++ Started at Thu Oct 31 05:20:46 2019
Thu Oct 31 05:20:46 2019 -> Received 0 file descriptor(s) from systemd.
Thu Oct 31 05:20:46 2019 -> clamd daemon 0.101.4 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Oct 31 05:20:46 2019 -> Running as user clam (UID 989, GID 988)
Thu Oct 31 05:20:46 2019 -> Log file size limited to 4294967295 bytes.
Thu Oct 31 05:20:46 2019 -> Reading databases from /var/lib/clamav
Thu Oct 31 05:20:46 2019 -> Not loading PUA signatures.
So far the restarting services has not fixed it but I may not be doing a specific order. What "fixes" it is to uninstall mail antivirus and gateway antivirus which removes content filtering and clamav tools and reinstall from the Marketplace.
Need to get this behaving a bit more stable. I'm still digging but will welcome all ideas?
Got a compromised box running Clear 6.9 and Zarafa community 7.0.15. We managed to get the box responsive but it is not trusted and I've reloaded another with similar hardware and plan to swap out the hard drives into the untrusted box. I've gotten most everything configured and believe the Zarafa restore should be pretty straight forward of replacing the database with the mysqldump of the existing and copying the attachments over to /var/lib/zarafa.
My question is for the mail archive. I know I can manually archive for the last couple of weeks and copy the mail archives over to the new box at /var/clearos/mail_archive. Is there anything I need to do or get from the old box before I try to access the archives from the new box?
Appreciate the thoughts/concerns.
Not quite perfect but close enough for to get by. I had increased logging for freshclam.conf and clamd.conf which greatly increased the clamd.log. Also had read about increasing the "TimeoutStartSec" time in clamd.service. Increased this from 120s to 240s and this had a significant impact on it. I still have brief Error connecting to ClamD socket but they are limited in duration to just a couple of minutes and at early morning hours when offices are closed. My guess if where clamd is restarted due to updates or automated system maintenance. Never found much in the verbose logs to be much help but I'm new to the clamav process and have reverted that change to get the log size back down.
Just wanted to share what is working for us and to give a shout out to Nick for his input. Thanks to the community!
Spoke too soon. Log shows where ClamD was unavailable again for about an hour Friday morning from 3-4am CDT and then failed to start at all Saturday. Service resolved Sunday morning around 4am.
This morning same "Error connecting to ClamD socket" this morning. Ran the yum clamav* remove and reinstalled Content Filter, CF blacklists, Gateway AntiPhishing/AntiVirus, and Mail AntiVirus from MarketPlace. Install went smoothly, started Content Filtering and can access http websites once again. Permissions appeared correct on databases compared to installation that is working normally.
Not sure where to look next.