Forums

Leon
Leon
Offline
Resolved
2 votes
My system seems to have updated this morning from 02:10 - Yum Log
Since than i am getting a HIGH volume of log files - messages - Started Session c4476 of user root - about 36 entries every second??

Apr 6 17:09:48 ZodiacTech systemd: Started Session c789 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c790 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c791 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c792 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c793 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c794 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c795 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c796 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c797 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c798 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c799 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c800 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c801 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c802 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c803 of user root.
Apr 6 17:09:48 ZodiacTech systemd: Started Session c804 of user root

Did a "HARD" reset so that i can get in and copy from /var/log/messages

It seems that my system is getting overwhelmed, so bad that i can't login even via ssh anymore.

What i did see before i could not get in anymore was a higher memory usage by clamd

Any idea where i can look to find out what is causing this?
Saturday, April 06 2019, 03:14 PM
Share this post:
Responses (61)
  • Accepted Answer

    Thursday, May 23 2019, 02:14 PM - #Permalink
    Resolved
    0 votes
    I have investigated Jeff's system and the root cause could be faced by a few people.
    Csplugin-events updated during the big 7.6 update. The only thing it did was to replace /etc/clearos/events.d/20-user-auth.conf which added a filter for the two events you were seeing. In Jeff's case he had edited the file in early 2017. Unfortunately, in the rpm system this file is flagged as a config file, so if yum notices the conf file has been edited, it does not replace it, but creates the new one with an rpmnew extension when it updates.

    In this sort of case the user should really regularly go through all rpmnew files ("locate rpmnew") and make a judgement call about them in case they need to make adjustments to their edited file. In this case I moved the original file to /root and renamed the rpmnew file to /etc/clearos/events.d/20-user-auth.conf and restarted clearsync. For the sake of cleanliness I also removed the old events database.

    What should now be done is to compare the old /root/20-user-auth.conf to the new /etc/clearos/events.d/20-user-auth.conf and see if there are any changes in the old file which you want to bring into the new file.

    In terms of Clearcenter, we could consider making the changes differently and adding an upgrade script to add the new lines to the old file to cover the case that it has been edited. This is possible and will be discussed in a technical meeting, but the file structure is no so neat for inserting lines programatically.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 22 2019, 09:12 PM - #Permalink
    Resolved
    0 votes
    @Brendon and Jeff,
    Please can you raise a General Enquiry ticket at https://secure.clearcenter.com/portal/ticket_manage.jsp and submit remote log in credentials to https://secure.clearcenter.com/portal/system_password.jsp and I'll see if someone can have a look tomorrow. This will be done free of charge.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 22 2019, 08:31 PM - #Permalink
    Resolved
    0 votes
    Jeff Rynders wrote:

    Looks like I'm having the same issues as Brendon (User root logged in and out via sudo continuously)

    Results of what you asked above:
    app-base-2.6.9-2.v7.noarch
    csplugin-events-1.2-3.v7.x86_64
    system-base-7.6.5-1.v7.x86_64


    Hi Jeff, I have not found a solution thus far, it is still logging sudo events like crazy. It is not as bad if you don't have webconfig open in a browser but it does not stop it. I'm wondering how concerned I should be with it thrashing logs on an SSD drive, but luckily it is running in a VM that is backed up frequently.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 22 2019, 08:29 PM - #Permalink
    Resolved
    0 votes
    Are you sure your messages are current? Pleas can you clear your messages database with a:
    systemctl stop clearsync.service
    rm -f /var/lib/csplugin-events/events.db
    systemctl start clearsync.service
    Then have another look.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 22 2019, 08:00 PM - #Permalink
    Resolved
    0 votes
    Looks like I'm having the same issues as Brendon (User root logged in and out via sudo continuously)

    Results of what you asked above:
    app-base-2.6.9-2.v7.noarch
    csplugin-events-1.2-3.v7.x86_64
    system-base-7.6.5-1.v7.x86_64
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 16 2019, 04:15 PM - #Permalink
    Resolved
    0 votes
    system-base will reduce the logging into /var/log/secure but not the events into the events manager (which is the killer). You will need to get the LogLevel=notice to work.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 16 2019, 03:53 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    I don't understand this. Those events are info events so should not be being pushed into the events now. What is the contents of /etc/systemd/system.conf.d/clearos-base.conf and what is the result of:
    grep LogLevel /etc/systemd/system.conf
    Can you also try the manual command which only works until the next reboot:
    systemd-analyze set-log-level notice
    This should have been fixed by system-base.

    In your secure log are you seeing horrible logging or not? This should have been fixed by app-base, but we will be moving the fix to system-base. Your packages are out of sync here.

    Please be careful as you've been updating packages out of clearos-updates-testing. You've pulled the latest app-base from there, so the filtering of secure messages has been dropped and it is now in system-base which you have not pulled. Please do a:
    yum update system-base --enablerepo=clearos-updates testing
    This should quieten down the secure log but does not affect the events.


    # grep LogLevel /etc/systemd/system.conf
    #LogLevel=info

    I am getting a ton of these sudo info logging in my secure log
    Apr 16 11:45:02 GW sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
    Apr 16 11:45:02 GW sudo: pam_unix(sudo:session): session closed for user root
    Apr 16 11:45:03 GW sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
    Apr 16 11:45:03 GW sudo: pam_unix(sudo:session): session closed for user root

    I will try updating the system base from testing and see if that helps, thanks!
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 16 2019, 03:29 PM - #Permalink
    Resolved
    0 votes
    I don't understand this. Those events are info events so should not be being pushed into the events now. What is the contents of /etc/systemd/system.conf.d/clearos-base.conf and what is the result of:
    grep LogLevel /etc/systemd/system.conf
    Can you also try the manual command which only works until the next reboot:
    systemd-analyze set-log-level notice
    This should have been fixed by system-base.

    In your secure log are you seeing horrible logging or not? This should have been fixed by app-base, but we will be moving the fix to system-base. Your packages are out of sync here.

    Please be careful as you've been updating packages out of clearos-updates-testing. You've pulled the latest app-base from there, so the filtering of secure messages has been dropped and it is now in system-base which you have not pulled. Please do a:
    yum update system-base --enablerepo=clearos-updates testing
    This should quieten down the secure log but does not affect the events.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 16 2019, 02:29 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    @Brendon,
    What is the output from:
    rpm -q app-base csplugin-events system-base



    rpm -q app-base csplugin-events system-base
    app-base-2.6.7-2.v7.noarch
    csplugin-events-1.2-3.v7.x86_64
    system-base-7.6.3-1.v7.x86_64
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 16 2019, 07:22 AM - #Permalink
    Resolved
    0 votes
    @Brendon,
    What is the output from:
    rpm -q app-base csplugin-events system-base


    The problem largely goes away of you close down your webconfig screen, and some pages are less susceptible than others (IP Settings is terrible). So, to get your system responsive again, come out of the webconfig and wait a while. You may hear your disk activity quieten down.

    You can remove all the messages in bulk with:
    systemctl stop clearsync.service
    rm -f /var/lib/csplugin-events/events.db
    systemctl start clearsync.service
    The reply is currently minimized Show
  • Accepted Answer

    Monday, April 15 2019, 10:20 PM - #Permalink
    Resolved
    0 votes
    I have followed this thread and made sure the updates were installed and am still having this problem, thousands instances per hour of
    User root logged in via sudo 2019-04-15 18:16:19
    User root logged out via sudo 2019-04-15 18:16:19
    User root logged in via sudo 2019-04-15 18:16:19
    User root logged out via sudo 2019-04-15 18:16:19
    User root logged in via sudo 2019-04-15 18:16:19
    User root logged out via sudo 2019-04-15 18:16:19
    User root logged in via sudo 2019-04-15 18:16:17
    User root logged out via sudo 2019-04-15 18:16:17
    User root logged in via sudo 2019-04-15 18:16:17
    User root logged out via sudo 2019-04-15 18:16:17
    It started about 10 days ago when my system updated to 7.6, it gets to the point that it causes the dashboard to give a http 500 error. This morning I had 1.5 million notices before clearing them when I couldn't access the dashboard.
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, April 14 2019, 12:54 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    @Siv,
    Did your "smbd_audit" messages also disappear? If so, I am not sure that it is anything we fixed and may have been a separate issue. I've not been able to replicate it.


    Nick,

    No, since the updates the machine has quietened down and I am not getting the same events logged.
    Whatever you did fixed that issue as well.

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Friday, April 12 2019, 04:19 PM - #Permalink
    Resolved
    0 votes
    @Siv,
    Did your "smbd_audit" messages also disappear? If so, I am not sure that it is anything we fixed and may have been a separate issue. I've not been able to replicate it.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, April 11 2019, 04:13 PM - #Permalink
    Resolved
    0 votes
    There are a few other minor improvements to be made here. Some of this activity is because we changed the way that the ClearOS system interacts with the core and we are using a 'sudo' method which fixes some best practices issues but also made the logging issues. We still have a bunch of useless logging events and we are going to be making targeted filters so that the logs do not rotate like made with useless data that we already know about.
    The reply is currently minimized Show
  • Accepted Answer

    Greg
    Greg
    Offline
    Thursday, April 11 2019, 03:47 PM - #Permalink
    Resolved
    0 votes
    Thank you Nick for working on my server. Everything is good. I updated my second server today via the webclient and everything looks good this server as well.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, April 11 2019, 02:36 PM - #Permalink
    Resolved
    0 votes
    Nick, Dave,

    All seems well today, I have had no pages of log files in my reports, in fact I had no report from the server last night so whatever you guys did it has really quietened things down.
    I checked through some of the other logs and nothing appears to be going mad today.
    Thanks again for all your help resolving this.


    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Leon
    Leon
    Offline
    Wednesday, April 10 2019, 08:05 PM - #Permalink
    Resolved
    0 votes
    Hi All

    Thank you for all the effort that was put into resolving this issue.
    I was able to leave my server off until today, not everyone can do this, and waited for a solution - i just ran the update from the web interface the moment i booted.
    After the update ran, i also did:

    systemctl stop clearsync.service
    rm -f /var/lib/csplugin-events/events.db
    systemctl start clearsync.service

    Everything is now responding and i do not see any more log files going "NUTS"

    Thank you again for everyone involved

    Leon
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 03:30 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    Both clearsync and rsyslog should have restarted as part of the update if they were already running. This should fix all the events messages about root opening and closing sessions. You can clean the events log completely with:
    systemctl stop clearsync.service
    rm -f /var/lib/csplugin-events/events.db
    systemctl start clearsync.service


    It should sort out some of the responsiveness, but I don't think it will do anything about the samba issue and that could also be hammering the events database. Only time will tell.


    Nick,

    Thanks for the update. I ran these three commands none gave any response so I assume they all completed successfully.
    I will watch out for my emailed report tonight and will post a new thread here tomorrow if I am still getting the samba errors.

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 03:10 PM - #Permalink
    Resolved
    0 votes
    Both clearsync and rsyslog should have restarted as part of the update if they were already running. This should fix all the events messages about root opening and closing sessions. You can clean the events log completely with:
    systemctl stop clearsync.service
    rm -f /var/lib/csplugin-events/events.db
    systemctl start clearsync.service


    It should sort out some of the responsiveness, but I don't think it will do anything about the samba issue and that could also be hammering the events database. Only time will tell.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 02:46 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    That is weird. Just a plain old "yum update" should work as I moved them into the clearos-update this morning. The --enablerepo bit is irrelevant now and the yum command should work with or without it. Can I suggest you leave the clearsync service disabled for the moment (or until you get the updates) and then do a:
    yum clean all && yum update


    Sorry Nick your post came in whilst I was working through Dave's recommendation.
    It does appear that I have now updated the csplugin.

    From the command response I posted do you think that this should do the trick or am I missing something else?

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 02:38 PM - #Permalink
    Resolved
    0 votes
    Dave Loper wrote:

    The update that you should be running is csplugin-events-1.2-3.v7

    http://koji.clearos.com/koji/buildinfo?buildID=2618

    You can look at what version you are running by using:

    rpm -qi csplugin-events

    If you are not running this version, run the following to clean up your yum cache:

    yum --enablerepo=* clean all

    As you can see from koji, this package is in clearos-updates. If you are running ClearOS Community you will see this by just doing a 'yum update' after doing the 'yum --enablerepo=* clean all'

    If you are running ClearOS Business and have installed these updates already, then you will need to enable the clearos-udpates repo in order to grab this update:

    yum --enablerepo=clearos-updates update csplugin-events



    Dave,

    Thanks for the advice, I ran the commands and I __was__ running the previous version.
    I then did:

    yum --enablerepo=* clean all

    Then:
    yum update

    Below is the response:

    [root@remote ~]# yum update
    Loaded plugins: clearcenter-marketplace, fastestmirror
    ClearCenter Marketplace: fetching repositories...
    Determining fastest mirrors
    * clearos: www.mirrorservice.org
    * clearos-centos: download3.clearsdn.com
    * clearos-centos-sclo-rh: download3.clearsdn.com
    * clearos-centos-updates: download3.clearsdn.com
    * clearos-contribs: www.mirrorservice.org
    * clearos-epel: download3.clearsdn.com
    * clearos-fast-updates: download3.clearsdn.com
    * clearos-infra: www.mirrorservice.org
    * clearos-updates: www.mirrorservice.org
    * private-clearcenter-dyndns: download3.clearsdn.com:80
    * private-clearcenter-roundcubemail: download1.clearsdn.com:80
    clearos | 3.7 kB 00:00
    clearos-centos | 3.0 kB 00:00
    clearos-centos-sclo-rh | 3.0 kB 00:00
    clearos-centos-updates | 3.0 kB 00:00
    clearos-contribs | 3.5 kB 00:00
    clearos-epel | 2.9 kB 00:00
    clearos-fast-updates | 3.0 kB 00:00
    clearos-infra | 3.5 kB 00:00
    clearos-updates | 3.5 kB 00:00
    (1/13): clearos/7/group_gz | 1.6 kB 00:00
    (2/13): clearos-contribs/7/updateinfo | 96 B 00:00
    (3/13): clearos-contribs/7/primary_db | 71 kB 00:00
    (4/13): clearos/7/primary_db | 739 kB 00:00
    (5/13): clearos-fast-updates/x86_64/primary_db | 10 kB 00:00
    (6/13): clearos-infra/7/updateinfo | 96 B 00:00
    (7/13): clearos-infra/7/primary_db | 11 kB 00:00
    (8/13): clearos-updates/7/updateinfo | 96 B 00:00
    (9/13): clearos-updates/7/primary_db | 861 kB 00:00
    (10/13): clearos-centos-sclo-rh/x86_64/primary_db | 3.5 MB 00:07
    (11/13): clearos-centos-updates/x86_64/primary_db | 2.3 MB 00:09
    (12/13): clearos-epel/7/x86_64/primary_db | 7.4 MB 00:09
    (13/13): clearos-centos/x86_64/primary_db | 5.0 MB 00:14
    private-clearcenter-dyndns | 3.0 kB 00:00
    private-clearcenter-dyndns/primary_db | 2.6 kB 00:00
    private-clearcenter-roundcubemail | 3.0 kB 00:00
    private-clearcenter-roundcubemail/primary_db | 2.5 kB 00:00
    Resolving Dependencies
    --> Running transaction check
    ---> Package app-base.noarch 1:2.6.3-1.v7 will be updated
    ---> Package app-base.noarch 1:2.6.5-1.v7 will be an update
    ---> Package app-base-core.noarch 1:2.6.3-1.v7 will be updated
    ---> Package app-base-core.noarch 1:2.6.5-1.v7 will be an update
    ---> Package csplugin-events.x86_64 0:1.2-2.v7 will be updated
    ---> Package csplugin-events.x86_64 0:1.2-3.v7 will be an update
    --> Finished Dependency Resolution

    Dependencies Resolved

    ================================================================================
    Package Arch Version Repository Size
    ================================================================================
    Updating:
    app-base noarch 1:2.6.5-1.v7 clearos-updates 32 k
    app-base-core noarch 1:2.6.5-1.v7 clearos-updates 454 k
    csplugin-events x86_64 1.2-3.v7 clearos-updates 84 k

    Transaction Summary
    ================================================================================
    Upgrade 3 Packages

    Total download size: 570 k
    Is this ok [y/d/N]: y
    Downloading packages:
    Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
    (1/3): app-base-2.6.5-1.v7.noarch.rpm | 32 kB 00:00
    (2/3): app-base-core-2.6.5-1.v7.noarch.rpm | 454 kB 00:00
    (3/3): csplugin-events-1.2-3.v7.x86_64.rpm | 84 kB 00:00
    --------------------------------------------------------------------------------
    Total 1.0 MB/s | 570 kB 00:00
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
    Updating : 1:app-base-core-2.6.5-1.v7.noarch 1/6
    Updating : 1:app-base-2.6.5-1.v7.noarch 2/6
    Updating : csplugin-events-1.2-3.v7.x86_64 3/6
    Cleanup : 1:app-base-2.6.3-1.v7.noarch 4/6
    Cleanup : 1:app-base-core-2.6.3-1.v7.noarch 5/6
    Cleanup : csplugin-events-1.2-2.v7.x86_64 6/6
    Verifying : 1:app-base-core-2.6.5-1.v7.noarch 1/6
    Verifying : csplugin-events-1.2-3.v7.x86_64 2/6
    Verifying : 1:app-base-2.6.5-1.v7.noarch 3/6
    Verifying : csplugin-events-1.2-2.v7.x86_64 4/6
    Verifying : 1:app-base-2.6.3-1.v7.noarch 5/6
    Verifying : 1:app-base-core-2.6.3-1.v7.noarch 6/6

    Updated:
    app-base.noarch 1:2.6.5-1.v7 app-base-core.noarch 1:2.6.5-1.v7
    csplugin-events.x86_64 0:1.2-3.v7

    Complete!
    [root@remote ~]#


    So I now have "csplugin-events.x86_64 0:1.2-3.v7", let's hope this qquitens down the logging of events.
    I will see if after applying this update the log files stop writing events every second or so.

    Should I restart the server or a specific service?

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 02:26 PM - #Permalink
    Resolved
    0 votes
    That is weird. Just a plain old "yum update" should work as I moved them into the clearos-update this morning. The --enablerepo bit is irrelevant now and the yum command should work with or without it. Can I suggest you leave the clearsync service disabled for the moment (or until you get the updates) and then do a:
    yum clean all && yum update
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 02:23 PM - #Permalink
    Resolved
    0 votes
    The update that you should be running is csplugin-events-1.2-3.v7

    http://koji.clearos.com/koji/buildinfo?buildID=2618

    You can look at what version you are running by using:

    rpm -qi csplugin-events

    If you are not running this version, run the following to clean up your yum cache:

    yum --enablerepo=* clean all

    As you can see from koji, this package is in clearos-updates. If you are running ClearOS Community you will see this by just doing a 'yum update' after doing the 'yum --enablerepo=* clean all'

    If you are running ClearOS Business and have installed these updates already, then you will need to enable the clearos-udpates repo in order to grab this update:

    yum --enablerepo=clearos-updates update csplugin-events
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 02:07 PM - #Permalink
    Resolved
    0 votes
    @Nick,
    When I ran the command it came up with "No packages marked for update":

    yum update csplugin-events app-base --enablerepo=clearos-updates-testing
    Loaded plugins: clearcenter-marketplace, fastestmirror
    ClearCenter Marketplace: fetching repositories...
    Loading mirror speeds from cached hostfile
    * clearos: www.mirrorservice.org
    * clearos-centos: download3.clearsdn.com
    * clearos-centos-sclo-rh: download3.clearsdn.com
    * clearos-centos-updates: download3.clearsdn.com
    * clearos-contribs: www.mirrorservice.org
    * clearos-epel: download3.clearsdn.com
    * clearos-fast-updates: download3.clearsdn.com
    * clearos-infra: www.mirrorservice.org
    * clearos-updates: www.mirrorservice.org
    * clearos-updates-testing: www.mirrorservice.org
    * private-clearcenter-dyndns: download1.clearsdn.com:80
    * private-clearcenter-roundcubemail: download1.clearsdn.com:80
    clearos-updates-testing | 3.5 kB 00:00
    (1/2): clearos-updates-testing/7/updateinfo | 96 B 00:00
    (2/2): clearos-updates-testing/7/primary_db | 736 kB 00:00

    No packages marked for update

    [root@remote ~]# service clearsync start
    Redirecting to /bin/systemctl start clearsync.service


    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 02:01 PM - #Permalink
    Resolved
    0 votes
    @Nick,
    I just checked my system and no outstanding updates, the following were the updates that were applied most recently and seemed to trigger all sorts of log activity:
     Package	Action	Date/Time
    elfutils-default-yama-scope-0.172-2.el7 Updated Apr 08, 01:02:54
    linux-firmware-20180911-69.git85c5d90.el ... Updated Apr 07, 05:48:51
    lm_sensors-libs-3.4.0-6.20160601gitf9185 ... Updated Apr 07, 05:48:41
    samba-python-4.8.3-4.3.v7 Updated Apr 07, 05:48:41
    grub2-2.02.1-0.76.v7.1 Updated Apr 07, 05:48:40
    samba-winbind-clients-4.8.3-4.3.v7 Updated Apr 07, 05:48:40
    samba-client-4.8.3-4.3.v7 Updated Apr 07, 05:48:39
    samba-dc-4.8.3-4.3.v7 Updated Apr 07, 05:48:39
    libsmbclient-4.8.3-4.3.v7 Updated Apr 07, 05:48:38
    samba-dc-libs-4.8.3-4.3.v7 Updated Apr 07, 05:48:38
    samba-winbind-4.8.3-4.3.v7 Updated Apr 07, 05:48:38
    samba-winbind-modules-4.8.3-4.3.v7 Updated Apr 07, 05:48:35
    grub2-pc-2.02.1-0.76.v7.1 Updated Apr 07, 05:48:34
    grub2-pc-modules-2.02.1-0.76.v7.1 Updated Apr 07, 05:48:34
    grub2-tools-extra-2.02.1-0.76.v7.1 Updated Apr 07, 05:48:33
    samba-4.8.3-4.3.v7 Updated Apr 07, 05:48:33
    samba-common-tools-4.8.3-4.3.v7 Updated Apr 07, 05:48:27
    grub2-tools-2.02.1-0.76.v7.1 Updated Apr 07, 05:48:26
    grub2-tools-minimal-2.02.1-0.76.v7.1 Updated Apr 07, 05:48:26
    grub2-common-2.02.1-0.76.v7.1 Updated Apr 07, 05:48:25
    samba-libs-4.8.3-4.3.v7 Updated Apr 07, 05:48:25
    samba-client-libs-4.8.3-4.3.v7 Updated Apr 07, 05:48:24
    samba-common-libs-4.8.3-4.3.v7 Updated Apr 07, 05:48:23
    libwbclient-4.8.3-4.3.v7 Updated Apr 07, 05:48:22
    samba-common-4.8.3-4.3.v7 Updated Apr 07, 05:48:21
    mesa-libwayland-egl-17.2.3-8.20171019.el ... Erased Apr 06, 02:07:11
    iwl3160-firmware-22.0.7.0-69.el7 Updated Apr 06, 02:02:59
    iwl3945-firmware-15.32.2.9-69.el7 Updated Apr 06, 02:02:59
    iwl6050-firmware-41.28.5.1-69.el7 Updated Apr 06, 02:02:59
    iwl100-firmware-39.31.5.1-69.el7 Updated Apr 06, 02:02:58
    iwl2000-firmware-18.168.6.1-69.el7 Updated Apr 06, 02:02:58
    iwl6000-firmware-9.221.4.1-69.el7 Updated Apr 06, 02:02:58
    iwl7260-firmware-22.0.7.0-69.el7 Updated Apr 06, 02:02:58
    iwl1000-firmware-39.31.5.1-69.el7 Updated Apr 06, 02:02:57
    iwl135-firmware-18.168.6.1-69.el7 Updated Apr 06, 02:02:57
    iwl5150-firmware-8.24.2.2-69.el7 Updated Apr 06, 02:02:57
    iwl6000g2b-firmware-17.168.5.2-69.el7 Updated Apr 06, 02:02:57
    emacs-filesystem-24.3-22.el7 Updated Apr 06, 02:02:56
    iwl105-firmware-18.168.6.1-69.el7 Updated Apr 06, 02:02:56
    iwl2030-firmware-18.168.6.1-69.el7 Updated Apr 06, 02:02:56
    iwl4965-firmware-228.61.2.24-69.el7 Updated Apr 06, 02:02:56
    iwl5000-firmware-8.83.5.1_1-69.el7 Updated Apr 06, 02:02:56
    iwl6000g2a-firmware-17.168.5.3-69.el7 Updated Apr 06, 02:02:56
    dialog-1.2-5.20130523.el7 Updated Apr 06, 02:02:55
    dosfstools-3.0.20-10.el7 Updated Apr 06, 02:02:55
    dmidecode-3.1-2.el7 Updated Apr 06, 02:02:53
    iw-4.3-2.el7 Updated Apr 06, 02:02:53
    iptraf-ng-1.1.4-7.el7 Updated Apr 06, 02:02:52
    libgomp-4.8.5-36.el7 Updated Apr 06, 02:02:52
    augeas-libs-1.4.0-6.el7_6.1 Updated Apr 06, 02:02:51


    Because the initial errors I posted seemed to relate to Samba and seemed to start around the time the 7th April updates were being applied i have assumed the two things are related.

    One other thing that occurred was the Windows machines that had mapped links to Flexshares on the server suddenly would not connect and I had to restart the server to get them to connect up again.

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 01:49 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    @Siv,
    I've released the programs into clearos-updates so they will update tonight. You could try Webconfig > Cloud > Updates > Software Updates and see if that pull them down. I can't test as I've updated my systems. If it shows them but does not give you the chance to update, try switching to a manual update (disable "Automatic Updates").

    If you can't update, I suggest you shut down the webconfig on your PC or avoid the screens with a lot of ajax. I don't know which they are, but any with fields that update as you watch the screen, so certainly avoid the IP Settings screen.


    Nick,
    I finally did manage to get in via SSH and was able to run the command:

    yum update csplugin-events app-base --enablerepo=clearos-updates-testing


    So I am hopeful that your updates will get through and quieten things down again.

    I will watch to see if the other messages come up in the emailed report I get sent at the end of the day that I initially reported as I thought they were related to this. If they do I will raise them as a new thread in these forums.

    As always thanks for all you do Nick.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 11:51 AM - #Permalink
    Resolved
    0 votes
    Sorry - wasn't clear enough. Fedora 29 still the same result as with ClearOS/CentOS 7.x - systemctl doesn't work without the edit to the conf file, service does work regardless...
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 11:43 AM - #Permalink
    Resolved
    0 votes
    Tony Ellis wrote:

    Nick - still the same on the latest Fedora, just tried it... thanks for the workaround...

    Maybe "service" is classified under "may be requested by dependency only"
    I suspect it is just an upstream bug which someone noticed and patched and FC is way ahead of ClearOS/Centos/RHEL. We are just using the el7 version of systemd unmodified.

    [edit]
    auditd changed to systemd
    [/edit]
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 11:23 AM - #Permalink
    Resolved
    0 votes
    Nick - still the same on the latest Fedora, just tried it... thanks for the workaround...

    Maybe "service" is classified under "may be requested by dependency only"
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 11:03 AM - #Permalink
    Resolved
    0 votes
    @Siv,
    I've released the programs into clearos-updates so they will update tonight. You could try Webconfig > Cloud > Updates > Software Updates and see if that pull them down. I can't test as I've updated my systems. If it shows them but does not give you the chance to update, try switching to a manual update (disable "Automatic Updates").

    If you can't update, I suggest you shut down the webconfig on your PC or avoid the screens with a lot of ajax. I don't know which they are, but any with fields that update as you watch the screen, so certainly avoid the IP Settings screen.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 10:53 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    @Graham, I don't think you have a choice. It should just kill the process from running, i.e brutally stop it. It will not delete the service. You should then regain SSH access.

    I've just released the updates, but they won't come through automatically until tonight.


    Nick I managed to kill clearsyncd but still can't get in via SSH it just keeps timing out? The users don't appear to be having any issues so I think I will wait and see if the update gets through tonight, or do I have to run the command:

    yum update csplugin-events app-base --enablerepo=clearos-updates-testing


    Before I can get your fixes?

    I looked again at the processes and clearsyncd has turned back on again. It did get killed through the web GUI but has started back up again.

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 10:51 AM - #Permalink
    Resolved
    0 votes
    Tony Ellis wrote:


    [root@danda rules.d]# systemctl stop auditd
    Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).
    See system logs and 'systemctl status auditd.service' for details.

    Same on every system - all 7.x community...
    Here is a nice bug for you. Instead try:
    service auditd stop


    [root@gateway-h ~]# service auditd start
    Redirecting to /bin/systemctl start auditd.service
    [root@gateway-h ~]# systemctl stop auditd
    Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).
    See system logs and 'systemctl status auditd.service' for details.

    [root@gateway-h ~]# service auditd stop
    Stopping logging: [ OK ]
    I was being lazy and using the service command!
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 10:35 AM - #Permalink
    Resolved
    0 votes

    [root@danda rules.d]# systemctl stop auditd
    Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).
    See system logs and 'systemctl status auditd.service' for details.

    Same on every system - all 7.x community...
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 10:08 AM - #Permalink
    Resolved
    0 votes
    Tony Ellis wrote:

    Nick

    adding a rules file to /etc/audit/rules.d and restarting the audit daemon

    You cannot do this using systemctl on a standard ClearOS system - manual start,stop and restart of auditd.. es ist verboten
    Are you sure? I've been doing it yesterday and today to troubleshoot adding rules to /etc/audit/rules.d.

    There may be a particular auditd rule you can enable which may enforce the "RefuseManualStop" (/usr/share/doc/audit-2.8.4/rules), but with a system I installed a couple of days ago, I can stop/start/restart.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 09:38 AM - #Permalink
    Resolved
    0 votes
    Nick

    adding a rules file to /etc/audit/rules.d and restarting the audit daemon

    You cannot do this using systemctl on a standard ClearOS system - manual start,stop and restart of auditd.. es ist verboten

    Reboot, or a quick and dirty way...
    1. Edit /etc/systemd/system/multi-user.target.wants/auditd.service
    2. Find the line "RefuseManualStop=yes" and change to "RefuseManualStop=no"
    3. Run "systemctl daemon-reload"
    4. run "systemctl restart auditd"

    auditd is an SSD killer, run most of my ClearOS systems on an SSD - where there is also a standard disk, edit /etc/audit/auditd.conf and log to another directory not on the SSD. For systems with SSD only - crude solution at the moment is to change the line "write_logs = yes" to "write_logs = no" in the conf file..

    Still immune to csplugin-events problems as the /etc/clearsync.d/csplugin-events.conf on all systems is still being deleted by a cron job. A CPU being driven to nearly 100% by this event file is not a trivial problem...
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 09:33 AM - #Permalink
    Resolved
    0 votes
    @Graham, I don't think you have a choice. It should just kill the process from running, i.e brutally stop it. It will not delete the service. You should then regain SSH access.

    I've just released the updates, but they won't come through automatically until tonight.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 09:11 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    @Graham, from earlier in the thread:
    yum update csplugin-events app-base --enablerepo=clearos-updates-testing
    I'll have to get the fixes released tonight. They did not finish compiling before I shut down last night.

    If you do manage to get in by ssh, you can shutdown clearsyncd with a:
    service clearsync stop
    Alternatively, from the webconfig, go Webconfig > System > Settings > Services and shut it down. Once you have done the update, you can start the service again.


    In my server I can't see System > Settings > Services. I do get Process Viewer by going to Reports > Performance And Resouces > Process Viewer and in their I get a running process list with a "Kill" button next to each one. I can find clearsyncd in there. When I hit "kill" it says "Confirm delete" and the process ID, will this stop the service or will it actually delete it, I certainly don't want to do that? If it means just stop teh service then I do.

    I won't press "OK" until you get back t me.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 08:48 AM - #Permalink
    Resolved
    0 votes
    @Graham, from earlier in the thread:
    yum update csplugin-events app-base --enablerepo=clearos-updates-testing
    I'll have to get the fixes released tonight. They did not finish compiling before I shut down last night.

    If you do manage to get in by ssh, you can shutdown clearsyncd with a:
    service clearsync stop
    Alternatively, from the webconfig, go Webconfig > System > Settings > Services and shut it down. Once you have done the update, you can start the service again.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 08:27 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    @Graham,
    That is a different issue. Which log file are you seeing that in? Are you using the AD Connector? Or do you have Audit Log enabled in a flexshare?

    Perhaps it would be a good idea to start a new thread and also post the output of "testparm -s"


    I can't remote in via SSH as it appears to be not responding I just get "Connection timed out" so can't send you the results of testparm -s

    Though I can get in via the web gui?

    Siv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 10 2019, 08:10 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    @Graham,
    That is a different issue. Which log file are you seeing that in? Are you using the AD Connector? Or do you have Audit Log enabled in a flexshare?

    Perhaps it would be a good idea to start a new thread and also post the output of "testparm -s"


    [/quote]
    I also am seeing this in the "Events and Notifications" section of the Web GUI:
    User root logged out via sudo 2019-04-10 09:06:16
    User root logged in via sudo 2019-04-10 09:06:15
    User root logged in via sudo 2019-04-10 09:06:13
    User root logged out via sudo 2019-04-10 09:06:13
    User root logged in via sudo 2019-04-10 09:06:00
    User root logged out via sudo 2019-04-10 09:06:00
    User root logged out via sudo 2019-04-10 09:05:57
    User root logged in via sudo 2019-04-10 09:05:56
    User root logged in via sudo 2019-04-10 09:05:01
    User root logged out via sudo 2019-04-10 09:05:01

    There are currenty 10,000 odd entries and it's occurring every second. I can access the server but it's obviously hammering it whatever "it" is.

    Siv
    The reply is currently minimized Show
Your Reply