Community Forum

jorma erik
jorma erik
Offline
Resolved
1 votes
ClearOS release 7.5.0 (Final)

Two nights ago my system was automatically updated and I couldn't login as a regular user over SSH. I started looking into the issue and found out that the system was updated and OpenLDAP was left at a broken state after the update. I'm not sure if it's relevant, but my extrenal IP-address was changed recently (if there are some dependencies, I don't know where to edit those, even then this system shouldn't simply die to an external IP-address changing, right?)

from yum.log:
Jul 06 02:27:46 Updated: openldap-2.4.44-15.v7.x86_64


----

from messages:
Jul  7 01:30:34 gateway systemd: Starting OpenLDAP Server Daemon...
Jul 7 01:30:34 gateway prestart.sh: Configuration directory '/etc/openldap/slapd.d' does not exist.
Jul 7 01:30:34 gateway prestart.sh: Warning: Usage of a configuration file is obsolete!
Jul 7 01:30:34 gateway nslcd[1323]: [8f08b8] <passwd="ldap.ldap"> failed to bind to LDAP server ldap://127.0.0.1/: Can't contact LDAP server: Transport endpoint is not connected
Jul 7 01:30:34 gateway nslcd[1323]: [8f08b8] <passwd="ldap.ldap"> no available LDAP server found: Can't contact LDAP server: Transport endpoint is not connected
Jul 7 01:30:34 gateway systemd: slapd.service: control process exited, code=exited status=1
Jul 7 01:30:34 gateway systemd: Failed to start OpenLDAP Server Daemon.
Jul 7 01:30:34 gateway systemd: Unit slapd.service entered failed state.
Jul 7 01:30:34 gateway systemd: slapd.service failed.
Jul 7 01:30:35 gateway systemd: Removed slice User Slice of root.
Jul 7 01:30:35 gateway systemd: Stopping User Slice of root.
Jul 7 01:31:05 gateway clearsyncd[1127]: OpenLDAPOnlineEvent: sudo /usr/sbin/trigger openldap_online: 256


----

Some further troubleshooting, probably root cause of the issue, but I have no idea how to fix this:


$ slapd -h "ldap://127.0.0.1/" -u ldap -f "/etc/openldap/slapd.conf" -d 256
5b4055f3 @(#) $OpenLDAP: slapd 2.4.44 (Jul 4 2018 20:05:05) $
mockbuild@build64-1.clearsdn.local:/builddir/build/BUILD/openldap-2.4.44/openldap-2.4.44/servers/slapd
TLSMC: MozNSS compatibility interception begins.
tlsmc_convert: INFO: cannot open the NSS DB, expecting PEM configuration is present.
tlsmc_intercept_initialization: INFO: successfully intercepted TLS initialization. Continuing with OpenSSL only.
TLSMC: MozNSS compatibility interception ends.
TLS: could not use key file `/etc/openldap/certs/clearos-key.pem'.
TLS: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch x509_cmp.c:341
5b4055f3 main: TLS init def ctx failed: -1
5b4055f3 slapd stopped.

5b4055f3 connections_destroy: nothing to destroy.


----


$ openssl x509 -noout -text -in /etc/openldap/certs/clearos-key.pem
unable to load certificate
140197412693920:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:707:Expecting: TRUSTED CERTIFICATE

Saturday, July 07 2018, 06:15 AM
Share this post:

Accepted Answer

Sunday, July 08 2018, 03:07 PM - #Permalink
Resolved
1 votes
hi,
I manage some COS boxes; two of them have 7.4 , and last friday smb stopped authenticate users.
It seemed to be openldap service that didn't came up anymore. It was completely broken, and I could see 'Account Manager is offline' in WebConfig
After reinstalling one of them and see the same problem was back on newly uinstalled box I investigate updates.
so I came up with downgrading updated openldap components:


yum downgrade openldap openldap-servers openldap-clients


and everything was back. then I stopped autoupdates (hey: how one can achieve this through command line? should one simply disable them in crontab?)

let us know if this works for you too.
paolo
The reply is currently minimized Show
Responses (23)
  • Accepted Answer

    Friday, July 13 2018, 06:58 PM - #Permalink
    Resolved
    0 votes
    If you come across this bug, rather than downgrade, please can you copy in the system certificates again.:
    mkdir /etc/openldap/certs/old
    mv /etc/openldap/certs/*.pem /etc/openldap/certs/old
    cp /etc/pki/CA/ca-cert.pem /etc/openldap/certs/clearos-ca-cert.pem
    cp /etc/pki/CA/sys-0-cert.pem /etc/openldap/certs/clearos-cert.pem
    cp /etc/pki/CA/private/sys-0-key.pem /etc/openldap/certs/clearos-key.pem
    chgrp ldap /etc/openldap/certs/*.pem
    systemctl start slapd.service
    This makes a backup of the current certificates from /etc/openldap/certs to /etc/openldap/certs/old.

    You should just be able to copy and paste the whole block of commands.

    If anyone does come across the bug, the devs would love remote access to debug the issue.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, July 12 2018, 03:24 PM - #Permalink
    Resolved
    0 votes
    For sure this bug is triggered by an update but since only a few are affected, it is hard to know the cause. We have also not been able to reproduce the bug and since the workaround is to not update for those affected, we cannot troubleshoot it without willing participants. Without more data we may not be able to move forward with any fixes for those affected and they will, unfortunately, be resigned to a state of not being able to get updates.

    The weekend will afford those using Community in business environments to please move forward and apply the update so that we can troubleshoot the issue. If you are willing to work personally on the issue, please submit a ClearCARE support ticket (preferably with remote credentials) and reference this article. We are not opposed to running a different build of OpenLDAP if that is required but this bug is getting stale without a resolution in site.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, July 11 2018, 09:33 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    Please send the backup but it is from before the system was upgraded. Peter can probably restore it then upgrade and see what happened.

    The issue you linked to should have been solved as it was a back end problem but I do note that today the clearcenter.com is mainly down, occasionally allowing a connection. I don't know if that will cause any problems. My webconfig is still responsive on my production box, but I can't register a system with the test iso.

    Hi Nick
    I'm pretty sure I kept some config backups before and after the system update.
    Unfortunately i'm busy and i'll look forward to post files as soon as i can
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, July 11 2018, 09:18 AM - #Permalink
    Resolved
    0 votes
    Please send the backup but it is from before the system was upgraded. Peter can probably restore it then upgrade and see what happened.

    The issue you linked to should have been solved as it was a back end problem but I do note that today the clearcenter.com is mainly down, occasionally allowing a connection. I don't know if that will cause any problems. My webconfig is still responsive on my production box, but I can't register a system with the test iso.
    The reply is currently minimized Show
  • Accepted Answer

    Riven
    Riven
    Offline
    Wednesday, July 11 2018, 09:04 AM - #Permalink
    Resolved
    0 votes
    Hi Peter.

    I can send a backup that I saved off the 2nd June.
    I already formatted the system as I am totally out of ideas (client on my neck) and went the Paid Business edition route with this system, but having issues there as well, but that is more relating to another thread that I will continue there.
    https://www.clearos.com/clearfoundation/social/community/webconfig-slow-performance/oldest#filter-sort

    Dirk
    The reply is currently minimized Show
  • Accepted Answer

    PeterB
    PeterB
    Offline
    Tuesday, July 10 2018, 07:00 PM - #Permalink
    Resolved
    0 votes
    This is an interesting edge case. If you have this issue, please send us a backup file from the "Configuration Backup" app -- it would be much appreciated. You can send the file to developer@clearfoundation.com.

    Alternatively, we can also investigate over SSH. You can submit the system credentials via your ClearCenter account @ https://secure.clearcenter.com/portal/system_password.jsp

    [edit by Nick]
    As you don't have a subscription you will have to raise the ticket as a "General Enquiry" at https://secure.clearcenter.com/portal/ticket_manage.jsp
    Please reference the ticket number when you submit your credentials.
    [/edit]
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, July 10 2018, 11:40 AM - #Permalink
    Resolved
    0 votes
    As a thought, is it worth moving the nssdb files out of the way and then trying to start slapd? The files to move from /etc/openldap/certs are:
    .slapd-leave
    cert8.db
    key3.db
    secmod.db
    password
    Perhaps put them into /etc/openldap/certs/temp. You can always move them back afterwards if it does not help.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, July 10 2018, 09:52 AM - #Permalink
    Resolved
    0 votes
    Have you set the Webconfig > Server > Directory > Directory Server > Policy to anything other than "Not Published"

    Are you showing the same symptoms as the O/P when you run it in interactive mode:
    service slapd stop
    slapd -h "ldap://127.0.0.1/" -u ldap -f "/etc/openldap/slapd.conf" -d 256


    @Dave, is it possible for the nssdb and certs/key in /etc/openldap/certs to get out of sync?

    @Riven, do you need that machine immediately? I will (hopefully) speak to Dave this evening and ask him what he would like to do about it. He may request remote access.
    The reply is currently minimized Show
  • Accepted Answer

    Riven
    Riven
    Offline
    Tuesday, July 10 2018, 09:21 AM - #Permalink
    Resolved
    0 votes
    I am having the same problem on one of my machines.
    It has ClearOS 7.5 on it and all issues started after the update the 6/7th
    The reply is currently minimized Show
  • Accepted Answer

    Monday, July 09 2018, 03:13 PM - #Permalink
    Resolved
    0 votes
    Nick,

    I don't see how to recreate this bug. It obviously didn't show up when I went through the testing before the release nor in the testing post-community release for the samba bug.

    I'll ping Peter on this as well.
    The reply is currently minimized Show
  • Accepted Answer

    Monday, July 09 2018, 11:14 AM - #Permalink
    Resolved
    0 votes
    Interesting bug on el7 here which leads to here. I'll have to ask the devs on this one. t looks like the server I set up in November to test the 7.4 upgrade (and may be a pure 7.4 installation) also has an NSS database. I wonder why the devs switched and if it was intentional - the recommendation seems to be to not use nssdb.
    The reply is currently minimized Show
  • Accepted Answer

    Monday, July 09 2018, 10:11 AM - #Permalink
    Resolved
    0 votes
    That is odd. Your set up has created what looks like an nss database. All have is:
    [root@7 ~]# ls -la /etc/openldap/*certs
    /etc/openldap/cacerts:
    total 12
    drwxr-xr-x. 2 root root 35 Apr 8 20:07 .
    drwxr-xr-x. 5 root root 4096 Jul 6 19:53 ..
    -rw-r----- 1 root ldap 1009 Aug 1 2015 cert.pem
    -rw-r----- 1 root ldap 887 Aug 1 2015 key.pem

    /etc/openldap/certs:
    total 20
    drwxr-xr-x. 2 root root 77 Jul 5 03:06 .
    drwxr-xr-x. 5 root root 4096 Jul 6 19:53 ..
    -rw-r-----. 1 root ldap 1545 Oct 25 2017 clearos-ca-cert.pem
    -rw-r-----. 1 root ldap 4756 Oct 25 2017 clearos-cert.pem
    -rw-r-----. 1 root ldap 1704 Oct 25 2017 clearos-key.pem


    I wonder if this is a 7.4 installation issue?

    FWIW, the samba share issue is possibly a different cause. Please see this post. Note if you do not have the parameter in smb.conf you don't need to add it.
    The reply is currently minimized Show
  • Accepted Answer

    Monday, July 09 2018, 10:01 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    Odd, this one as only some people are getting it. Can you post your /etc/openldap/slapd.conf? If you have a setup which came from 6.x it appears to be pulling certificates from /etc/openldap/cacerts. If you have a vanilla set up, it puls them from /etc/openldap/certs. Either way it does not explain the problem of why it works with the old version of openssl and not the new.

    dear Nick, firstly, thank you for watching this ...

    as I wrote yesterday I had this in a fresh installed 7.4 box. It happened on Friday morning: the customer called me saying samba shares were not reachable anymore, and he was asked for authentication, but no users could reach his files and folders; after trying out to solve, I decided the server will be sooner functional reinstalling it (i rsync data and home dirs very often on another HDD). So I got the box, pulled out HDDs and installed a freshly downloaded 7.4 DVD image.
    Then I re-entered the few users and groups and retored back data. Restored sync and backup scripts and we were happy to be functional again.
    The day after I was called by the customer: he was back without access. Once i saw the problem was once again in slapd daemon, I read logs and I realized openldap components were updated during the night.

    Anyway here you are certs directory ownership:
    -rw-r--r--. 1 root root 65536  6 lug 16.13 cert8.db
    -rw-r----- 1 root ldap 1363 6 lug 16.47 clearos-ca-cert.pem
    -rw-r----- 1 root ldap 1363 6 lug 16.47 clearos-cert.pem
    -rw-r-----. 1 root ldap 1679 6 lug 16.17 clearos-key.pem
    -rw-r--r--. 1 root root 16384 6 lug 16.13 key3.db
    -r--r-----. 1 root ldap 45 6 lug 16.12 password
    -rw-r--r--. 1 root root 16384 6 lug 16.12 secmod.db
    -rw-r--r--. 1 root root 0 6 lug 16.13 .slapd-leave


    and this one is slapd.conf
    # Schemas
    # Schemas
    #----------------------------------------------------------

    # Core schemas
    include /etc/openldap/schema/core.schema
    include /etc/openldap/schema/cosine.schema
    include /etc/openldap/schema/inetorgperson.schema

    # ClearFoundation base
    include /etc/openldap/schema/rfc2307bis.schema
    include /etc/openldap/schema/clearfoundation.schema

    # ClearCenter extension
    include /etc/openldap/schema/clearcenter.schema

    # Password policy extension
    include /etc/openldap/schema/ppolicy.schema

    # RADIUS extension
    include /etc/openldap/schema/RADIUS-LDAPv3.schema

    # Kolab extension
    include /etc/openldap/schema/rfc2739.schema
    include /etc/openldap/schema/kolab2.schema

    # Horde extension
    include /etc/openldap/schema/horde.schema

    # Samba extension
    include /etc/openldap/schema/samba3.schema

    # OwnCloud
    include /etc/openldap/schema/owncloud.schema

    # Zarafa extension
    include /etc/openldap/schema/zarafa.schema

    # Kopano extension
    include /etc/openldap/schema/kopano.schema


    # Global configuration directives
    #----------------------------------------------------------

    pidfile /var/run/openldap/slapd.pid
    argsfile /var/run/openldap/slapd.args

    TLSCACertificateFile /etc/openldap/certs/clearos-ca-cert.pem
    TLSCertificateFile /etc/openldap/certs/clearos-cert.pem
    TLSCertificateKeyFile /etc/openldap/certs/clearos-key.pem

    defaultsearchbase "dc=system,dc=lan"

    allow bind_v2

    loglevel 0

    sizelimit 10000

    moduleload accesslog.la
    moduleload ppolicy.la
    moduleload memberof.la
    moduleload syncprov.la


    # Monitor database
    #----------------------------------------------------------

    database monitor


    # Accesslog database
    #----------------------------------------------------------

    database bdb
    directory /var/lib/ldap/accesslog
    suffix cn=accesslog
    rootdn cn=accesslog
    index default eq
    index entryCSN,objectClass,reqEnd,reqResult,reqStart

    overlay syncprov
    syncprov-nopresent TRUE
    syncprov-reloadhint TRUE

    access to *
    by self write
    by peername.ip=127.0.0.1 read
    by * none stop


    # Primary database
    #----------------------------------------------------------

    database bdb
    directory /var/lib/ldap
    suffix "dc=system,dc=lan"
    rootdn "cn=manager,ou=Internal,dc=system,dc=lan"
    rootpw "{SSHA}xyz1234567890.........."

    cachesize 20000
    checkpoint 512 5
    idlcachesize 20000
    idletimeout 300
    dirtyread

    index default sub
    index entryCSN eq
    index entryUUID eq
    index objectClass pres,eq
    index uid approx,pres,sub,eq
    index displayName pres,sub,eq
    index uidNumber eq
    index gidNumber eq
    index memberUID eq
    index member eq,pres
    index mail approx,sub,pres,eq
    index cn approx,sub,pres,eq
    index sn approx,sub,pres,eq
    index givenName approx,sub,pres,eq

    # Contact extension
    index clearMailAliases approx,sub,pres,eq

    # Samba extension
    index sambaSID eq,sub
    index sambaSIDList eq
    index sambaPrimaryGroupSID eq
    index sambaDomainName eq
    index sambaGroupType eq

    # Kolab extension
    index kolabDelegate approx,sub,pres,eq
    index kolabHomeServer pres,eq
    index kolabDeleteflag pres,eq

    # password policies
    overlay ppolicy

    # memberOf
    overlay memberof

    # syncrepl provider for primary database
    overlay syncprov
    syncprov-checkpoint 100 5

    # accesslog overlay definitions for primary database
    overlay accesslog
    logdb cn=accesslog
    logops writes
    logsuccess TRUE
    logpurge 32+00:00 01+00:00

    # syncuser granted limitless searches
    limits dn.exact="cn=syncuser,ou=Internal,dc=system,dc=lan" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited

    access to *
    by dn="cn=syncuser,ou=Internal,dc=system,dc=lan" write
    by * break

    access to attrs=userPassword,clearMicrosoftNTPassword,clearSHA1Password,clearSHAPassword,sambaLMPassword,sambaNTPassword
    by self =wx
    by anonymous =x
    by * none stop

    access to attrs=mail
    by * read stop

    access to attrs=uid
    by * read stop

    access to *
    by self write
    by peername.ip=127.0.0.1 read
    by * none stop
    The reply is currently minimized Show
  • Accepted Answer

    Monday, July 09 2018, 09:50 AM - #Permalink
    Resolved
    0 votes
    Please can you also check file ownership on your certs and cert folders:
    ls -la /etc/openldap/*certs
    The reply is currently minimized Show
  • Accepted Answer

    Monday, July 09 2018, 09:37 AM - #Permalink
    Resolved
    0 votes
    Odd, this one as only some people are getting it. Can you post your /etc/openldap/slapd.conf (remove the rootpw entry before posting)? If you have a setup which came from 6.x it appears to be pulling certificates from /etc/openldap/cacerts. If you have a vanilla set up, it pulls them from /etc/openldap/certs. Either way it does not explain the problem of why it works with the old version of openssl and not the new.
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, July 08 2018, 09:08 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:

    Paolo MACOR wrote:
    (hey: how one can achieve this through command line? should one simply disable them in crontab?)
    You don't need to do that. You could add:
    exclude=openldap*
    to /etc/yum.conf. Autoupdates would then work excluding those packages.


    thank you, Nick.
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, July 08 2018, 07:32 PM - #Permalink
    Resolved
    0 votes
    Paolo MACOR wrote:
    (hey: how one can achieve this through command line? should one simply disable them in crontab?)
    You don't need to do that. You could add:
    exclude=openldap*
    to /etc/yum.conf. Autoupdates would then work excluding those packages.
    The reply is currently minimized Show
  • Accepted Answer

    jorma erik
    jorma erik
    Offline
    Sunday, July 08 2018, 04:57 PM - #Permalink
    Resolved
    0 votes
    Paolo MACOR wrote:

    hi,
    I manage some COS boxes; two of them have 7.4 , and last friday smb stopped authenticate users.
    It seemed to be openldap service that didn't came up anymore. It was completely broken, and I could see 'Account Manager is offline' in WebConfig
    After reinstalling one of them and see the same problem was back on newly uinstalled box I investigate updates.
    so I came up with downgrading updated openldap components:


    yum downgrade openldap openldap-servers openldap-clients


    and everything was back. then I stopped autoupdates (hey: how one can achieve this through command line? should one simply disable them in crontab?)

    let us know if this works for you too.
    paolo


    Thanks a lot Paolo! I also downgraded OpenLDAP and it started working immediately.

    Looks like the update is somehow broken.
    The reply is currently minimized Show
  • Accepted Answer

    jorma erik
    jorma erik
    Offline
    Saturday, July 07 2018, 01:14 PM - #Permalink
    Resolved
    0 votes

    If you wanted to, you could pull the certificates out of the backup manually.

    Have you checked the integrity of your current certificates and keys from the link I posted?


    Not sure if these are the correct ones; however I checked also keys at: /etc/openldap/cacerts and the md5 hashes were the same two as shown below. I also extracted the oldest backup archive, it seemed to contain only keys located at /etc/openldap/cacerts and they seemed to be identical.

    openssl x509 -noout -modulus -in /etc/openldap/certs/clearos-cert.pem | openssl md5
    (stdin)= 513fc49760645b47d9c814a375068d0d

    openssl rsa -noout -modulus -in /etc/openldap/certs/clearos-key.pem | openssl md5
    (stdin)= e8c7a2f91b637c014da91bcf586d1b28


    I haven't done anything to the keys myself. The installation is fairly stock I have only installed
    screen
    and
    fail2ban
    .

    Might have been that the cert warning I saw in the logs got me sidetracked, maybe the explanation is something more simple.

    Things I've tried so far:

    restoring from an older backup (restore failed)
    rebooting the server (didn't help)
    /usr/bin/db_recover -v -h /var/lib/ldap/

    The reply is currently minimized Show
  • Accepted Answer

    Saturday, July 07 2018, 12:02 PM - #Permalink
    Resolved
    0 votes
    jorma erik wrote:

    Nick Howitt wrote:
    Do you by any chance have a /etc/openldap/slapd.d folder?

    The directory doesn't seem to exist.

    That is good. It was an older upgrade problem.


    Nick Howitt wrote:
    You should have backup copies in the backup files in /var/clearos/configuration_backup. Can I suggest you save a copy from prior to the update locally as this is only a rolling 10 days of files.

    Took one of the older ones. Prior to posting here I already tried to restore from a backup, but the restoration also fails, presumably due to not being able to start the OpenLDAP.
    If you wanted to, you could pull the certificates out of the backup manually.

    Have you checked the integrity of your current certificates and keys from the link I posted?
    The reply is currently minimized Show
  • Accepted Answer

    jorma erik
    jorma erik
    Offline
    Saturday, July 07 2018, 10:17 AM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:
    Do you by any chance have a /etc/openldap/slapd.d folder?

    The directory doesn't seem to exist.


    Nick Howitt wrote:
    You should have backup copies in the backup files in /var/clearos/configuration_backup. Can I suggest you save a copy from prior to the update locally as this is only a rolling 10 days of files.

    Took one of the older ones. Prior to posting here I already tried to restore from a backup, but the restoration also fails, presumably due to not being able to start the OpenLDAP.
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, July 07 2018, 10:05 AM - #Permalink
    Resolved
    0 votes
    Do you by any chance have a /etc/openldap/slapd.d folder?
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, July 07 2018, 09:15 AM - #Permalink
    Resolved
    0 votes
    I am not sure what you are trying to do with the key file, but certificates have the "x509" in the openssl command and keys have "rsa" so:
    openssl rsa -noout -text -in /etc/openldap/certs/clearos-key.pem
    or
    openssl rsa  -in /etc/openldap/certs/clearos-key.pem -check


    FWIW these both work on my old Community server. You should have backup copies in the backup files in /var/clearos/configuration_backup. Can I suggest you save a copy from prior to the update locally as this is only a rolling 10 days of files.

    To check certificate integrity go to the Debugging Using OpenSSL section of this link.
    The reply is currently minimized Show
Your Reply