Forums

Resolved
0 votes
Hello

Testing community 7 release and all is nice until several days ago.

System is working. I can access from on and off-site via IP (No FQDN for this server as I am testing)

Keep getting connection errors:
--------------------------------------------------------
Warning
The following recommendations/warnings have been identified:

Consider enabling "clearos-updates" repository... it provides software updates to fix bugs, add features and in some cases, patch critical security vulnerabilities.


**Cannot perform this step as the Repository list never loads

---------------------------------------------------

All modules "Additional Info - Connection Failure"

--------------------------------------------------

Events Module has dozens of this:
Software repository configuration warning(s)

--------------------------------------------------------


Thanks
Ken
Wednesday, March 06 2019, 06:03 PM
Share this post:
Responses (6)
  • Accepted Answer

    Wednesday, March 06 2019, 09:07 PM - #Permalink
    Resolved
    0 votes
    It would seem there are repository issues on COS' end or a database error of some sort. I'm getting repository errors in my system log and can't access it in my Webconfig. I get a blank browser with this text in it:

    Ooooops: error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery


    This also means my subscription services aren't getting updated. In fact, logging into my ClearCenter account shows my last updates to be a week ago. However, my Webconfig shows my last updates over a month ago, which doesn't jive with what my ClearCenter account shows. Below is an excerpt from my log, which appears to have been an issue for a week now. Didn't notice until you said something. Following that is what shows up when I try a yum update from shell.

    Man I really hope this isn't a database issue again after the proxy report db issue I had a month ago that Nick and Dave helped me through. My mariadb and system-mariadb logs don't show any issues.

    system log excerpt:

    Mar  6 15:41:31 gateway engine: exception: debug backtrace: GUI (0): get_app_details
    Mar 6 15:41:31 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array
    Mar 6 15:41:31 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once
    Mar 6 15:41:55 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: GUI (0): get_app_details
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array
    Mar 6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once
    Mar 6 15:44:00 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: GUI (0): get_app_details
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array
    Mar 6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once



    yum update
    error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    error: cannot open Packages index using db5 - (-30973)
    error: cannot open Packages database in /var/lib/rpm
    CRITICAL:yum.main:

    Error: rpmdb open failed
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, March 06 2019, 09:08 PM - #Permalink
    Resolved
    0 votes
    Are you by any chance behind a proxy? If so, have you configured it in the IP settings?

    What warnings are you getting in Webconfig > Cloud > Updates Software Repository.

    What you you get from a:
    yum clean all && yum update
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, March 06 2019, 09:31 PM - #Permalink
    Resolved
    0 votes
    In addition to what I posted below, every app in the Webconfig shows "connection failure" in red next to the Additional Info widget. Clicking on the "Details" button inside the widget gives me the same db error:
    error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery


    The thing I noticed when I looked within /var/lib/rpm are that most of the files have a changed date and time of 02/27/2019 6:37:49 AM, which is when the errors started showing up in my system log:

    Feb 27 06:30:04 gateway app-intrusion-protection-updates: Service is up-to-date.
    Feb 27 06:35:04 gateway app-antimalware-updates: Service requires an update.
    Feb 27 06:36:45 gateway events: software_updates - event occurred
    Feb 27 06:36:45 gateway events: software_updates - triggered hook: base
    Feb 27 06:36:46 gateway events: software_updates - triggered hook: central-management
    Feb 27 06:36:55 gateway antivirus: signatures in working order
    Feb 27 06:38:49 gateway events: software_updates - event occurred
    Feb 27 06:38:49 gateway events: software_updates - triggered hook: base
    Feb 27 06:38:50 gateway events: software_updates - triggered hook: central-management
    Feb 27 06:43:52 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): Command execution failed.
    Feb 27 06:43:52 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (198): execute
    Feb 27 06:43:52 gateway engine: exception: debug backtrace: /usr/clearos/apps/clearcenter/libraries/Application_Web_Service.php (180): install
    Feb 27 06:43:52 gateway engine: exception: debug backtrace: /usr/sbin/clearcenter-update (79): install_update
    Feb 27 06:43:54 gateway app-antimalware-updates: Service update failed: Software is not installed - clearsdn-antimalware
    Feb 27 09:57:01 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: GUI (0): get_app_details
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array
    Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once
    Feb 27 09:57:07 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
    Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute
    Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list
    Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list
    Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras
    Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details
    Feb 27 09:57:07 gateway engine: exception: debug backtrace: GUI (0): get_app_details


    I have auto updates disabled, so why would my rpm database have a changed date a week ago? I haven't done any yum updates on my own, albeit I have some ClearCenter subscriptions for antimalware, etc.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, March 06 2019, 09:34 PM - #Permalink
    Resolved
    0 votes
    My last post was directed at Ken.

    Weird. My VM has just worked fine and picked up updates to certbot and OpenVPN. I've pinged the devs anyway as I know they were doing some ClearOS6 repo changes, but I don't know anything about 7.x. Both my Community and Business update OK
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, March 06 2019, 10:24 PM - #Permalink
    Resolved
    0 votes
    OK, I went through a rebuild of rpmdb using the following steps. It seems to have fixed the issue as I was able to do a yum update and everything seems functional in my Webconfig.

    I followed this link
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, March 06 2019, 10:53 PM - #Permalink
    Resolved
    0 votes
    Hi and thanks for looking at this.

    Not sure of source of issue; I did have some DNS failures last week and I had to change the primary DNS on a different server to clear that up. That being said I have since changed back to original DNS server (provided by ISP) and all is good as far as the rest of my network is concerned.


    For my test ClearOS, I changed the primary DNS to 8.8.4.4 - while that didn't fix it out-of-the-box I did a reboot and then all connections restored.

    So we can mark this as resolved

    Thank you very much

    Ken
    The reply is currently minimized Show
Your Reply