Hi and thanks for looking at this.

Not sure of source of issue; I did have some DNS failures last week and I had to change the primary DNS on a different server to clear that up. That being said I have since changed back to original DNS server (provided by ISP) and all is good as far as the rest of my network is concerned.


For my test ClearOS, I changed the primary DNS to 8.8.4.4 - while that didn't fix it out-of-the-box I did a reboot and then all connections restored.

So we can mark this as resolved

Thank you very much

Ken

OK, I went through a rebuild of rpmdb using the following steps. It seems to have fixed the issue as I was able to do a yum update and everything seems functional in my Webconfig.

I followed this link

My last post was directed at Ken.

Weird. My VM has just worked fine and picked up updates to certbot and OpenVPN. I've pinged the devs anyway as I know they were doing some ClearOS6 repo changes, but I don't know anything about 7.x. Both my Community and Business update OK

In addition to what I posted below, every app in the Webconfig shows "connection failure" in red next to the Additional Info widget. Clicking on the "Details" button inside the widget gives me the same db error:

error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

The thing I noticed when I looked within /var/lib/rpm are that most of the files have a changed date and time of 02/27/2019 6:37:49 AM, which is when the errors started showing up in my system log:

Feb 27 06:30:04 gateway app-intrusion-protection-updates: Service is up-to-date.

Feb 27 06:35:04 gateway app-antimalware-updates: Service requires an update.

Feb 27 06:36:45 gateway events: software_updates - event occurred

Feb 27 06:36:45 gateway events: software_updates - triggered hook: base

Feb 27 06:36:46 gateway events: software_updates - triggered hook: central-management

Feb 27 06:36:55 gateway antivirus: signatures in working order

Feb 27 06:38:49 gateway events: software_updates - event occurred

Feb 27 06:38:49 gateway events: software_updates - triggered hook: base

Feb 27 06:38:50 gateway events: software_updates - triggered hook: central-management

Feb 27 06:43:52 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): Command execution failed.

Feb 27 06:43:52 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (198): execute

Feb 27 06:43:52 gateway engine: exception: debug backtrace: /usr/clearos/apps/clearcenter/libraries/Application_Web_Service.php (180): install

Feb 27 06:43:52 gateway engine: exception: debug backtrace: /usr/sbin/clearcenter-update (79): install_update

Feb 27 06:43:54 gateway app-antimalware-updates: Service update failed: Software is not installed - clearsdn-antimalware

Feb 27 09:57:01 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute

Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list

Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list

Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras

Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details

Feb 27 09:57:01 gateway engine: exception: debug backtrace: GUI (0): get_app_details

Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array

Feb 27 09:57:01 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once

Feb 27 09:57:07 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute

Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list

Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list

Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras

Feb 27 09:57:07 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details

Feb 27 09:57:07 gateway engine: exception: debug backtrace: GUI (0): get_app_details

I have auto updates disabled, so why would my rpm database have a changed date a week ago? I haven't done any yum updates on my own, albeit I have some ClearCenter subscriptions for antimalware, etc.

Are you by any chance behind a proxy? If so, have you configured it in the IP settings?

What warnings are you getting in Webconfig > Cloud > Updates Software Repository.

What you you get from a:

yum clean all && yum update

It would seem there are repository issues on COS' end or a database error of some sort. I'm getting repository errors in my system log and can't access it in my Webconfig. I get a blank browser with this text in it:

Ooooops: error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

This also means my subscription services aren't getting updated. In fact, logging into my ClearCenter account shows my last updates to be a week ago. However, my Webconfig shows my last updates over a month ago, which doesn't jive with what my ClearCenter account shows. Below is an excerpt from my log, which appears to have been an issue for a week now. Didn't notice until you said something. Following that is what shows up when I try a yum update from shell.

Man I really hope this isn't a database issue again after the proxy report db issue I had a month ago that Nick and Dave helped me through. My mariadb and system-mariadb logs don't show any issues.

system log excerpt:

Mar  6 15:41:31 gateway engine: exception: debug backtrace: GUI (0): get_app_details

Mar  6 15:41:31 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array

Mar  6 15:41:31 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once

Mar  6 15:41:55 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

Mar  6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute

Mar  6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list

Mar  6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list

Mar  6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras

Mar  6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details

Mar  6 15:41:55 gateway engine: exception: debug backtrace: GUI (0): get_app_details

Mar  6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array

Mar  6 15:41:55 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once

Mar  6 15:44:00 gateway engine: exception: error: /usr/clearos/apps/base/libraries/Shell.php (227): error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

Mar  6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/base/libraries/Yum.php (468): execute

Mar  6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1538): get_repo_list

Mar  6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (1567): _get_repository_list

Mar  6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/libraries/Marketplace.php (730): _get_common_extras

Mar  6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/apps/marketplace/controllers/ajax.php (81): get_app_details

Mar  6 15:44:00 gateway engine: exception: debug backtrace: GUI (0): get_app_details

Mar  6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/framework/system/core/CodeIgniter.php (535): call_user_func_array

Mar  6 15:44:00 gateway engine: exception: debug backtrace: /usr/clearos/framework/htdocs/app/index.php (224): require_once

yum update

error: db5 error(-30973) from dbenv->open: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

error: cannot open Packages index using db5 -  (-30973)

error: cannot open Packages database in /var/lib/rpm

CRITICAL:yum.main:



Error: rpmdb open failed