I've continued to check this out and haven't been able to figure anything more until today.
I noticed in the /etc/clearos/firewall.d/local there is a line and nothing else.
Also there is a /etc/clearos/firewall.d/90-attack-detector file with a bunch of code.
Is it possible that the 90-attack-detector has started fail2ban and local tries to start fail2ban causing it to hang?
Thanks again for your help and patience as I try to figure this out.
Nick Howitt wrote:
It looks like a firewall restart has happened and this has wiped a lot of f2b firewall bits. Why it restarted, I don't know. When that happens, when f2b tries to shut down, lots of its rules which it wants to delete no longer exist generating those errors. Ignore them.
It restarted when I added an IP to the Firewall:Incoming list in the Firewall App. This happens each and every time I add an IP to the list. It isn't in the custom firewall menu.
Nick Howitt wrote:Correct me if I'm wrong, but this is ClearOS 7. Do you have app-attack-detector installed any more?
Yes, this is COS 7.x. I reinstalled app-attack-detector from the MarketPlace after I realized that the problems that I was having was with using systemd log. I left my changes in the jail.local but everything else is standard.
Nick Howitt wrote:What firewall files do you have related to fail2ban or attack-detector in /etc/clearos/firewall.d/?
I haven't added anything to the firewall.d manually here is the list:
Nick Howitt wrote:Also what is the contents of /etc/clearos/firewall.d/custom and /etc/clearos/firewall.d/local, if any.
Nick Howitt wrote:AFAIK, the firewall only restarts when deleting a rule, not when adding it.
In my case it restarts after adding a rule. I haven't tried deleting any rules.
It should be able to delete a rule without restarting too.
Nick Howitt wrote:Although it restarts when you change and save /etc/clearos/firewall.d/local, so it may do the same with /etc/clearos/firewall.d/custom, which I think is what you are effectively editing.
I haven't a clue where the block rules that are added in the Firewall:Incoming are added. But each time I add one, the firewall restarts and kills fail2ban.
On COS5.2, I had a list of IPs that I blocked because they were a pest. It was some sort of local file rc.firewall.local that had a list of commands with IPTABLES -A INPUT -s ipaddress -j -DROP. I used to manually add the IPs there so they would reblock upon a system restart. The IPTABLES command would add the block manually without restarting the firewall. I don't know where to add these now with this new firewall.d.
So here is what happened when I added an ip address to the Network:Firewall:Incoming Firewall:Blocked Incoming Connections (Add).
The firewall has restarted as far as I can tell. Attack Detector shows "Restarting" but isn't. [So here I am full circle back to the start]
Please note: at 15:01 I added DEBUG to the log level and did a condrestart. It is now 15:41 and attack detector hasn't started.
If I do it hangs. If I do it starts again.
Nick Howitt wrote:
There is no way adding a firewall rule should stop f2b.
I am wondering how much work it would be to reload ClearOS? Brutal but it may be the best. Otherwise we can start again with the troubleshooting now netify has gone.
I really don't want to reinstall COS. My installation, now 5 months old, is only COS + a few Marketplace apps. Getting websites up, mail running etc. etc. will take days that I don't have time for at the moment. I haven't spent any time doing any customizations like I did with COS5.2. It's a vanilla install. Interestingly enough this all started this year around the time of the firmware updates due to the Intel security issue. Before that, everything ran OK.
I left the logging for f2b as default but will change that right after I post this message.
While I agree that adding a firewall rule shouldn't stop f2b, it appears that the behaviour is something like this: add a firewall rule, restart the firewall, restart netify, restart attack detector etc.
I agree it should be "just add this ip address to the block list". Is there a way to trace the firewall add rule behaviour in detail?
Since netify was hanging (I have a strong suspicion that it hung at the reference to the non-existant dat file ...) and systemd never got a message that netify hadn't started, I think the restart of the attack detector timed out. ???
I've finally got all the protocol filter and netify stuff removed.
I'm still seeing a problem with the Attack Detector/fail2ban.
If I add an IP to the block list on the incoming firewall rules, then the Attack Detector shuts down and hangs on the restart. I think the shut down and restart makes sense but I'm still at a loss why the Attack Detector won't restart.
It feels like I've spent days trying to figure this out and end up back in the same place. Aaaaaaahhhhhh.
Figured it out.
listed all the installed files. These were removed manually.
Then I ran which removed everything else. It looks like it also cleaned up the turds in yum from app-netify-fwa-core.
Thanks again for all your help Nick.
Thanks again for the help, Nick.
The never completes. I've forced quit and run but I can't remove those two apps.
How does one go about determining all the pieces of this app that were installed so I can manually delete the files?
One more potential issue.
In it looks like the permissions are inconsistent.
So we have 10-netify-fwa & 10-snortsam as 644 and the rest are 755.
Would this be causing a problem with the starting of netify-fwa?
If yes, then which permissions should it be?
I'm still having a bunch of problems with this netify-fwa.
I removed both the Protocol filter and the Application filter but the netify-fwa and netifyd are still present. (I thought they should be removed when I removed Protocol and Application filters???)
Each time I make any change to the firewall, netify-fwa tries to restart and clobbers the Attack Defender (fail2ban).
netify-fwa hangs and times out.
Having read some more posts regarding netify and what should be installed and where, I may have found a bug or inconsistency with my install.
The file contains:
According to the github for netify, the state.dat file was removed. There was a post on this forum about the state.dat file causing problems. I checked in /var/lib/netify-fwa/ and there is no state.dat file. There is however a nfa_protocols.dat. Since the state.dat file shouldn't exist and is listed in the conf file, I think there is a bug.
Either the shouldn't be there, the right line should be or some other reference should be to nfa_prototcols.dat.
1) can I safely remove netify-fwa and netifyd since I have removed both the Protocol and Application filters?
2) is the netify-fwa.conf in error?
And one more update.
I wanted to remove the protocol filter but could figure out what it was called in the marketplace. Ended up on the Application filter which is also stopped. Tried to start the Application filter (which I was using) and it killed the Attack Detector too. This Gateway Management Community is running though.