Hi everyone,

I recently built a PC firewall running ClearOS to replace a Mikrotik hEX firewall I was using for my home network. Due to various services I run off my home server and so on, I starting experiencing problems with the hEX itself in terms of packet drop that couldn't be attributed to the internet connection itself or the LAN, so I decided it was time to replace it with something else. I already had hardware available to replace it and got everything up and running last night.

However, I noticed that both on the WAN interface (via internet speed test from the ClearOS interface and my PC), and on the LAN interface (via iperf throughput testing between my PC and the firewall), I am never exceeding ~100Mbit/s.

On the WAN to internet speed test, I am getting a read of about 95Mbit/s downlink, 25Mbit/s uplink. With the hEX and performing a speed test from my PC that would average about 380Mbit/s down and 25Mbit/s up.

On the LAN, I am seeing from iperf on send and receive 95Mbit/s. That is to say, without going over the internet, uplink and downlink to the ClearOS firewall I have built never exceeds 100Mbit/s. The obvious thing to check here of course, is that both links are negotiated to gigabit speeds and that there are no hard network errors. However, as you'll see below there are no errors, and both connections are negotiating with either my network switch or the WAN modem at gigabit rates.

WAN stats:

[root@rammstein ~]# ethtool enp3s0

Settings for enp3s0:

        Supported ports: [ TP MII ]

        Supported link modes:   10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Half 1000baseT/Full

        Supported pause frame use: No

        Supports auto-negotiation: Yes

        Supported FEC modes: Not reported

        Advertised link modes:  10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Half 1000baseT/Full

        Advertised pause frame use: Symmetric Receive-only

        Advertised auto-negotiation: Yes

        Advertised FEC modes: Not reported

        Link partner advertised link modes:  10baseT/Half 10baseT/Full

                                             100baseT/Half 100baseT/Full

                                             1000baseT/Full

        Link partner advertised pause frame use: No

        Link partner advertised auto-negotiation: Yes

        Link partner advertised FEC modes: Not reported

        Speed: 1000Mb/s

        Duplex: Full

        Port: MII

        PHYAD: 0

        Transceiver: internal

        Auto-negotiation: on

        Supports Wake-on: pumbg

        Wake-on: g

        Current message level: 0x00000033 (51)

                               drv probe ifdown ifup

        Link detected: yes

[root@rammstein ~]# ifconfig enp3s0

enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet xx.xx.xx.xx  netmask 255.255.224.0  broadcast 255.255.255.255

        inet6 fe80::d63d:7eff:fe4d:b5ae  prefixlen 64  scopeid 0x20<link>

        ether d4:3d:7e:4d:b5:ae  txqueuelen 1000  (Ethernet)

        RX packets 40362651  bytes 37118063810 (34.5 GiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 9551628  bytes 1011443861 (964.5 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

LAN stats:

[root@rammstein ~]# ethtool enp2s0f0

Settings for enp2s0f0:

        Supported ports: [ TP ]

        Supported link modes:   10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Half 1000baseT/Full

        Supported pause frame use: No

        Supports auto-negotiation: Yes

        Supported FEC modes: Not reported

        Advertised link modes:  10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Half 1000baseT/Full

        Advertised pause frame use: Symmetric

        Advertised auto-negotiation: Yes

        Advertised FEC modes: Not reported

        Link partner advertised link modes:  10baseT/Half 10baseT/Full

                                             100baseT/Half 100baseT/Full

                                             1000baseT/Full

        Link partner advertised pause frame use: No

        Link partner advertised auto-negotiation: Yes

        Link partner advertised FEC modes: Not reported

        Speed: 1000Mb/s

        Duplex: Full

        Port: Twisted Pair

        PHYAD: 1

        Transceiver: internal

        Auto-negotiation: on

        MDI-X: on

        Supports Wake-on: g

        Wake-on: d

        Current message level: 0x000000ff (255)

                               drv probe link timer ifdown ifup rx_err tx_err

        Link detected: yes

[root@rammstein ~]# ifconfig enp2s0f0

enp2s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 172.20.30.2  netmask 255.255.255.0  broadcast 172.20.30.255

        inet6 fe80::da9d:67ff:fe2d:50cc  prefixlen 64  scopeid 0x20<link>

        ether d8:9d:67:2d:50:cc  txqueuelen 1000  (Ethernet)

        RX packets 10023538  bytes 1782488963 (1.6 GiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 25216420  bytes 35852822180 (33.3 GiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        device interrupt 18

The WAN connection is using the motherboard built-in NIC, while the LAN connection is using one of the four ports on a Broadcom NetExtreme PCIe NIC. They don't use the same driver, but both appear to be plagued with the same issues:

02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

02:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

02:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

[root@rammstein ~]#

At this moment, I have configured some port forwarding rules using the port forwarding marketplace app, which required adding static "return" routes to various networks that are managed by my Catalyst 3750 switch. I added the routes to the appropriate /etc/sysconfig/network-scripts/routes-enp2s0f0 file, and also added the extra networks to the /etc/clearos/networks.conf file as documentation recommends. I have IDS and IPS marketplace apps installed, but I have disabled them for testing to no avail. Similarly, I have the free antivirus file scanning marketplace app installed, and similarly removed it for testing. No difference. All three apps have negligible impact on overall performance.

I have also installed the 1:1 NAT marketplace app, various logging marketplace apps, and the bandwidth control (QoS) marketplace app. The 1:1 NAT app I have not configured any rules for yet. The bandwidth control app I have similarly not configured any queues, nor have I set the bandwidth limits for the WAN connection because I am not able to get full use of my internet connection right now.

I checked the iptables rules from the console (including all the prerouting rules) and verified there's no throughput limits set there, to no avail. Running speed tests, using iperf, etc. I am not seeing any meaningful dent in CPU or memory resource utilization that would explain this. The hEX if I hook it back up exhibits none of these throughput issues (just the aforementioned packet drop problem, e.g. the reason I swapped it out). This replacement PC has an older AMD Phenom II x2 (dual core) CPU and 4GB of DDR3 RAM which I'd expect to be more than adequate for this task.

I don't see any funny messages in dmesg that would raise concern. I can say for certain the connection instability I used to face with the old hEX firewall is gone, but I am only getting use of a fraction of my WAN connection. My LAN networks are all routed between one another by my switch so it doesn't really hurt me there, but I might have to try a different solution if I can't figure out where the bottleneck is with ClearOS. Any ideas?

Thank you.