I'll start by saying I have no experience of multiwan, but no one else is responding.

1 - Have you tried connecting another device directly to the Kaiomy modem to see if it works on its own?
2 - have you fully configured the WAN interface manually? What is the output of:

ifconfig | grep HWa -A 1

cat /etc/resolv-peerdns.conf

3 - What happens if you try to bring up the interface manually ("ifup eth2")
4 - Do the lights on the modem and ClearOS NIC indicate a connection is present? Does either end support MDI-X or do you need a crossover cable?
5 - What is the output of:

lspci -k | grep eth -A 3

uname -r

Hello Nick, thanks for the reply.

1. Yes, the modem work fine when I connect a workstation directly to one LAN port.

2.

[root@gateway ~]# ifconfig | grep HWa -A 1
eth0 Link encap:Ethernet HWaddr BC:AE:C5:50:7D:30
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
--
eth1 Link encap:Ethernet HWaddr F8:1A:67:04:4E:F5
inet addr:192.168.20.201 Bcast:192.168.20.255 Mask:255.255.255.0
--
eth2 Link encap:Ethernet HWaddr B0:48:7A:83:21:E4
inet addr:192.168.10.2 Bcast:192.168.10.255 Mask:255.255.255.0
--
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.8.0.1 P-t-P:10.8.0.2 Mask:255.255.255.255
--
tun1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.8.10.1 P-t-P:10.8.10.2 Mask:255.255.255.255


[root@gateway ~]# cat /etc/resolv-peerdns.conf
; generated by /sbin/dhclient-script
nameserver 192.168.10.1
nameserver 192.168.20.1


3.The eth2 interface is up. What I think is causing the issue is that like it is "offline" / "not in use" by the multiwan, then eth2 has no network traffic, but the interface is up.

If I try to ping from eth2 to the modem, I have nothing.

[root@gateway ~]# ping -I 192.168.10.2 192.168.10.1
PING 192.168.10.1 (192.168.10.1) from 192.168.10.2 : 56(84) bytes of data.
From 192.168.10.2 icmp_seq=2 Destination Host Unreachable
From 192.168.10.2 icmp_seq=3 Destination Host Unreachable
From 192.168.10.2 icmp_seq=4 Destination Host Unreachable


4. The lights are correct and the connection is present.

5. The lspci returns nothing.

[root@gateway ~]# uname -r
2.6.32-573.26.1.v6.x86_64

2 - I was hoping to see the gateway as well and I can't remember if you have to do the command with a "-A 2" instead or if you have to print the routing table.
5 - My typo. Please do "lspci -k | grep Eth -A 3".

[edit]
Is eth2 defined as external?

cat /etc/clearos/network.conf

[/edit]

oh, ok

2. with -A 2 it shows nothing about the gateway

this is what it show with "ip route", I dont know if there is another command for the routing table.

[root@gateway ~]# ip route
10.8.10.2 dev tun1 proto kernel scope link src 10.8.10.1
10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1
192.168.20.0/24 dev eth1 proto kernel scope link src 192.168.20.201
10.8.0.0/24 via 10.8.0.2 dev tun0
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1
10.8.10.0/24 via 10.8.10.2 dev tun1
192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.2
default via 192.168.20.1 dev eth1


5.

[root@gateway ~]# lspci -k | grep Eth -A 3
01:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0)
Subsystem: ASUSTeK Computer Inc. Device 847e
Kernel driver in use: atl1c
Kernel modules: atl1c
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Subsystem: TP-LINK Technologies Co., Ltd. TG-3468 Gigabit PCI Express Network Adapter
Kernel driver in use: r8169
Kernel modules: r8169
03:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8169/8110 Family PCI Gigabit Ethernet NIC
Kernel driver in use: r8169
Kernel modules: r8169

[edit]
I think so,

[root@gateway ~]# cat /etc/clearos/network.conf
# Network mode
MODE="gateway"

# Network interface roles
EXTIF="eth1 eth2"
LANIF="eth0"
DMZIF=""
HOTIF=""

# Domain and Internet Hostname
DEFAULT_DOMAIN="agrobledo.lan"
INTERNET_HOSTNAME="gateway.agrobledo.lan"

# Extra LANS
EXTRALANS=""

[/edit]

You do have one issue and I hope that your RTL8111/8168/8411 NIC is eth2. Although the r8169 driver appears to work with this card, it can give all sorts of problems. Please can you download and install the kmod-r8168 and kmod-r8169 driver from here then reboot. An "lspci -k | grep Eth -A 3" should then show the r8168 driver being used for the RTL8111/8168/8411 NIC.

All other output that I know is OK. I am not sure about the routing table.

I am not convinced about your DNS settings and they may cause a slowdown (each lookup to 192.168.10.1 would need to timeout before it fails over to using 192.168.20.1). I would recommend you set your DNS to a public DNS service such as OpenDNS or GoogleDNS. However, that should not be the cause of your current problem.

OK,

I have installed de r8168 driver. Now this is the lspci command

[root@gateway ~]# lspci -k | grep Eth -A 3
01:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0)
Subsystem: ASUSTeK Computer Inc. Device 847e
Kernel driver in use: atl1c
Kernel modules: atl1c
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Subsystem: TP-LINK Technologies Co., Ltd. TG-3468 Gigabit PCI Express Network Adapter
Kernel driver in use: r8168
Kernel modules: r8168
03:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8169/8110 Family PCI Gigabit Ethernet NIC
Kernel driver in use: r8169
Kernel modules: r8169

Sadly, the issue persists.

Multiwan page still shows eth2 as offline - not in use.

[edit]

by the way, eth2 is r8169

[/edit]

Can you please provide the output for cat /etc/sysconfig/network-scripts/ifcfg-eth[1,2]
Also the full output from ifconfig for the two external interfaces...
and lastly an extract from /var/log/syswatch from "service syswatch restart" until it settles down

Your answer to Tony's request may clarify eth2's config - the issue could be that eth2 does not have an internet gateway i.e. it is not an external interface. When it was bridged to the PPPoE interface it had the gateway

Thank you Tony and Duncan for replying...

This is Tony's request:

[root@gateway ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE=eth2
TYPE="Ethernet"
ONBOOT="yes"
USERCTL="no"
BOOTPROTO="static"
IPADDR="192.168.10.2"
NETMASK="255.255.255.0"
GATEWAY="192.168.10.1"

[root@gateway ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
TYPE="Ethernet"
ONBOOT="yes"
USERCTL="no"
BOOTPROTO="static"
HWADDR="F8:1A:67:04:4E:F5"
DNS1="192.168.20.1"
DNS2="8.8.8.8"
IPADDR="192.168.20.201"
NETMASK="255.255.255.0"
GATEWAY="192.168.20.1"

Note: yesterday I manually added HWADDR, DNS1 and DNS2 in both files, then restarted network but nothing happened. Then I this other tests so the eth2 file was regenerated, that is why you don't see those lines in the eth2 file now. Eth1 is working fine as usual.

ifconfig full output:

eth1 Link encap:Ethernet HWaddr F8:1A:67:04:4E:F5
inet addr:192.168.20.201 Bcast:192.168.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23150537 errors:0 dropped:0 overruns:0 frame:0
TX packets:385889 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2673715256 (2.4 GiB) TX bytes:71351441 (68.0 MiB)
Interrupt:28 Base address:0x2000

eth2 Link encap:Ethernet HWaddr B0:48:7A:83:21:E4
inet addr:192.168.10.2 Bcast:192.168.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10788 errors:0 dropped:0 overruns:0 frame:0
TX packets:17900 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1287085 (1.2 MiB) TX bytes:789296 (770.7 KiB)
Interrupt:16 Base address:0xcc00

this is syswatch log since restart:

Fri Nov 25 10:22:50 2016 info: system - syswatch terminated
Fri Nov 25 10:22:50 2016 info: system - syswatch started
Fri Nov 25 10:22:50 2016 info: config - IP referrer disabled in multi-WAN
Fri Nov 25 10:22:50 2016 info: config - IP referrer tool is installed
Fri Nov 25 10:22:50 2016 info: config - debug level - 0
Fri Nov 25 10:22:50 2016 info: config - retries - 3
Fri Nov 25 10:22:50 2016 info: config - heartbeat - 15
Fri Nov 25 10:22:50 2016 info: config - interval - 20 seconds
Fri Nov 25 10:22:50 2016 info: config - offline interval - 10 seconds
Fri Nov 25 10:22:50 2016 info: config - referrer IP detection - disabled
Fri Nov 25 10:22:50 2016 info: config - ping server auto-detect - enabled
Fri Nov 25 10:22:50 2016 info: config - try pinging gateway - yes
Fri Nov 25 10:22:50 2016 info: config - number of external networks - 2
Fri Nov 25 10:22:50 2016 info: config - monitoring external network - eth2
Fri Nov 25 10:22:50 2016 info: config - monitoring external network - eth1
Fri Nov 25 10:22:50 2016 info: info - loading network configuration
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth2 - config: ifcfg-eth2
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth2 - onboot: enabled
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth2 - type: static
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth2 - wifi: disabled
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth2 - gateway: 192.168.10.1
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth1 - config: ifcfg-eth1
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth1 - onboot: enabled
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth1 - type: static
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth1 - wifi: disabled
Fri Nov 25 10:22:50 2016 info: info - network configuration for eth1 - gateway: 192.168.20.1
Fri Nov 25 10:22:50 2016 info: eth2 - network - IP address - 192.168.10.2
Fri Nov 25 10:22:50 2016 info: eth2 - network - gateway - 192.168.10.1
Fri Nov 25 10:22:50 2016 info: eth2 - network - type - private IP range
Fri Nov 25 10:22:50 2016 info: eth1 - network - IP address - 192.168.20.201
Fri Nov 25 10:22:50 2016 info: eth1 - network - gateway - 192.168.20.1
Fri Nov 25 10:22:50 2016 info: eth1 - network - type - private IP range
Fri Nov 25 10:22:52 2016 info: eth2 - ping check on gateway failed - 192.168.10.1
Fri Nov 25 10:22:54 2016 info: eth2 - ping check on server #1 failed - 8.8.8.8
Fri Nov 25 10:22:59 2016 info: eth2 - ping check on server #2 failed - 54.152.208.245
Fri Nov 25 10:22:59 2016 warn: eth2 - connection is down
Fri Nov 25 10:23:01 2016 info: eth2 - waiting for static IP reconnect
Fri Nov 25 10:23:01 2016 info: system - changing active WAN list - eth1 (was startup)
Fri Nov 25 10:23:01 2016 info: system - current WANs in use - eth1
Fri Nov 25 10:23:01 2016 info: system - restarting firewall
Fri Nov 25 10:23:02 2016 info: system - updating intrusion prevention whitelist
Fri Nov 25 10:23:02 2016 info: system - adding ping server 54.152.208.245
Fri Nov 25 10:23:02 2016 info: system - adding ping server 8.8.8.8
Fri Nov 25 10:23:02 2016 info: system - adding DNS server 192.168.20.1
Fri Nov 25 10:23:02 2016 info: system - adding DNS server 201.217.1.230
Fri Nov 25 10:23:02 2016 info: system - adding DNS server 8.8.8.8
Fri Nov 25 10:23:02 2016 info: system - reloading intrusion prevention system
Fri Nov 25 10:23:08 2016 info: system - dynamic DNS updated
Fri Nov 25 10:23:20 2016 info: eth2 - ping check on gateway failed - 192.168.10.1
Fri Nov 25 10:23:22 2016 info: eth2 - ping check on server #1 failed - 8.8.8.8
Fri Nov 25 10:23:27 2016 info: eth2 - ping check on server #2 failed - 54.152.208.245
Fri Nov 25 10:23:27 2016 warn: eth2 - connection is down
Fri Nov 25 10:23:30 2016 info: eth2 - waiting for static IP reconnect
Fri Nov 25 10:23:42 2016 info: eth2 - ping check on gateway failed - 192.168.10.1
Fri Nov 25 10:23:44 2016 info: eth2 - ping check on server #1 failed - 8.8.8.8
Fri Nov 25 10:23:49 2016 info: eth2 - ping check on server #2 failed - 54.152.208.245
Fri Nov 25 10:23:49 2016 warn: eth2 - connection is down
Fri Nov 25 10:23:51 2016 info: eth2 - waiting for static IP reconnect

(The last seven lines are repeated endlessly since then)

I don't know where that 54.152.208.245 is coming from...

Thanks for the outputs...

Look at the bottom of /etc/syswatch to configure your own ping servers rather than the ClearOS supplied defaults... ( see about line 150 of /usr/sbin/syswatch for defaults)...

This doesn't look good

Fri Nov 25 10:22:52 2016 info: eth2 - ping check on gateway failed - 192.168.10.1

Contents please of /etc/clearos/mutiwan.conf and /etc/udev/rules.d/70-persistent-net.rules
Output from "rpm -q app-multiwan"

Three suggestions initially....

1) Are you able to try a different cable going from ClearOS to the adsl modem? Do you have a spare NIC to replace eth2?

2) temporarily use webconfig to configure eth2 as LAN - can you manually ping the adsl gateway at 192.168.10.1 ? You should be able to...

3) use webconfig to swap eth1 and eth2 around - and swap the eth1 and eth2 ethernet cables at the ClearOS end - which one now fails?

Here's my sample output of what should happen - there are slight differences as I tailor /etc/syswatch...

Thu Nov 24 12:43:58 2016  info:  system - syswatch terminated

Thu Nov 24 12:43:59 2016  info:  system - syswatch started

Thu Nov 24 12:43:59 2016  info:  config - IP referrer disabled in multi-WAN

Thu Nov 24 12:43:59 2016  info:  config - IP referrer tool is not installed

Thu Nov 24 12:43:59 2016  info:  config - debug level - 0

Thu Nov 24 12:43:59 2016  info:  config - retries - 16 

Thu Nov 24 12:43:59 2016  info:  config - heartbeat - 5

Thu Nov 24 12:43:59 2016  info:  config - interval - 60 seconds

Thu Nov 24 12:43:59 2016  info:  config - offline interval - 10 seconds     

Thu Nov 24 12:43:59 2016  info:  config - referrer IP detection - disabled  

Thu Nov 24 12:43:59 2016  info:  config - ping server auto-detect - disabled

Thu Nov 24 12:43:59 2016  info:  config - try pinging gateway - yes

Thu Nov 24 12:43:59 2016  info:  config - ping server - 8.8.4.4

Thu Nov 24 12:43:59 2016  info:  config - ping server - 61.9.173.37

Thu Nov 24 12:43:59 2016  info:  config - number of external networks - 2   

Thu Nov 24 12:43:59 2016  info:  config - monitoring external network - eth3

Thu Nov 24 12:43:59 2016  info:  config - monitoring external network - eth1

Thu Nov 24 12:43:59 2016  info:    info - loading network configuration

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth3 -  config: ifcfg-eth3 

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth3 -  onboot: enabled    

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth3 -    type: static     

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth3 -    wifi: disabled   

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth3 - gateway: 192.168.6.1

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth1 -  config: ifcfg-eth1

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth1 -  onboot: enabled 

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth1 -    type: static  

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth1 -    wifi: disabled

Thu Nov 24 12:43:59 2016  info:    info - network configuration for eth1 - gateway: 192.168.2.35

Thu Nov 24 12:43:59 2016  info:    eth3 - network - IP address - 192.168.6.26

Thu Nov 24 12:43:59 2016  info:    eth3 - network - gateway - 192.168.6.1    

Thu Nov 24 12:43:59 2016  info:    eth3 - network - type - private IP range  

Thu Nov 24 12:43:59 2016  info:    eth1 - network - IP address - 192.168.2.26

Thu Nov 24 12:43:59 2016  info:    eth1 - network - gateway - 192.168.2.35 

Thu Nov 24 12:43:59 2016  info:    eth1 - network - type - private IP range

Thu Nov 24 12:43:59 2016  info:  system - changing active WAN list - eth1 eth3 (was startup)

Thu Nov 24 12:43:59 2016  info:  system - current WANs in use - eth1 eth3

Thu Nov 24 12:43:59 2016  info:  system - restarting firewall

Thu Nov 24 12:43:59 2016  info:  system - updating intrusion prevention whitelist

Thu Nov 24 12:43:59 2016  info:  system - adding ping server 8.8.4.4

Thu Nov 24 12:43:59 2016  info:  system - adding ping server 61.9.173.37

Thu Nov 24 12:43:59 2016  info:  system - adding DNS server 192.168.2.35

Thu Nov 24 12:43:59 2016  info:  system - adding DNS server 192.168.2.26

Thu Nov 24 12:43:59 2016  info:  system - reloading intrusion prevention system   

Thu Nov 24 12:48:16 2016  info:  system - heartbeat...

Thu Nov 24 12:53:16 2016  info:  system - heartbeat...

FWIW my syswatch is also configured to use 54.152.208.245. I'm sure if I dig into the code I could find it. I have a feeling it is one of the Clearfoundation servers, but it resolves back to Amazon AWS.

I think I got it!

I changed the ping servers, put googleDNS and openDNS, restarted syswatch (the logs show it failed and sometimes it passed the ping server check on both servers)...

Then I tried pinging google using eth2, it started pinging but loosing packets for moments, then pinging again, so I changed the cable as Tony suggested on 1) and boila! the pinging was permanent! So maybe the cable was faulty?

The thing that confuses me is how it worked on pppoe and the modem on bridge mode?

But nonetheless, it is working now!

Thank you Tony, Duncan and Nick for your support, I am sincerely grateful!

Encouraging news... let's hope it was the cable...

Actually, looking again at your first post...

The problem was that the ppp0 renewed the dynamic IP usually every 3 minutes, sometimes 1 minutes and sometimes 5, but it was constantly renewed

That is unusual... Seems quite possible that the faulty cable may have been the cause for this behaviour... One way to find out :-)

I'm still struggling with this one as ifconfig gave no errors or dropped packets, but if it is working now, it is working!

When you get confident you may also try PPoE again!

Hi Nick

ifconfig gave no errors or dropped packets

Yes - that surprised me somewhat as well, but...
1) We don't know what percentage of those packets occurred before there was a problem...
2) The line would have to go down in the middle of a packet to create an error logged in ifconfig.
If the line was down - no packet could start to send - hence no error for ifconfig...
If a packet made it every time the connection was up before it came down again - then no error in ifconfig
(A stretch of the imagination I know - but ping packets are very small)
With the interface was not up according to Multiwan - the number of packets would be very low - just the occasional ping to see if it had come up... All other traffic was being directed to the other external interface... Certainly not enough ping traffic to account for the number of packets shown by ifconfig for eth2 I would have thought.

What would have been interesting would be to see what errors the modem recorded. Fortunately my adsl modem has ssh access so I can run ifconfig within the modem. An example output for the wireless NIC within the modem... (It runs Linux kernel 2.6.8.1)

[root@alice ~]# ssh netcomm.sraellis.com -l xxxxxxx

xxxxxxx@netcomm.sraellis.com's password: 

> ifconfig wl0

wl0             Link encap:Ethernet  HWaddr 00:60:64:1C:49:0C  

                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

                RX packets:462852 errors:0 dropped:0 overruns:0 frame:24251

                TX packets:582222 errors:47 dropped:0 overruns:0 carrier:0

                collisions:0 txqueuelen:1000 

                RX bytes:339719101 (323.9 MiB)  TX bytes:295368623 (281.6 MiB)

                Interrupt:32 



> logout



Bye bye. Have a nice day!!!

Connection to netcomm.sraellis.com closed.

[root@alice ~]# 

Should have thought to ask if OP had checked the modem for errors (If there was a facility available...)

Hi Tony and Nick, I am just available to reply, yes I will give it a try to the PPPoE as soon as posible, maybe tomorrow afternoon.

Now it makes complete sense about the dropping on the conection with the PPPoE, I am almost sure that it will now work.

Thank you again for all you advice!

Best wishes!