0 votes
This is a bit complex (uh, sorry) but I've tried to distill it all down to the relevent stuff. Some IP addresses are obfuscated and lots of stuff omitted.

- WAN on port enp5s0, local, gateway, Wireless
- WAN on port enp6s0, local, gateway, Wireless
- WAN on dev ppp0 (ADSL) local 444.64.31.32, gateway 303.29.31.1 (I think)
- LAN on port enp4s0, local, with VLANs .10 and .20.

- ClearOS 7, last updated late 2016. OpenVPN 2.3.12 x86_64-redhat-linux-gnu
- Busy gateway server with three WANs, two VLANs, businesses, and paying guests using internet at all hours, plus VoIP.
- Default traffic is shared across the three WANs using COS Multi-WAN which works well.
- Physically a long drive (3hrs) to get to server, I administer it from afar via VPN, SSH, etc.
- ADSL modem connects to ISP who are also the VoIP exchange. Modem is on dev ppp0.
- VoIP only works through ADSL/ppp0, other WANs unavoidably have multiple NATs, which kills VoIP dead.
- The ADSL on ppp0 goes down more than occasionally, disrupting phones and making everyone antsy including me.
- We are trying to get an OpenVPN client to tunnel thru a public VPN gateway so VoIP bypasses the NATs on the other WANs.
- Incidentally, there's an OVPN server running here as well, on port 1194 which I use for admin, via modem and ppp0.
- Experimentation is tricky, need to keep this box ticking over.

One problem is that using a public VPN gateway ( we have no control over their OpenVPN server config, however they do provide sample configs for linux and I've used that with a certain amount of success. Plainly we can't have their OVPN server pushing a default route, so I've added a route-nopull to the local client config plus a few routes that we want going through the VPN. It produces a bunch of errors as a result, but I they seem to be benign. This is the stuff I'm least sure about.

Result is no VoIP traffic routed. Traceroute shows the traffic for the VoIP network (333.111.14.0/24) is still going out ppp0. Pinging remote end of tunnel gives nothing. tunnel interface (tun1) looks as dead as king Tut. OVPN seems happy though, reports tunnel set up over one of the WANs.

Occasionally ppp0 goes down (I lose the inwards VPN) after a couple of hours of starting the OVPN client which seems a bit strange. Getting the local guy to reboot the whole box fixes it but VoIP is dead (ping is silent) until I stop the OVPN client and then it all comes right.

It looks to me like a clash of Multi-WAN with the static routes added by OVPN. It sometimes works with metric set to zero but I need to use metrics so that I can change routes without taking down ppp0 and screwing up VoIP calls and my own VPN. Also, I think the route back from tun1 to the sender isn't working - tun1 shows traffic both ways but no ping response (logs and config below).

I've battled with network-scripts/route-xxx files, but that system seems unable to cope with tun1, mostly I think because there's no ifcfg-tun1 (perhaps should I make one?)

Does anyone have a clue how to fix this?

FYI, is just a random website which will return my pings and is handy for testing. 333.111.x.x is the VoIP provider.

This ping goes out some other WAN:
# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=57 time=65.1 ms

Forcing it out tun1 goes nowhere:
# ping -I tun1
PING ( from tun1: 56(84) bytes of data.
--- ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 6999ms

But there was traffic (tun1 had zero traffic just prior):
# ifconfig tun1
inet netmask destination
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 100 (UNSPEC)
RX packets 8 bytes 672 (672.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 672 (672.0 B)

And when I look closely at the ping traffic:
# tcpdump -vvv -i tun1
tcpdump: listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes
17:22:53.910472 IP (tos 0x0, ttl 64, id 58729, offset 0, flags [DF], proto ICMP (1), length 84)
mako.lan > ICMP echo request, id 19801, seq 1, length 64
17:22:54.378427 IP (tos 0x0, ttl 60, id 33888, offset 0, flags [none], proto ICMP (1), length 84) > mako.lan: ICMP echo reply, id 19801, seq 1, length 64

# cat /etc/openvpn/safervpn.config
dev tun
proto tcp
cipher AES-256-CBC
remote-cert-tls server
remote 443 # - Your server IP and OpenVPN Port
resolv-retry infinite
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
auth-user-pass blahblah-pass
user nobody
group nobody
verb 3

# ifconfig # Grizzles if it's included
route-metric 40
route 333.111.14.0
route 333.111.13.0
<certificate stuff>

# ip route
default via dev enp6s0 via dev tun1 metric 40 dev tun1 proto kernel scope link src via dev tun0 dev tun0 proto kernel scope link src dev enp4s0.20 proto kernel scope link src via dev tun1 metric 40
333.111.14.0/24 via dev tun1 metric 40
333.111.14.0/24 dev ppp0 metric 80
333.111.14.0/24 dev enp6s0 metric 100
303.29.31.1 dev ppp0 proto kernel scope link src 444.64.31.32
303.38.90.35 dev ppp0 dev enp4s0.10 proto kernel scope link src via dev enp4s0 dev enp6s0 proto kernel scope link src dev enp5s0 proto kernel scope link src

# cat route-ppp0 via dev ppp0 metric 80

# cat ifcfg-ppp0

# tail -n 200 /var/log/messages | grep openvpn
OpenVPN 2.3.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug 23 2016
library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.06
WARNING: file 'safervpn-pass' is group or others accessible
Socket Buffers: R=[87380->87380] S=[16384->16384]
NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Attempting to establish TCP connection with [AF_INET] [nonblock]
TCP connection established with [AF_INET]
TCPv4_CLIENT link local: [undef]
TCPv4_CLIENT link remote: [AF_INET]
TLS: Initial packet from [AF_INET], sid=003655db 9a7c7966
WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
VERIFY OK: depth=1, C=US, ST=CA, L=SanFrancisco, O=SaferVPN, OU=SaferVPN, CN=SaferVPN, name=SaferVPN, emailAddress=support@s
Validating certificate key usage
++ Certificate has key usage 00a0, expects 00a0
Validating certificate extended key usage
++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
VERIFY OK: depth=0, C=US, ST=CA, L=SanFrancisco, O=SaferVPN, OU=SaferVPN, CN=SaferVPN, name=SaferVPN, emailAddress=support@s
Data Channel Encrypt <messages omitted>
[SaferVPN] Peer Connection Initiated with [AF_INET]
PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS,dhcp-option DNS,explicit-exit-notify,redirect-ga.1,topology net30,ping 5,ping-restart 30,ifconfig'
Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])
Options error: option 'route' cannot be used in this context ([PUSH-OPTIONS])
OPTIONS IMPORT: timers and/or timeouts modified
OPTIONS IMPORT: --explicit-exit-notify can only be used with --proto udp
OPTIONS IMPORT: --ifconfig/up options modified
ROUTE_GATEWAY IFACE=enp6s0 HWADDR=dg:ch:id:11:ff:bb
TUN/TAP device tun1 opened
TUN/TAP TX queue length set to 100
do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
/usr/sbin/ip link set dev tun1 up mtu 1500
/usr/sbin/ip addr add dev tun1 local peer
/usr/sbin/ip route add metric 40 via
/usr/sbin/ip route add 333.111.14.0/24 metric 40 via
GID set to nobody
UID set to nobody
Initialization Sequence Completed
In OpenVPN
Saturday, October 28 2017, 01:33 PM
Share this post:
Responses (1)
  • Accepted Answer

    Sunday, November 26 2017, 09:12 AM - #Permalink
    0 votes
    Hi Lindsay,
    Welcome to the forum and sorry for ignoring you. New users posts get moderated and if they are not released very quickly I tend to miss them if the don't appear high up on the first page.

    If you have not managed to do anything, I think this is beyond me unless I spend a huge amount of time on it, but one of the devs has done some clever routing in the ibVPN app to route particular LAN IP's via ibVPN. It may give you some ideas. Have a look at the /etc/clearos/ibvpn.d/route-up script in app-ibvpn-core in clearos contribs
    The reply is currently minimized Show
Your Reply