Forums

Resolved
0 votes
Since 07:40 last Monday (13/07) arpwatch has started reporting differently and it is making a mess of the Network Map. Since then any dhcp request was followed by three bogon reports from arpwatch:
Jan 13 07:40:48 server dnsmasq-dhcp[1972]: DHCPOFFER(eth1) 172.17.2.129 c0:64:c6:4d:a9:78 
Jan 13 07:40:48 server dnsmasq-dhcp[1972]: DHCPREQUEST(eth1) 172.17.2.129 c0:64:c6:4d:a9:78
Jan 13 07:40:48 server dnsmasq-dhcp[1972]: DHCPACK(eth1) 172.17.2.129 c0:64:c6:4d:a9:78 Windows-Phone
Jan 13 07:40:48 server arpwatch: changed ethernet address 172.17.2.129 c0:64:c6:4d:a9:78 (98:52:b1:7c:ab:e8)
Jan 13 07:40:48 server arpwatch: bogon 0.0.0.0 c0:64:c6:4d:a9:78
Jan 13 07:40:49 server arpwatch: bogon 0.0.0.0 c0:64:c6:4d:a9:78
Jan 13 07:40:50 server arpwatch: bogon 0.0.0.0 c0:64:c6:4d:a9:78
Jan 13 08:16:11 server dnsmasq-dhcp[1972]: DHCPREQUEST(eth1) 172.17.2.100 00:1b:21:1c:65:9d
Jan 13 08:16:11 server dnsmasq-dhcp[1972]: DHCPACK(eth1) 172.17.2.100 00:1b:21:1c:65:9d Black
Jan 13 08:16:12 server arpwatch: bogon 0.0.0.0 0:1b:21:1c:65:9d
Jan 13 08:16:13 server arpwatch: bogon 0.0.0.0 0:1b:21:1c:65:9d
Jan 13 08:16:14 server arpwatch: bogon 0.0.0.0 0:1b:21:1c:65:9d
Also it has started picking up 169 type addresses:
Jan 13 08:16:15 server arpwatch: bogon 169.254.197.223 0:1b:21:1c:65:9d
Jan 13 08:16:15 server dnsmasq-dhcp[1972]: DHCPREQUEST(eth1) 172.17.2.100 00:1b:21:1c:65:9d
Jan 13 08:16:15 server dnsmasq-dhcp[1972]: DHCPACK(eth1) 172.17.2.100 00:1b:21:1c:65:9d Black
Jan 13 08:16:16 server arpwatch: bogon 0.0.0.0 0:1b:21:1c:65:9d
Jan 13 08:16:17 server arpwatch: bogon 0.0.0.0 0:1b:21:1c:65:9d
Jan 13 08:16:18 server arpwatch: bogon 0.0.0.0 0:1b:21:1c:65:9d
A 169 IP address is taken by a PC when it fails to get a lease.

I've managed to stop the bogon's by adding the -N switch to /etc/sysconfig/arpwatch and the messages I am now getting are like:
Jan 19 21:08:22 server arpwatch: new station 169.254.237.57 9c:4:eb:a8:8c:a7
Jan 19 21:32:52 server arpwatch: reused old ethernet address 0.0.0.0 c0:64:c6:4d:a9:78 (9c:4:eb:a8:8c:a7)
Jan 20 07:48:59 server arpwatch: reused old ethernet address 0.0.0.0 c0:41:f6:80:b0:a7 (c0:64:c6:4d:a9:78)
Jan 20 07:49:55 server arpwatch: flip flop 0.0.0.0 c0:64:c6:4d:a9:78 (c0:41:f6:80:b0:a7)
Jan 20 08:40:43 server arpwatch: reused old ethernet address 0.0.0.0 0:1b:21:1c:65:9d (c0:64:c6:4d:a9:78)
Jan 20 12:46:23 server arpwatch: reused old ethernet address 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)
Jan 20 13:18:57 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
Jan 20 13:49:10 server arpwatch: flip flop 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)
Jan 20 13:49:14 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
Jan 20 14:19:24 server arpwatch: flip flop 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)
Jan 20 14:19:26 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
Jan 20 14:49:39 server arpwatch: flip flop 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)
Jan 20 14:49:43 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
Jan 20 15:50:09 server arpwatch: flip flop 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)
Jan 20 15:50:12 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
Jan 20 16:17:26 server arpwatch: reused old ethernet address 0.0.0.0 c0:41:f6:80:b0:a7 (0:1b:21:1c:65:9d)
Jan 20 16:20:24 server arpwatch: reused old ethernet address 0.0.0.0 0:1c:c0:39:18:10 (c0:41:f6:80:b0:a7)
Jan 20 16:20:28 server arpwatch: reused old ethernet address 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
Jan 20 17:28:01 server arpwatch: flip flop 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)
Jan 20 17:28:06 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
Jan 20 18:01:44 server arpwatch: reused old ethernet address 0.0.0.0 c0:41:f6:80:b0:a7 (0:1b:21:1c:65:9d)
Jan 20 19:07:02 server arpwatch: reused old ethernet address 0.0.0.0 0:1c:c0:39:18:10 (c0:41:f6:80:b0:a7)
Jan 20 19:07:06 server arpwatch: reused old ethernet address 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
The 0.0.0.0 and 169 addresses are making a mess of the Network Map: http://www.clearfoundation.com/media/kunena/attachments/legacy/images/Capture-20140120.PNG

I don't think anything in particular has been done to the server. The last yum log was from a few days prior. I accidentally rebooted the server the evening before (when I thought I was in a Raspberry Pi console!) but there is nothing special.

Has anyone any ideas?
Monday, January 20 2014, 08:13 PM
Share this post:
Responses (8)
  • Accepted Answer

    Tuesday, January 21 2014, 05:11 PM - #Permalink
    Resolved
    1 votes
    I added a tracker to make sure we tackle it:
    http://tracker.clearfoundation.com/view.php?id=1525
    The reply is currently minimized Show
  • Accepted Answer

    Monday, January 20 2014, 10:22 PM - #Permalink
    Resolved
    0 votes
    Hi Nick,

    I'm not sure what's going on there. Arpwatch is just reporting what it sees, but I can't figure out what would generate those messages on your network.

    Regardless, the Network Map app should do something better to handle this scenario. One approach would be to filter the 0.0.0.0 and 169 addresses out of the "Unknown Devices" summary page. Instead, these could be moved to a separate "Weird Stuff" summary page. That doesn't answer the fundamental question though -- what exactly is happening!?
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, January 21 2014, 12:34 PM - #Permalink
    Resolved
    0 votes
    Information an aprwatch on the internet appears to be scant.

    From what I can make out a lot of the 0.0.0.0 stuff is generally from devices requesting a DHCP lease but why are they not requesting their original IP addresses or is that the initiating broadcast message from the client trying to locate the DHCP server? It is not just a Windoze thing as Apple, Draytek and LG (TV) devices also have the same behaviour. The Draytek device does not even use DHCP. It has a fixed IP.

    I've tried stopping aprwatch and cleaning its dat file in /var/lib/arpwatch (I can't remember exact the name) then restarting arpwatch. I've similarly cleaned up the dnsmasq.leases file. Each time I do that, the arpwatch dat file slowly re-populates. Over this weekend I also powered down the Draytek which acts as a WAP and switch. Perhaps tonight I can try power cycling the other switch.

    One thing I did notice was in previous weeks' logs was that arpwatch was very quiet except in the 4 week old log where it also had a lot of bogon messages.

    I am very puzzled.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, January 21 2014, 01:13 PM - #Permalink
    Resolved
    0 votes
    I observe a similar behaviour here, but only get one 'bogon' warning about an IP that is outside the NIC subnet range (0.0.0.0) - this seems to occur at the same time as a DHCP request, and given the logging interval perhaps at the same second as it is served a DHCP IP

    Do the entries exist in your ClearOS ARP table? run 'arp' from the command line?

    Perhaps another device on your network has a corrupt ARP cache? To determine that might require use of Wireshark of tcpdump to see what ARP traffic is doing on your network
    tcpdump -i eth1 arp
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, January 21 2014, 07:15 PM - #Permalink
    Resolved
    0 votes
    same here

    Jan 21 19:48:24 pdebrabander arpwatch: bogon 0.0.0.0 c0:f8:da:72:41:f5
    Jan 21 19:48:25 pdebrabander arpwatch: bogon 0.0.0.0 c0:f8:da:72:41:f5
    Jan 21 19:48:26 pdebrabander arpwatch: bogon 0.0.0.0 c0:f8:da:72:41:f5
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, January 21 2014, 09:48 PM - #Permalink
    Resolved
    0 votes
    I had a look at the arp cache and it only has valid LAN and WAN IP's:
    [root@server ~]# arp -n
    Address HWtype HWaddress Flags Mask Iface
    172.17.2.104 ether b8:27:eb:cb:bf:e0 CM eth1
    172.17.2.107 ether 00:24:f3:d7:13:7a CM eth1
    69.90.141.72 ether 00:30:b8:d0:b9:34 CM eth0
    172.17.2.3 ether 00:1e:8f:4a:26:d2 CM eth1
    172.17.2.100 ether 00:1b:21:1c:65:9d CM eth1
    82.19.147.1 ether 00:30:b8:d0:b9:34 C eth0
    172.17.2.106 ether d8:30:62:26:09:9b CM eth1
    172.17.2.109 ether 00:24:1e:31:9c:04 CM eth1
    172.17.2.103 ether c0:41:f6:80:b0:a7 CM eth1
    172.17.2.108 ether 9c:04:eb:a8:8c:a7 CM eth1
    192.168.100.1 ether 00:30:b8:d0:b9:34 CM eth0
    172.17.2.102 ether 00:1c:c0:39:18:10 CM eth1
    I set up a monitor of arp data with tcpdump when I turned on a PC (Antec-Black, 00:1c:c0:39:18:10):
    20:02:54.791558 ARP, Request who-has TV.howitts.lan tell WAP.howitts.lan, length 46
    20:02:54.791696 ARP, Request who-has Pi.howitts.lan tell WAP.howitts.lan, length 46
    20:03:04.766829 ARP, Request who-has canonmp640.howitts.lan tell WAP.howitts.lan, length 46
    20:03:05.669457 ARP, Request who-has server.howitts.lan tell iPad.howitts.lan, length 46
    20:03:05.669485 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:03:07.402493 ARP, Request who-has server.howitts.lan (a0:36:9f:26:7c:52 (oui Unknown)) tell Black.howitts.lan, length 46
    20:03:07.402523 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:03:34.692752 ARP, Request who-has Black.howitts.lan tell WAP.howitts.lan, length 46
    20:03:40.686400 ARP, Request who-has 169.254.7.99 tell 0.0.0.0, length 46
    20:03:41.685323 ARP, Request who-has 169.254.7.99 tell 0.0.0.0, length 46
    20:03:42.683210 ARP, Request who-has 169.254.7.99 tell 0.0.0.0, length 46
    20:03:43.681684 ARP, Request who-has 169.254.7.99 tell 169.254.7.99, length 46
    20:03:49.188480 ARP, Request who-has Antec-Black.howitts.lan tell 0.0.0.0, length 46
    20:03:49.975812 ARP, Request who-has server.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:03:49.975847 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:03:50.186875 ARP, Request who-has Antec-Black.howitts.lan tell 0.0.0.0, length 46
    20:03:50.792864 ARP, Request who-has server.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:03:50.792894 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:03:51.185278 ARP, Request who-has Antec-Black.howitts.lan tell 0.0.0.0, length 46
    20:03:52.186680 ARP, Request who-has Antec-Black.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:03:53.178382 ARP, Request who-has canonmp640.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:03:54.643355 ARP, Request who-has TV.howitts.lan tell WAP.howitts.lan, length 46
    20:03:54.643499 ARP, Request who-has Pi.howitts.lan tell WAP.howitts.lan, length 46
    20:03:57.094208 ARP, Request who-has server.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:03:57.094235 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:03:57.104511 ARP, Request who-has server.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:03:57.104525 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:04:00.289236 ARP, Request who-has server.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:04:00.289261 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:04:00.301793 ARP, Request who-has server.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:04:00.301807 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:04:00.694943 ARP, Request who-has server.howitts.lan tell Antec-Black.howitts.lan, length 46
    20:04:00.694966 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:04:04.618628 ARP, Request who-has server.howitts.lan tell WAP.howitts.lan, length 46
    20:04:04.618674 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    20:04:04.618746 ARP, Request who-has canonmp640.howitts.lan tell WAP.howitts.lan, length 46
    20:04:34.544550 ARP, Request who-has Black.howitts.lan tell WAP.howitts.lan, length 46
    20:04:35.697092 ARP, Request who-has server.howitts.lan tell iPad.howitts.lan, length 46
    20:04:35.697136 ARP, Reply server.howitts.lan is-at a0:36:9f:26:7c:52 (oui Unknown), length 28
    Here I don't know what to expect but it seems strange to me to see lots of "tell 0.0.0.0"

    As the PC started I only got one arpwatch message in /var/log/messages (bogons are suppressed)
    Jan 21 20:03:40 server arpwatch: reused old ethernet address 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)


    Power cycling the switch gave:
    Jan 21 21:26:54 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
    Jan 21 21:26:54 server arpwatch: flip flop 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)
    Jan 21 21:26:55 server arpwatch: flip flop 0.0.0.0 0:1b:21:1c:65:9d (0:1c:c0:39:18:10)
    Jan 21 21:26:55 server arpwatch: flip flop 0.0.0.0 0:1c:c0:39:18:10 (0:1b:21:1c:65:9d)


    I really don't know what is happening here as it is beyond my knowledge.

    @Patrick,
    You should be able to get rid of the bogon messages by adding the -N switch to /etc/sysconfig/arpwatch.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, January 22 2014, 02:02 PM - #Permalink
    Resolved
    0 votes
    After a bit of research, its part of the Address Conflict Detection RFC, used by devices to detect duplicate IP's on the network before accepting a DHCP request. It would appear 0.0.0.0 in this instance is the 'any' field
    http://tools.ietf.org/html/draft-cheshire-ipv4-acd-04

    Perhaps this might also explain it?
    http://ask.wireshark.org/questions/5178/why-gratuitous-arps-for-0000

    See also item 3.
    https://www.ams-ix.net/downloads/arpsponge/3.12.2/arpsponge-3.12.2/arpsponge.txt

    Either way, I think the network-map app should hide these ARP requests from duplicate MAC addresses
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, January 22 2014, 07:44 PM - #Permalink
    Resolved
    0 votes
    Thanks for searching. I've been googling:
    "who-has" "tell 0.0.0.0"
    I've been through about 3 pages of links and not found anything obvious. It appears to point to a device looking for a gateway but all my devices have been re-powered and only the WAP does not get configured with a gateway. Everything else works by DHCP. I will isolate the WAP at some point but I doubt if it is the cause.

    There is a RedHat bug report from December which we are not allowed to access which has me wondering.

    I'll keep searching for a bit. I'll also upgrade the kernel as I'm a couple of updates behind.

    I certainly agree about the need to mod the network-map app to exclude 0.0.0.0 and 169.*
    The reply is currently minimized Show
Your Reply