We have had an issue with broadcast storms in our network. Checking the CoPP setup in the switches, we could see massive drops of ARP. This is a good link to know how to check CoPP drops in NXOS.
N9K:# show copp status
N9K# show policy-map interface control-plane | grep 'dropped [1-9]' | diff
Having so many ARP drops by CoPP is bad because very likely good ARP requests are going to be dropped.
Initially i thought it was related to ARP problems in EVPN like this link. But after taking a packet capture in a switch from an interface connected to a server, I could see that over 90% ARP traffic coming from the server was not getting a reply…. Checking in different switches, I could see the same pattern all over the place.
So why the server was making so many ARP requests?
After some time, managed to help help from a sysadmin with access to the servers so could troubleshoot the problem.
But, how do you find the process that is triggering the ARP requests? I didnt make the effort to think about it and started to search for an easy answer. This post gave me a clue.
ss does show you connections that have not yet been resolved by arp. They are in state SYN-SENT. The problem is that such a state is only held for a few seconds then the connection fails, so you may not see it. You could try rapid polling for it with
while ! ss -p state syn-sent | grep 1.1.1.100; do sleep .1; done
Somehow I couldnt see anything anything with “ss” so tried netstat as it shows you too the status of the TCP connection (I wonder what would happen is the connection was UDP instead???)
Initially I tried “netstat -a” and it was too slow to show me “SYN-SENT” status
Shame on me, I had to search how to get to show the ports quickly here:
watch netstat -ntup | grep -i syn_sent | awk '{print $4,$5,$6,$7}'
It was slow because it was trying to resolve all IPs to hostname…. :facepalm. Tha is fixed with “-n” (no-resolve)
Anyway, with the command above, finally managed to see the process that were in “SYN_SENT” state
This is not the real thing, just an example:
# netstat -ntup | grep -i syn_sent
tcp 0 1 192.168.1.203:35460 4.4.4.4:23 SYN_SENT 98690/telnet
#
We could see that the destination port was TCP 179, so something in the node was trying to talk BGP! They were “bird” processes. As the node belonged to a kubernetes cluster, we could see a calico container as CNI. Then we connected to the container and tried to check the bird config. We could see clearly the IPs that dont get ARP reply were configured there.
So in summary, basic TCP:
Very summarize, TCP is L4, then goes down to L3 IP. For getting to L2, you need to know the MAC of the IP, so that triggers the ARP request. Once the MAC is learned, it is cached for the next request. For that reason the first time you make a connection is slow (ping, traceroute, etc)
Now we need to workout why the calico/bird config is that way. Fix it to only use IPs of real BGP speakers and then verify the ARP storms stop.
Hopefully, I will learn a bit about calico.
Notes for UDP:
If I generate an UDP connection to a non-existing IP
$ nc -u 4.4.4.4 4000
netstat tells me the UDP connection is established and I can’t see anything in the ARP table for an external IP, for an internal IP (in my own network) I can see an incomplete entry. Why?
# netstat -ntup | grep -i 4.4.4.4
udp 0 0 192.168.1.203:42653 4.4.4.4:4000 ESTABLISHED 102014/nc
#
# netstat -ntup | grep -i '192.168.1.2:'
udp 0 0 192.168.1.203:44576 192.168.1.2:4000 ESTABLISHED 102369/nc
#
#
# arp -a
? (192.168.1.2) at <incomplete> on wlp2s0
something.mynet (192.168.1.1) at xx:xx:xx:yy:yy:zz [ether] on wlp2s0
#
# tcpdump -i wlp2s0 host 4.4.4.4
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wlp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:35:45.081819 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1
23:35:45.081850 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1
23:35:46.082075 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1
23:35:47.082294 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1
23:35:48.082504 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
#
- UDP is stateless so we can’t have states…. so it is always going to be “established”. Basic TCP/UDP
- When trying to open an UDP connection to an external IP, you need to “route” so my laptop knows it needs to send the UDP connection to the default gateway, so when getting to L2, the destination MAC address is not 4.4.4.4 is the default gateway MAC. BASIC ROUTING !!!! For that reason you dont see 4.4.4.4 in ARP table
- When trying to open an UDP connection to a local IP, my laptop knows it is in the same network so it should be able to find the destination MAC address using ARP.
