{"id":1053,"date":"2022-11-10T23:17:30","date_gmt":"2022-11-10T23:17:30","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=1053"},"modified":"2022-11-10T23:46:25","modified_gmt":"2022-11-10T23:46:25","slug":"arp-storms-evpn","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2022\/11\/10\/arp-storms-evpn\/","title":{"rendered":"ARP Storms – EVPN"},"content":{"rendered":"\n

We have had an issue with broadcast storms in our network. Checking the CoPP setup in the switches, we could see massive drops of ARP. This is a good link<\/a> to know how to check CoPP drops in NXOS.<\/p>\n\n\n\n

N9K:# show copp status\nN9K# show policy-map interface control-plane | grep 'dropped [1-9]' | diff<\/pre>\n\n\n\n
Having so many ARP drops by CoPP is bad because very likely good ARP requests are going to be dropped.<\/p>\n\n\n\n
Initially i thought it was related to ARP problems in EVPN like this link<\/a>. But after taking a packet capture in a switch from an interface connected to a server,  I could see that over 90% ARP traffic coming from the server was not getting a reply…. Checking in different switches, I could see the same pattern all over the place.<\/p>\n\n\n\n
So why the server was making so many ARP requests?<\/p>\n\n\n\n
After some time, managed to help help from a sysadmin with access to the servers so could troubleshoot the problem.<\/p>\n\n\n\n
But, how do you find the process that is triggering the ARP requests? I didnt make the effort to think about it and started to search for an easy answer. This post<\/a> gave me a clue.<\/p>\n\n\n\n
ss does show you connections that have not yet been resolved by arp. They are in state SYN-SENT. The problem is that such a state is only held for a few seconds then the connection fails, so you may not see it. You could try rapid polling for it with\n\nwhile ! ss -p state syn-sent | grep 1.1.1.100; do sleep .1; done<\/pre>\n\n\n\n
Somehow I couldnt see anything anything with “ss” so tried netstat as it shows you too the status of the TCP connection (I wonder what would happen is the connection was UDP instead???)<\/p>\n\n\n\n
Initially I tried “netstat -a” and it was too slow to show me “SYN-SENT” status<\/p>\n\n\n\n
Shame on me, I had to search how to get to show the ports quickly here<\/a>:<\/p>\n\n\n\n
watch netstat -ntup | grep -i syn_sent | awk '{print $4,$5,$6,$7}'<\/pre>\n\n\n\n
It was slow because it was trying to resolve all IPs to hostname…. :facepalm. Tha is fixed with “-n” (no-resolve)<\/p>\n\n\n\n
Anyway, with the command above, finally managed to see the process that were in “SYN_SENT” state<\/p>\n\n\n\n
This is not the real thing, just an example:<\/p>\n\n\n\n
#  netstat -ntup | grep -i syn_sent \ntcp        0      1 192.168.1.203:35460     4.4.4.4:23              SYN_SENT    98690\/telnet        \n# \n<\/pre>\n\n\n\n
We could see that the destination port was TCP 179, so something in the node was trying to talk BGP! They were “bird” processes. As the node belonged to a kubernetes cluster, we could see a calico<\/a> container as CNI. Then we connected to the container and tried to check the bird config. We could see clearly the IPs  that dont get ARP reply were configured there.<\/p>\n\n\n\n
So in summary, basic<\/strong> TCP:<\/p>\n\n\n\n
 Very summarize, TCP is L4, then goes down to L3 IP. For getting to L2, you  need to know the MAC of the IP, so that triggers the ARP request. Once the MAC is learned, it is cached for the next request. For that reason the first time you  make a connection is slow (ping, traceroute, etc)<\/p>\n\n\n\n
Now we need to workout why the calico\/bird config is that way. Fix it to only use IPs of real BGP speakers and then verify the ARP storms stop.<\/p>\n\n\n\n
Hopefully, I will learn a bit about calico.<\/p>\n\n\n\n
Notes for UDP<\/strong>:<\/p>\n\n\n\n
 If I generate an UDP connection to a non-existing IP<\/p>\n\n\n\n
$ nc -u 4.4.4.4 4000<\/pre>\n\n\n\n
netstat tells me the UDP connection is established and I can’t see anything in the ARP table for an external IP, for an internal IP (in my own network) I can see an incomplete entry. Why<\/strong>?<\/p>\n\n\n\n
#  netstat -ntup | grep -i 4.4.4.4\nudp        0      0 192.168.1.203:42653     4.4.4.4:4000            ESTABLISHED 102014\/nc           \n# \n#  netstat -ntup | grep -i '192.168.1.2:'\nudp        0      0 192.168.1.203:44576     192.168.1.2:4000        ESTABLISHED 102369\/nc           \n# \n#\n# arp -a\n? (192.168.1.2) at <incomplete> on wlp2s0\nsomething.mynet (192.168.1.1) at xx:xx:xx:yy:yy:zz [ether] on wlp2s0\n# \n\n# tcpdump -i wlp2s0 host 4.4.4.4\ntcpdump: verbose output suppressed, use -v[v]... for full protocol decode\nlistening on wlp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes\n23:35:45.081819 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1\n23:35:45.081850 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1\n23:35:46.082075 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1\n23:35:47.082294 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1\n23:35:48.082504 IP 192.168.1.203.50186 > 4.4.4.4.4000: UDP, length 1\n^C\n5 packets captured\n5 packets received by filter\n0 packets dropped by kernel\n# <\/pre>\n\n\n\n