I have finished reading this book about Kaizen. Many years ago I heard the term Kaizen for the superior productivity in Japan, mainly from Toyota as the world’s number one car producer. Somehow, I bought this used copy to learn about it.
First, I noticed the book was printed in 1986… I realised I rarely read “technical” stuff so “old”.
The first surprise was that it seems the concept of quality control was actually brought by USA to post WWII Japan. The two main people were W E Deming and J M Juran.
It is interesting that Japan was very eager to learn the productivity secrets from USA and at the end, they created their on version.
I like the focus in people. They need to be engaged and feel part of something. At the end of the day, everybody has to push together to get to great results. As well, it seems key the achievement of small changes and not massive ones. They set for long-term goals, mainly for customer satisfaction that is not just the person who buys the product. It is not all about profits. It seems the profits will come as a by-product (reduce cost, increase customer satisfaction, more sells -> more profits). So the vision is product-oriented instead of result-oriented.
The point that “if you dont have problems, how you can improve?!” is so true.
And one slogan to measure how good is your product: “would you buy what you are producing?”
I can see many concepts are already in place in technology. The “Kambam” board, the constant search for small improvements, etc. If you think devops culture is something really modern, doesn’t look like that.
In general, the approach is quite different from the Western world and has been successful. But the book mentions that you need the innovation side for keep improving. So again, as life, you need balance.
And at the end, Kaizen becomes like a way of life. Or it is like I see it.
I am curious how the author would see Kaizen and Japan nowadays.
This year, in my employer, I completed the migration to a MPLS SR Arista core network from a Brocade MPLS LDP one. Our backbone is still pure IPv4 so anything IPv6 is not going to be added. But this week, via an APNIC blog post I read about SRv6. And it looks quite interesting. So I went to the first post to go a bit deeper about what SRv6 is. Based on the statements of the blog, really big networks are already using this technology and quite a lot of support from the open source community too. I missed Arista in that list though.
So I tried to find some “real” proof of this SRv6 is some pcap files to see the format and get a bit better view. I could find at lest a source with some. The examples are not like the ones mentioned in the APNIC blog post but just for taking a look, it is enough:
So I can see inside the IPv6 header, the SRv6 Header as defined in the rfc.
I dont really understand the second IPv6 header (Dst: b::2). From the first IPv6 header, the destination “f1::” has to be the first instruction SID1. I can see how it mentions it contains a SRH (Next Header: 43). And inside the routing header, we can see it is SR type (Type: 4). I assume that Address[0] and Address[1] are SID2 and SID3.
I think the only thing I miss from Microsoft Windows, it is “paint”. I can’t find anything in Linux that is simple and reliable. I have used pinta, gimp and others but they break or they are too pro.
So, if you have internet connection. This is your friend
And if you want something a bit more pro like Visio, then this one
I recommend both a lot!
I know you have GIMP, but I just a simple tool to draw red squares!
Today I have had “fun” troubleshooting an issue that looked easy at first sight. A colleague was trying to PXE boot some server from a network that we haven’t used for a while.
When the server boots up, asks for an IP via DHCP. As we have a centralized DHCP server infrastructure, we have configured DHCP relay in the firewall facing that server to send that request to the DHCP server.
First, let’s take a look at how DHCP relay works. This is a very good link. And this diagram from the mentioned link it is really useful:
One think I learned is the reply (DCHP Offer) doesnt have to use as destination IP the same IP it received as source in DHCP Discover. In the picture, it is packet 2a.
Checking in our environment, we confirm that:
Our server is in 10.94.240.x network. Our firewall is acting as DHCP relay, and send the DHCP Discovery (unicast) to our VIP DHCP Server IP.
The DHCP offer, uses as source the physical IP of the DHCP server and destination is the DHCP relay IP (so it is 10.94.240.1 – the firewall IP in 10.94.240.x network)
Ok, so everything looks fine? No really. The server receives the query, it answers… but we dont see a DCHP Request/ACK.
BTW, keep in mind that DHCP is UDP….
So, we need to see where the packets are lost.
This is a high level path flow between the client and server:
So we need to check this connection is three different firewall vendors….
The initial troubleshooting was just using the GUI tools from Palo/Fortigate. We couldn see anything…. but the server was constantly receiving DHCP Discover and sending DHCP Offer… I dont get it:
# tcpdump -i X udp port 67 or 66 -nn
14:58:06.969462 IP 10.81.25.1.67 > 10.81.251.47.67: BOOTP/DHCP, Request from 6c:2b:59:c1:32:73, length 300
14:58:06.969564 IP 10.81.251.201.67 > 10.94.240.1.67: BOOTP/DHCP, Reply, length 300
14:58:28.329048 IP 10.81.25.1.67 > 10.81.251.47.67: BOOTP/DHCP, Request from 6c:2b:59:c1:32:73, length 300
14:58:28.329157 IP 10.81.251.201.67 > 10.94.240.1.67: BOOTP/DHCP, Reply, length 300
Initially it took me a while to see the request/reply because I was assuming the dhcp request had source 10.94.240.1. So I was seeing only the Reply but not the Request. That was when I went to clarify my head about DHCP Relay and found the link.
So ok, we have the DHCP Request/Reply, but absolutely nothing in the Palo. Is the palo dropping the packets or is forwarding? No idea. The GUI says nothing, I took a packet capture and couldnt see that traffic neither…
Doesnt makes sense.
Let’s get back to basic.
Did I mention DHCP is UDP? So how a next generation firewall (like paloalto) with all the fancy features enable (we have nearly all of them enable…) treats a UDP connection? UDP is stateless… but the firewall is statefull… the firewall creates a flow with the first packet so it can track, any new packet is considered part of that flow. But why we dont see the flows? We actually have only one flow. The firewall has created that session and offloaded to hardware. So you dont see anything else in the control-plane / GUI. The GUI only shows the end of a connection/flow. And as our flow DHCP Relay hasnt’ terminated (it is UDP) and the firewall keeps receiving packets, it is considered life (the firewall doesnt really know what is going on). So for that reason we dont see the connection in the PaloUI. Ok, I got to that point after a while…. I need to proof that the packet from the server is reaching the firewall and it is leaving it too.
How can I do that? Well, I need to delete that flow so the firewall considers a new connection and the tcpdump can see the packets.
This is the a good link from paloalto to take captures. So I found my connection and the cleared it:
palo(active)> show session all filter destination 10.94.240.1
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
135493 dhcp ACTIVE FLOW 10.81.251.201[67]/ZONE1/17 (10.81.251.201[67])
vsys1 10.94.240.1[67]/ZONE2 (10.94.240.1[67])
palo(active)>
palo(active)> clear session id 135493
And now, my packet capture in paloalto confirms that it is sending the packet to the next firewall (checking the destination MAC) !!!
Ok, so we confirm the first firewall in the return path was fine…. next one, it is fortigate.
BTW, we were checked and assumed that the routing is fine in all routers, firewalls, etc. Sometimes is not the case… so when things dont follow your thoughts, get back to the very basics….
We have exactly the same issue as in PaloAlto. I can’t see anything in the logs about receiving a dhcp offer from palo and forwarding it to the last firewall Cisco.
And again, we apply the same reasoning. We have an UDP connection, we have a next-generation firewall (with fancy ASIC). And one more thing, in this fortigate firewall, we allow intra-zone traffic, so it is not going to show anyway in the GUI monitor…
Fantastic, we have confirmation that the second firewall receives and forwards the DHCP Reply!!!
Ok, now the last stop, Cisco ASA. This is an old firewall, I think it could be my father or Darth Vader.
I dont have the fancy tools for packet capture like Palo/Fortigate…. so I went to the basic “debug” commands and “packet-tracer”.
First, this was the dhcp config in Cisco:
vader/pri/act# show run | i dhcp
dhcprelay server 10.81.251.47 EGRESS
dhcprelay enable SERVERS-ZONE
dhcprelay timeout 60
And, the ACL allows all IP traffic in those interfaces… and couldnt see any deny in the logs.
So, I enabled all debugging things I could find for dhcp:
vader/pri/act# show debug
debug dhcpc detail enabled at level 1
debug dhcpc error enabled at level 1
debug dhcpc packet enabled at level 1
debug dhcpd packet enabled at level 1
debug dhcpd event enabled at level 1
debug dhcpd ddns enabled at level 1
debug dhcprelay error enabled at level 1
debug dhcprelay packet enabled at level 1
debug dhcprelay event enabled at level 200
vader/pri/act# DHCPD: Relay msg received, fip=ANY, fport=0 on SERVERS-ZONE interface
DHCPRA: relay binding found for client f48e.38c7.1b6e.
DHCPD: setting giaddr to 10.94.240.1.
dhcpd_forward_request: request from f48e.38c7.1b6e forwarded to 10.81.251.47.
DHCPD: Relay msg received, fip=ANY, fport=0 on SERVERS-ZONE interface
DHCPRA: relay binding found for client 6c2b.59c1.3273.
DHCPD: setting giaddr to 10.94.240.1.
dhcpd_forward_request: request from 6c2b.59c1.3273 forwarded to 10.81.251.47.
vader/pri/act#
So, the debugging doesnt says anything regarding the packet coming back from Fortigate… Not looking good I am afraid. I wasnt running out of ideas about debug commands. I coudn’t increase an log level neither….
Let’s give a go to packet tracer… doesnt looks good:
vader/pri/act# packet-tracer input EGRESS udp 10.81.251.201 67 10.94.240.1 67
Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Result:
input-interface: EGRESS
input-status: up
input-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
So, we are sure our ACL is totally open but the firewall is dropping the packet coming from fortigate. Why? How to fix it?
Ok, get back to basics. Focus in Cisco config. It uses as DHCP relay server, 10.81.251.47 (VIP). But the DHCP reply is coming from the physical IP 10.81.251.201….. maybe Cisco doesnt like that…. Let’s try to add the physical IPs as a new DHCP server:
vader/pri/act# sri dhcp
dhcprelay server 10.81.251.47 EGRESS
dhcprelay server 10.81.251.201 EGRESS
dhcprelay server 10.81.251.202 EGRESS
I think I nearly cried after seeing this in the dhcp logs in our server:
May 12 16:16:27 dhcp1 dhcpd[2561]: DHCPDISCOVER from f4:8e:38:c7:1b:6e via 10.94.240.1
May 12 16:16:28 dhcp1 dhcpd[2561]: DHCPOFFER on 10.94.240.50 to f4:8e:38:c7:1b:6e (cmc-111) via 10.94.240.1
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 0 class decls to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 0 deleted host decls to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 0 new dynamic host decls to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 1 leases to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: DHCPREQUEST for 10.94.240.50 (10.81.251.202) from f4:8e:38:c7:1b:6e (cmc-111) via 10.94.240.1
May 12 16:16:28 dhcp1 dhcpd[2561]: DHCPACK on 10.94.240.50 to f4:8e:38:c7:1b:6e (cmc-111) via 10.94.240.1
So at the end, finally fixed…. it took too many hours.
Notes:
DHCP Realy: It is not that obvious the flow regarding IPs.
UDP and firewalls, debugging it is a bit more challenging.
Cisco ASA dhcprelay server IPs…. VIPs and non-VIPs please.
Another issue I had during the weekend that took me hours. Thanks that I have been reading a bit this book (1.6) and had some clues.
I was trying to test a repo to start an Arista lab using docker and I assumed that everything should work if I followed the instructions. My problem was the script trying to push some basic config to the switches.
As you can see that differs with what we have inside the bash script. So from the bash script we need to put between ‘ the parameter for -c but inside the parameter we need to use “. So I had to make the change below:
- docker exec "${1}" bash -c "echo -e ${commands} | FastCli -p15 -e"
+ # need to update this command as the quoting doesnt work in my bash
+ docker exec "${1}" bash -c 'echo -e '"'${commands}'"' | FastCli -p15 -e'
The books says Enclose a string in single quotes ‘ unless it contains elements that you want the shell to interpolate. So let’s divide the solution in parts so can be easier to digest (and remember for me in the future because this will bit me again for sure)
1st part: ‘echo -e ‘
2nd part: “
3rd part: ‘${commands}’
4th part: “
5th part: ‘ | FastCli -p15 -e’
The ” need to be outside the ‘ region because the commands need to be between ” for the docker command. The 3rd part will expand the variable commands.
I guess the author is using a different version of bash? This is mine
$ bash --version
GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software; you are free to change and redistribute it.
Something very basic but took me several hours to workout. I had a VM that I wanted to increase a VG as I wanted to create a new LV. I increased the partition in the host server so the PV of the VG had the extra space, but then I couldnt see the increase inside the VM:
"fdisk" was telling me the disk was already 300G...
[root@VM ~]# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
The old LVM2_member signature will be removed by a write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xd46fa2fc.
Command (m for help): p
Disk /dev/vdb: 300 GiB, 322122547200 bytes, 629145600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd46fa2fc
I did a pvscan... and nothing. What I was missing? just "pvresize".... and then I can see my extra 100G in the PV and in the VG. So I can create the new LV I wanted...
I was reading an article about tools to write python using VIM with good formatting. I am not a pro-user of VIM neither a pro-python programmer but I would like to be more efficient and write better formatted python code.
So this is the link I was reading and ended here for the specific details.
At the end, my goal is to use more often inside VIM: splits (:sp), nerdtree (file browsing), autocompletion, git and be sure my code is formatted automatically if I make a mistake.
So I enabled most of the plugins from the article although I made some tweaks for me normal usage (I already had enabled some pluggins for jinja2). This is my .vimrc:
set nocompatible " be iMproved, required
filetype off " required
" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
" https://realpython.com/vim-and-python-a-match-made-in-heaven/#syntax-checkinghighlighting
set splitbelow
set splitright
"split navigations
nnoremap
nnoremap
nnoremap
nnoremap
" Enable folding
set foldmethod=indent
set foldlevel=99
" Enable folding with the spacebar
nnoremap za
" Automatic formating for tab, whitespace and max 80 chars per line, etc
au BufNewFile,BufRead *.py
\ set tabstop=4 |
\ set softtabstop=4 |
\ set shiftwidth=4 |
\ set textwidth=79 |
\ set expandtab |
\ set autoindent |
\ set fileformat=unix
highlight BadWhitespace ctermbg=red guibg=darkred
au BufRead,BufNewFile *.py,*.pyw,*.c,*.h match BadWhitespace /\s\+$/
set encoding=utf-8
let python_highlight_all=1
" VUNDLE PLUGINGS
call vundle#begin()
" alternatively, pass a path where Vundle should install plugins
" call vundle#begin('~/some/path/here')
" let Vundle manage Vundle, required
Plugin 'VundleVim/Vundle.vim'
" added nerdtree
Plugin 'scrooloose/nerdtree'
Plugin 'ctrlp.vim'
Plugin 'Jinja'
Plugin 'tmhedberg/SimpylFold'
Plugin 'vim-scripts/indentpython.vim'
Plugin 'vim-syntastic/syntastic'
Plugin 'nvie/vim-flake8'
Plugin 'tpope/vim-fugitive'
Plugin 'Lokaltog/powerline', {'rtp': 'powerline/bindings/vim/'}
" Keep Plugin commands between vundle#begin/end.
" All of your Plugins must be added before the following line
call vundle#end() " required
filetype plugin indent on " required
syntax on
nmap :NERDTreeToggle
" To ignore plugin indent changes, instead use:
"filetype plugin on
"
" Brief help
" :PluginList - lists configured plugins
" :PluginInstall - installs plugins; append `!` to update or just :PluginUpdate
" :PluginSearch foo - searches for foo; append `!` to refresh local cache
" :PluginClean - confirms removal of unused plugins; append `!` to auto-approve removal
"
" see :h vundle for more details or wiki for FAQ
" Put your non-Plugin stuff after this line
au BufNewFile,BufRead *.lmx set filetype=xml
au BufNewFile,BufRead *.dump set filetype=sql
au BufNewFile,BufRead *.j2 set filetype=jinja
"for case-insensitve searches"
set ignorecase
"Override the 'ignorecase' option if the search pattern contains upper"
"case characters. Only used when the search pattern is typed and"
"'ignorecase' option is on."
set smartcase
" I want to be able to resize the splits quickly so I want the mouse on
set mouse=a
" Always show statusline - This makes powerline always on
set laststatus=2
" autocmd vimenter * NERDTree
BTW, a quick reference for NerdTree (file browser) here.
I am reading The General Theory of Employment, Interest and Money of John Maynard Keynes. The intro is really good and give you a good glimpse of Keynes’ life. The book itself is quite hard for me to digest. It is not my field but I am courius about Economics/Finance. One day, I would like to understand how money really works…. I think I bought the book after references from other books from Yanis Varoufakis.
Things I have learned he was against the Treaty of Versailles as the conditions over Germany were too abusive and that could cause an economic disaster for future generations. Well, you have created the easiest scenario for a crazy guy to play with the feelings of a proud nation to raise and get back what it was theirs…. Yeah, very simplify, but it works in my head. He wrote about it in “Consequences of the Peace”.
I think we have seen similar scenarios (without war involved thanks x (x = choose your god’s religion)) with Greece and Argentina.
Then after WWII, he helped to create the “International Monetary Fund” and “World Bank”. And attended Bretton Woods Conference. One of the important points of that conference was the fix rate gold/dollar and the lack of banking crisis as a consequence of this agreement. As well, I think the book from “The Signal and the Noise” says that during this conference, the American president couldnt understand Keynes ideas as he couldnt explain them easily. That conference is important because when the deal was cancelled in 1971 it caused some interesting economic shocks (books from Varoufakis are good) as countries couldn’t change dollar/gold as before and the banking crisis started to roll.
As per my understaning, Keynes believed that economic balance (mainly low unemployment) was the basic of a peaceful world. If you have most of your needs satisfied, I dont think you will pick your rifle whenever a lunatic tells you to do so.
As well, he was pro-European…. look now, we are in Brexit times…
…look now… it seems unemployment is going to skyrocket all around the world…. and we have too many dumb presidents in too powerful countries…. look now… how far right has been gathering support in the last years….
I read once about how to do load-balancing when using Route-Reflectors (RR) in a MPLS L3VPN network. It is a insteresting topic because RRs only reflect the best prefixes to the its clients. So how we make the RR to send more than one?
We have one customer vrf “CUST-A” with three locations: TY, LD and NY.
We are using BGP for PE-CE routing. Each site will use a different private ASN. Our SP is ASN 100.
TY has two connection to our SP so we want to make use of both of them.
We have a RR SP2 that is in line. So we need a full-mesh iBGP from all PE to SP2.
Our SP IGP is OSFP.
The goal is to make all other PE connected to CUST-A sites to be able to load-balance to TY site prefixes 192.168.11.0/24 and 192.168.12.0/24 using TY-SP1 and TY-SP3.
We start building the whole network as standard. This is very similar as stated in our first lab:
This is RR SP2 config:
!
ip vrf CUST-A
rd 100:1
route-target export 1:100
route-target import 1:100
!
interface Loopback0
ip address 10.0.2.1 255.255.255.255
!
interface GigabitEthernet1/0
description to SP1-PE
ip address 10.0.12.2 255.255.255.0
negotiation auto
mpls ip
!
interface GigabitEthernet2/0
description to SP3-PE
ip address 10.0.23.2 255.255.255.0
negotiation auto
mpls ip
!
interface FastEthernet3/0
description TO-LD-SP4
ip address 10.0.24.2 255.255.255.0
duplex auto
speed auto
mpls ip
!
router ospf 1
log-adjacency-changes
network 10.0.2.0 0.0.0.255 area 0
network 10.0.12.0 0.0.0.255 area 0
network 10.0.23.0 0.0.0.255 area 0
network 10.0.24.0 0.0.0.255 area 0
!
router bgp 100
no synchronization
bgp log-neighbor-changes
neighbor 10.0.1.1 remote-as 100
neighbor 10.0.1.1 update-source Loopback0
neighbor 10.0.1.1 route-reflector-client
neighbor 10.0.3.1 remote-as 100
neighbor 10.0.3.1 update-source Loopback0
neighbor 10.0.3.1 route-reflector-client
neighbor 10.0.4.1 remote-as 100
neighbor 10.0.4.1 update-source Loopback0
neighbor 10.0.4.1 route-reflector-client
neighbor 10.0.5.1 remote-as 100
neighbor 10.0.5.1 update-source Loopback0
neighbor 10.0.5.1 route-reflector-client
no auto-summary
!
address-family vpnv4
neighbor 10.0.1.1 activate
neighbor 10.0.1.1 send-community both
neighbor 10.0.1.1 route-reflector-client
neighbor 10.0.3.1 activate
neighbor 10.0.3.1 send-community both
neighbor 10.0.3.1 route-reflector-client
neighbor 10.0.4.1 activate
neighbor 10.0.4.1 send-community both
neighbor 10.0.4.1 route-reflector-client
neighbor 10.0.5.1 activate
neighbor 10.0.5.1 send-community both
neighbor 10.0.5.1 route-reflector-client
exit-address-family
!
address-family ipv4 vrf CUST-A
no synchronization
exit-address-family
!
!
mpls ldp router-id Loopback0 force
The configs for the SP PE follow the same patern, this is TY-SP1:
!
ip vrf CUST-A
rd 100:1
route-target export 1:100
route-target import 1:100
!
interface Loopback0
ip address 10.0.1.1 255.255.255.255
!
interface FastEthernet0/0
description to HQ
ip vrf forwarding CUST-A
ip address 172.16.100.254 255.255.255.0
duplex half
!
interface GigabitEthernet1/0
description to SP2-P
ip address 10.0.12.1 255.255.255.0
negotiation auto
mpls ip
!
router ospf 1
log-adjacency-changes
network 10.0.1.0 0.0.0.255 area 0
network 10.0.12.0 0.0.0.255 area 0
!
router bgp 100
no synchronization
bgp log-neighbor-changes
neighbor 10.0.2.1 remote-as 100
neighbor 10.0.2.1 update-source Loopback0
no auto-summary
!
address-family vpnv4
neighbor 10.0.2.1 activate
neighbor 10.0.2.1 send-community both
exit-address-family
!
address-family ipv4 vrf CUST-A
neighbor 172.16.100.1 remote-as 65001
neighbor 172.16.100.1 activate
neighbor 172.16.100.1 soft-reconfiguration inbound
no synchronization
exit-address-family
!
mpls ldp router-id Loopback0 force
!
Let’ see if LD-CE1 can ping our TY-C1
LD-CE1#traceroute 192.168.12.1 source 172.16.30.1
Type escape sequence to abort.
Tracing the route to 192.168.12.1
1 172.16.101.254 8 msec 20 msec 8 msec
2 10.0.24.2 [MPLS: Labels 18/23 Exp 0] 40 msec 40 msec 36 msec
3 172.16.200.254 [MPLS: Label 23 Exp 0] 12 msec 32 msec 28 msec
4 172.16.200.1 60 msec 40 msec 40 msec
5 192.168.12.1 [AS 65001] 40 msec 60 msec 60 msec
LD-CE1#
LD-CE1#
LD-CE1#ping 192.168.11.1 source 172.16.30.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.11.1, timeout is 2 seconds:
Packet sent with a source address of 172.16.30.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/54/72 ms
LD-CE1#
LD-CE1#
LD-CE1#
LD-CE1#sh
LD-CE1#show ip rou
LD-CE1#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
B 192.168.12.0/24 [20/0] via 172.16.101.254, 01:19:31
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 172.16.30.1/32 is directly connected, Loopback0
C 172.16.101.0/24 is directly connected, FastEthernet0/0
B 192.168.11.0/24 [20/0] via 172.16.101.254, 01:19:31
LD-CE1#
So, what do we see when everything is configured?
From SP2-RR, we see all BGP peers up to PEs and in the vpnv4 table we can see the TY prefixes 192.168.11.0/24 and 192.168.12.0/24. But only the path from TY-SP1 is preferred….
SP2#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
10.0.4.1 1 FULL/DR 00:00:39 10.0.24.1 FastEthernet3/0
10.0.3.1 1 FULL/DR 00:00:39 10.0.23.1 GigabitEthernet2/0
10.0.1.1 1 FULL/BDR 00:00:37 10.0.12.1 GigabitEthernet1/0
SP2#
SP2#
SP2#show ip bgp summary
BGP router identifier 10.0.2.1, local AS number 100
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.1.1 4 100 98 111 1 0 0 01:25:16 0
10.0.3.1 4 100 93 108 1 0 0 01:25:05 0
10.0.4.1 4 100 96 114 1 0 0 00:55:06 0
10.0.5.1 4 100 29 32 1 0 0 00:28:02 0
SP2#
SP2#show ip bgp vpnv4 all
BGP table version is 9, local router ID is 10.0.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 100:1 (default for vrf CUST-A)
*>i172.16.30.1/32 10.0.4.1 0 100 0 65002 i
*>i192.168.11.0 10.0.1.1 0 100 0 65001 i
* i 10.0.3.1 0 100 0 65001 i
*>i192.168.12.0 10.0.1.1 0 100 0 65001 i
* i 10.0.3.1 0 100 0 65001 i
SP2#
Let confirm that the PE only receive the best prefix from the RR. So, from LD-SP4, we can see the paths to TY 192.168.11/12 via TY-SP1 only:
LD-SP4#show ip bgp vpnv4 all
BGP table version is 18, local router ID is 10.0.4.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 100:1 (default for vrf CUST-A)
*> 172.16.30.1/32 172.16.101.1 0 0 65002 i
*>i192.168.11.0 10.0.1.1 0 100 0 65001 i
*>i192.168.12.0 10.0.1.1 0 100 0 65001 i
LD-SP4#
How do we make RR-SP2 to learn and advertise TY-SP1 and TY-SP3 paths. We need to use different RD in TY-SP1/3 respectively.
We have RD 100:1 assigned to CUST-A in all PEs. We are going to change that in TY-SP1/3 so RR will see two different VPNv4 prefixes for the same destination.
Let’s change TY-SP1 RD 100:1 to 100:101 and TY-SP3 to 100:102. Watch out as all routing config related to VRF CUST-A will disappear.
And what about the RT config? Do we have to change anything? Actually, we need to keep it the same (we need to retype it), nothing changes here. Keep in mind that RT is used to import/export vpnv4 prefixes into the VRF. The RD is not used to import/export so for that reason (as we are going to see) we could actually use any RD for a VRF in a PE.
Let’s see the changes for TY-SP1:
TY-SP1(config)#ip vrf CUST-A
TY-SP1(config-vrf)#no rd 100:1
% "rd 100:1" for VRF CUST-A scheduled for deletion
TY-SP1(config-vrf)#
*Apr 27 22:28:48.347: %BGP-5-ADJCHANGE: neighbor 172.16.100.1 vpn vrf CUST-A Down Neighbor deleted
TY-SP1(config-vrf)#rd 100:101
% Deletion of "rd" in progress; wait for it to complete
TY-SP1(config-vrf)#
TY-SP1(config-vrf)#rd 100:101
TY-SP1(config-vrf)#route-target export 100:1
TY-SP1(config-vrf)#route-target import 100:1
TY-SP1(config-vrf)#exit
TY-SP1(config)#router bgp 100
TY-SP1(config-router)#address-family ipv4 vrf CUST-A
TY-SP1(config-router-af)# neighbor 172.16.100.1 remote-as 65001
TY-SP1(config-router-af)# neighbor 172.16.100.1 activate
TY-SP1(config-router-af)# neighbor 172.16.100.1 soft-reconfiguration inbound
TY-SP1(config-router-af)#
*Apr 27 22:33:50.571: %BGP-5-ADJCHANGE: neighbor 172.16.100.1 vpn vrf CUST-A Up
TY-SP1(config-router-af)#
So after repeating the same step in TY-SP3 (using RD 100:102), let’s see what happens in RR-SP2:
SP2#show ip bgp vpnv4 all
BGP table version is 51, local router ID is 10.0.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 100:1 (default for vrf CUST-A)
*>i172.16.30.1/32 10.0.4.1 0 100 0 65002 i
* i192.168.11.0 10.0.3.1 0 100 0 65001 i
*>i 10.0.1.1 0 100 0 65001 i
* i192.168.12.0 10.0.3.1 0 100 0 65001 i
*>i 10.0.1.1 0 100 0 65001 i
Route Distinguisher: 100:101
*>i192.168.11.0 10.0.1.1 0 100 0 65001 i
*>i192.168.12.0 10.0.1.1 0 100 0 65001 i
Route Distinguisher: 100:102
*>i192.168.11.0 10.0.3.1 0 100 0 65001 i
*>i192.168.12.0 10.0.3.1 0 100 0 65001 i
SP2#
Now we can see VPNv4 for 100:101 (TY-SP1) and 100:102 (TY-SP2)!!!
Ok, let’s what the other PE are seeing. In our case, let’s check LD-SP4:
LD-SP4#show ip bgp vpnv4 all
BGP table version is 18, local router ID is 10.0.4.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 100:1 (default for vrf CUST-A)
*> 172.16.30.1/32 172.16.101.1 0 0 65002 i
* i192.168.11.0 10.0.3.1 0 100 0 65001 i
*>i 10.0.1.1 0 100 0 65001 i
* i192.168.12.0 10.0.3.1 0 100 0 65001 i
*>i 10.0.1.1 0 100 0 65001 i
Route Distinguisher: 100:101
*>i192.168.11.0 10.0.1.1 0 100 0 65001 i
*>i192.168.12.0 10.0.1.1 0 100 0 65001 i
Route Distinguisher: 100:102
*>i192.168.11.0 10.0.3.1 0 100 0 65001 i
*>i192.168.12.0 10.0.3.1 0 100 0 65001 i
LD-SP4#
LD-SP4#
LD-SP4#show ip route vrf CUST-A
Routing Table: CUST-A
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
B 192.168.12.0/24 [200/0] via 10.0.1.1, 00:01:05
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
B 172.16.30.1/32 [20/0] via 172.16.101.1, 00:33:47
C 172.16.101.0/24 is directly connected, FastEthernet0/0
B 192.168.11.0/24 [200/0] via 10.0.1.1, 00:01:05
LD-SP4#
So, LD-SP4 is receiving the VPNv4 100:101 and 100:102 from RR-SP2!!! That’s good, but we are still seeing the path to TY 192.168.11/12 prefixes via TY-SP1 (10.0.1.1) only.
So why BGP ECMP is not working? Because we have to enable it.
LD-SP4(config)#router bgp 100
LD-SP4(config-router)#address-family ipv4 vrf CUST-A
LD-SP4(config-router-af)#maximum-paths eibgp 2
LD-SP4(config-router-af)#
*Apr 27 22:58:25.447: BGP: VPNv4 Unicast multipath configuration changed
*Apr 27 22:58:25.447: BGP-VPN(4): MPLS label changed for prefix 100:1:192.168.11.0/24
*Apr 27 22:58:25.447: BGP-VPN(4): multipath from neighbor 10.0.2.1 nexthop 10.0.3.1 new outlabel 24
*Apr 27 22:58:25.447: vpn: free local label 1048577 for remote prefix CUST-A:192.168.11.0/24
*Apr 27 22:58:25.447: vpn: get path labels: 100:1:192.168.11.0/255.255.255.0
*Apr 27 22:58:25.451: vpn(4): inlabel=nolabel, outlabel=22, outlabel owner=BGP
*Apr 27 22:58:25.451: vpn(4): Announce labels to IPRM CUST-A:192.168.11.0/24 gw 10.0.1.1 inlabel=nolabel, outlabel=22
*Apr 27 22:58:25.451: BGP-VPN(4): MPLS label changed for prefix 100:1:192.168.12.0/24
*Apr 27 22:58:25.451: BGP-VPN(4): multipath from neighbor 10.0.2.1 nexthop 10.0.3.1 new outlabel 23
*Apr 27 22:58:25.451: vpn: free local label 1048577 for remote prefix CUST-A:192.168.12.0/24
*Apr 27 22:58:25.451: vpn: get path labels: 100:1:192.168.12.0/255.255.255.0
*
LD-SP4(config-router-af)#endApr 27 22:58:25.451: vpn(4): inlabel=nolabel, outlabel=21, outlabel owner=BGP
*Apr 27 22:58:25.451: vpn(4): Announce labels to IPRM CUST-A:192.168.12.0/24 gw 10.0.1.1 inlabel=nolabel, outlabel=21
*Apr 27 22:58:25.455: vpn: get path labels: 100:1:192.168.11.0/255.255.255.0
*Apr 27 22:58:25.459: vpn(4): inlabel=nolabel, outlabel=24, outlabel owner=BGP
*Apr 27 22:58:25.459: vpn(4): Announce labels to IPRM CUST-A:192.168.11.0/24 gw 10.0.3.1 inlabel=nolabel, outlabel=24
*Apr 27 22:58:25.459: vpn(4): get path labels; 100:1:192.168.11.0/24 nexthop 10.0.3.1, not bestpath
*Apr 27 22:58:25.475: vpn: get path labels: 100:1:192.168.12.0/255.255.255.0
*Apr 27 22:58:25.475: vpn(4): inlabel=nolabel, outlabel=23, outlabel owner=BGP
*Apr 27 22:58:25.475: vpn(4): Announce labels to IPRM CUST-A:192.168.12.0/24 gw 10.0.3.1 inlabel=nolabel, outlabel=23
*Apr 27 22:58:25.479: vpn(4): get path labels; 100:1:192.168.12.0/24 nexthop 10.0.3.1, not bestpath
LD-SP4(config-router-af)#end
LD-SP4#
*Apr 27 22:58:27.411: %SYS-5-CONFIG_I: Configured from console by console
LD-SP4#
LD-SP4#
LD-SP4#show ip route vrf CUST-A
Routing Table: CUST-A
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
B 192.168.12.0/24 [200/0] via 10.0.3.1, 00:00:07
[200/0] via 10.0.1.1, 00:02:18
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
B 172.16.30.1/32 [20/0] via 172.16.101.1, 00:35:00
C 172.16.101.0/24 is directly connected, FastEthernet0/0
B 192.168.11.0/24 [200/0] via 10.0.3.1, 00:00:07
[200/0] via 10.0.1.1, 00:02:18
LD-SP4#
We finally got it! Our PE LD-SP4 is able to see two paths to TY prefixes!
In summary:
We need to change the VRF RD in the PE we want to be participant in load-balancing
