Advance Junos Troubleshooting

SWITCHING

inter-vlan
- verify network settings for each user: ip, mac, gw
- verify gw reachability
- verify MAC table:
show ethernet-switching table (control plane)
show route forwarding-table family ethernet-switching (forwarding plane - PFE) both should be in sync
- verify STP: show spanning-tree interface
- verify vlan-interface association: show vlans X [detail]
- verify interface config: show configuration interface XXX
- where is gw: show route table inet.0
- fw session: show security flow session protocol icmp source-prefix IP

Route "rejected" for irb => check vlan config has l3-interface configured

LAG not coming up but config looks ok -> show configuration chassis aggregated-devices => device-count!

EVPN

east-west

- verify network settings for each user: ip, mac, gw
- verify gw reachability

- evpn-vxlan:
-- underlay: IGP (ospf-isis) or eBGP (loopbacks exchanged, peers up, etc)
---show bgp summary / show route receive/advertised-protocol bgp IP table inet.0
---show evpn databases

-- overlay: iBGP up, EVPN peers up
--- show evpn instance extensive
--- show evpn database
--- show route table inet.0 protocol evpn

-- VXLAN: VTEPs established, up, EVPN routes (typ2/5 exchanged?
--- show interfaces vtep
type source: for the locally attached networks
type Remote: for remote networks
--- show vlan (mapping vlan-vni)
--- show vlan X details -> l3-irbs, mapped interfaces, etc

-- communication between same vlan/vni? vlan-vni mapping identical in all devices?, IRB using commong VGA, IRB bound to VLANs
--- show ethernet-switching vxlan-tunnel-end-point source
--- show ethernet-switching vxlan-tunnel-end-point remote
--- show interfaces terse irb
--- show configuration interfaces irb
--- show configuration vlans
--- show ethernet-switching table

show bgp summary
- bgp.evpn.0 receives all EVPN routes (LE and SP)
- default-switch.evpn.0 - typ1,2,3 copied here (only LE, no in SP)
- default_evpn.evpn.0 - typ4 copied here (only LE, no in SP)

show route summary
:vxlan.inet.0 - holds the routes to remote VTEPs and uses inet.0 for route selection (because it has : in the name)

OSPF

ospf operational
show ospf neighbor
show ospf interface (loopbacks are in??)
show interfaces x | match mtu
show ospf overview
show ospf statistics
set protocols ospf traceoptions file X size 5m files 3 flag all
show log X
show configuration interface Y
monitor traffic interface Y detail matching "proto ospf"
request system decrypt password "xxxx" (onlu for $8$ or $9$ hashes)

adj state full
show ospf database (* = entries generated by the local router)
show route protocol ospf table inet.0

spf stable: duplicate RID?, flapping links?

routes in RIB: lsdb, asbr over non-bb area, 0/0 in nssa and stubs, prefix export limit, import/export policies
- show route table inet.0
- abr: nssa no-summaries -> routes in that area dont see any summary LSA for other areas => you may need to advertise a default (set protocols ospf area X nssa default-lsa default-metric 1)
- show configuration protocols ospf

ISIS

(initializing -> mtu issue (at least 1492)

isis operational
show isis adjacency
show isis interface (loopbacks are in??)
show interfaces x | match mtu
show isis overview (if attached bit evaluation disabled -> default route (sent by L1L2) is not installed
show isis statistics
set protocols isis traceoptions file X size 5m files 3 flag error detail flag hello detail
show log X
show configuration interface Y
monitor traffic interface Y detail matching "proto isis"
request system decrypt password "xxxx" (onlu for $8$ or $9$ hashes)

adj state full
show isis database (* = entries generated by the local router)
show isis route
show route protocol isis table inet.0

spf stable: duplicate NET?, flapping links?

routes in RIB: lsdb, levels?, 0/0 in l1, prefix export limit, import/export policies
show route table inet.0
abr: nssa no-summaries -> routes in that area dont see any summary LSA for other areas => you may need to advertise a default (set protocols ospf area X nssa default-lsa default-metric 1)
show configuration protocols ospf
show isis hostname
show route table iso.0 <-- NET addresses!

BGP

iBGP
-ibgp established: security policies, fw rules, IGP routing, local address, auth, logs
set protocols bgp group int-group traceoptions file X size 5m files 3 flap open detail
show system connections

-ibgp stable: IGP stable? MTU, TCP MSS

-ibgp adverts: export policy, inactive routes, no-advertise community, bgp family

-ibgp receives: BGP next-hop, recursive routing

-routes in RIB: import policy

ROUTING LOOPS

take several traceroute
check where route redistribution takes place
set cli timestamp
show bgp summary (several times to see flapping)
show route received-protocol bgp PEER_IP table x.0 (several times to see flapping)
show route NET/MASK (several times)
show route hidden detail table x.0

MPLS L3VPN

control plane:
- pe-ce routing protocols
-- show route table inet.0 (for P and PE Lo)
-- show isis database detail

- ldp/rsvp
-- show route table inet.3 (table with ingress LSPs from this router
-- show route table mpls.0 (next LSR in each LSP). Loop-up LSPs and VPN labels
-- show ldp session/neighbor
-- show ldp database
*RT mismatch -> drop (no stored neither hiddne => go to the other peer and show route advertisedd-protocol bgp PE-IP)

- bgp
-- show bgp summary
-- show route table X.inet.0 (l3vpn table)
-- show route advertising-protocol bgp PE-IP
-- show route receive-protocol bgp PE-IP (is bgp.l3vpn.0 table???)
-- show route receive-protocol bgp PE-IP detail table bgp.l3vpn.0
-- *vrf-table-lable -> a LSI logical interface label is created and mapped to the VRF routing table.

data plane:
-- ping mpls
-- traceroute mpls ldp PE-Lo

Stateful FW

srx -> session-based
- show security flow status -> flow/packet-based
- show security flow session
- set security flow traceoptions file X size 2m files 3 flag basic-datapath packet-filter f1 protocol icmp destination-prefix FW-IP/32
- show interface X -> check security zone (Null drops all packets)
- show interface x detail | find Flow
- show security zones NAME
- show security screen statistics zone untrust | except 0
- show log messages | match RT_SCREEN or RT_IDS
- show security screen ids-option untrust-screen
- show configuration security screen
- set security zones security-zone X host-inbound-traffic system-override traceroute ping …

CHASSIS CLUSTER

OOB

fxp0: oob mgmt interface
fxp1/em0 - cp interface
fab - dp interace
swfab - switching dp interface
reth - redundant interface

show chassis cluster status
show chassis cluster interfaces
show interfaces terse | grep fxp
show configuration | display set | match apply-groups
set apply-groups ${node}
show system users

fabric link

show chassis cluster interfaces
show chassis cluster status
show chassis cluster statistics (sent/received??)
show interfaces terse | grep fab

data-plane

show chassis cluster status
show chassis cluster interfaces
show interfaces terse | grep reth
show configuration chassis cluster

*routing table is not running in secondary node

CoS

CoS-Based IPSec VPN

show security ike security-associations
show security ipsec security-associations
show security ipsec inactive-tunnels
show configuration security ipsec vpn X
show class-of-service forwarding-class

CoS implementation

show interfaces X detail | find Egress -> check counters in queues

show configuration firewall family inet -> check filtes
show interfaces filters | match X
show class-of-service interface Y
show class-of-service rewrite-rule type dscp
scheduler-map X | match "Scheduler|Transmit"
classifier type dscp name X

DeepSeek, AWS HPC SDR vs Multipath-TCP, OCSP death, AlphaChip, Visual AI agents, Ollama, Local AI, Bob Bowman

Nice analysis about DeepSeek without hype.

AWS HPC: Didn’t know AWS offered HPC services (articule from 2021). I liked to find more details about SDR: Multipath LB, Out of Order delivery, congestion control similar to BBR. I wonder, this is not the same as UltraEthernet consortium is trying to achieve?

Multipath-tcp: The above probably works in “close” networks (managed by one entity) but maybe it is not going to work in the Wild internet. Still this looks still quite far from production. I believe this like QUID. Somebody like google deploys it and the rest jump in the wagon (more or less)

OCSP death: “OCSP is not making anyone more secure. Browsers are either not checking it or are implementing it in a way that provides no security benefits.Β “

AlphaChip: As far as I have read, designing chip is one of the most complex things and getting help from AI can even increase the advances in chip design. I read that NVIDIA had something similar. And this should be applied to ASICs too so networking is benefited

Vision Agent AskUI: need to try

ByteDance UI Agent – UI-TARS: as above

Crawl4AI: Interesting for digestion your local knowledge base sites and using with your local LLM….

Run your locally AI: I tried this in my work MacBook and it worked! I want to create an AI agent for a work project (actually i am dreaming to be able to achieve it….)

Open Web UI + Ollama: I tested this too in my MacBook and works like magic! You can even use DeepSeek πŸ™‚

Bolt.diy + DeepSeek: I didnt manage to install bolt.diy ….

Training your AI: My idea is to get an open-source LLM trained with my data so I can use it to do my “job” But in the video there was too much publicity and I dont have access to a GPU… but I dont much data neither (or that’s what I think)

Bob Bowman (Michael Phelps coach): Show up, do the job.

AusNOG 2024

A bit late to review, but some interesting talks. Agenda.

Arista: Practical AI Networking Innovations:

rail optimized
all reduce

low entropy, 2-3 flows per nic, elephant, bursty

JCT job completion time
TSN time spent networking

overprovisioning 1:1.2

Nokia: NUTS python network testing:

It looks quite nice, it is based on pytest and nornir/napalm. It looks similar to batfish?

Measuring Starlink:

High jitter, each 15sec change satellite -> micro-loss. BBR (non-loss sensitive) is the best flow protocol protocol with Starlink. ECN.

Photonics in Computing, Usb cable hack, Stutz, Building AI Networks Arista

Lightmatter: Based on this video, they are using photonics to connect chips, looks interesting, I remember Google has something with optical but for networking. But It is pretty clear this is not photonics computing.

Hacking USB cable: impressive, and expensive πŸ™‚

Phil Stutz: Interesting conversation. But somehow, I am still looking for that thing that unlocks me…. can’t find it for the life of me….

Building AI Networks Arista:

- allreduce: collect elements from all nodes, apply a reduction operator(eg sum) then distribute reduction to all nodes
-allgather: collect elements from all nodes, and distribute the to all other nodes
- gpu: cpu for parallelization
- RDMA: RoCE2 GPU memory to GPU memory - origin in IB
- issues: flow collision, trafic polarization. low entropy!!! > dificult to ecmp => Dynamic LB
incast: many2one -> ECN + buffering (in spine!)
- use chassis!

With an operations hat on, dealing with chassis is expensive and no efficient. It kind of a vendor lock-in. AWS is all in pizza boxes and I remember one presentation in Cisco Live where the Cisco EVPN authority recommended pizza boxes.

Potato Pizza, TCP Conversation Completeness, IBM power10, AI developer kit, 2 not 3

This is a pizza that I tried several years ago, and it seems the restaurant is out of business. I have done some pizzas trying to emulate it but never matching that memory. So this is a placeholder to try:

Original Ingredients: Bechamel, Smoked Mozzarella, red onions, pancetta, sliced potatoes, and Buffalo Mozzarella.

Some example1, example2


This a old for today’s news. IBM Power10 Memory network but looks interesting:

...thanks to the coherence IBM already has across NVLink (which is really BlueLink running a slightly different protocol that makes the GPUs think DRAM is really slow but really fat HBM2, in effect, and also makes the CPUs think the HBM2 on the GPUs is really skinny but really fast DRAM). 

Checking some wireshark traces last week, I cam across the concept of TCP Conversation Completeness. This was totally new for me. This video gave some idea too. This was useful for me for finding TCP conversation that showed retransmissions when trying to stablish the TCP handshake, and not just showing the retransmission, so I used “tcp.completeness<33” so I should see TCP flows with a sync + RST.


AI developer Kit by NVIDIA: This card looks nice, I would mind to buy it and give a chance to learn, but it is sold out everywhere…. This is a video about it.


2 not 3 (make a choice!):

Quantum AI Chip, InfraHub, python UV, SR controller, Kobe Bryant, Hell

Google Quantum AI: This looks remarkable.

python vu: replacement for pip, pyenv, etc. Need to try

InfraHub: As a network engineer interested in Automation. This looks interesting and I would like to go deeper to fully understand as it is the merge of the typical source of truth (DB) that you can’t get in git.

Segment Routing Controller: This is another thing I played with some years ago, but never found a controller to make TE. I dont see clearly this software is OSS but at least doesnt look like is a vendor-lock…

Kobe Bryant: venting, and it is ok.

Jordan B Peterson: Hell

AWS re:Invent 2024, Oracle Cloud AI, GenCast, videos

AWS re:Invent 2024 – Monday Night:

  • Graviton evolution: ARM based chip for EC2. 50% new capacity of last 2y is Graviton.
  • Nitron Cards: security chip too.
  • AES Trainium2: min 47. 2xHead per rack and then accelerators, and switch. Trainnium != CPU|GPU. And this is a great analysis about Trainium2
  • Neurnlink: min 60, I guess this is the equivalent of NVLink, etc
  • Ultraserver, quite beefy pic, min 61.
  • Networking: min 73: 10p10u is a fabric = 10petabits under 10micro latency.
  • Cabling proprietary trunk connector 16:1 fiber. min 77. I pretty use i have used pig-tails some years ago, so not sure why this is new?
  • Firefly optic plug: loopback testing. This is interesting for DC operations. Min 78.
  • AWS design their own optics, reduced failure
  • Network topology: Min 81, new protocol SIDR – Scalable Intent Driven Routing. <1s reconvergence. not centralized.
  • And this is a better summary than mine.

AWS re:Invent 2024 – NET201: The only interesting thing is minute 29 with the usage of hollow core fiber, to improve latency. I assume it is used in very specific parts of the network, looks a bit fragile. Elastic Fabric Adapter, not really good explanation what it is, where doest it run: network, server, nic? but it seems important. Looks like SIDR?

AWS re:Invent 2024 – NET403: I think 401 and 402 were more interesting. There were repeated things from the two other talks. Still worth watching and hopefully there is a new one in 2025.

Oracle Cloud Infra – AI: First time I visit the OCI page about their AI infra.

GenCast: weather predict by Google Mind. Not sure until what point, this can be used by anybody? And how much hardware you need to run it?

we’ve made GenCast an open model and released its code and weights, as we did for our deterministic medium-range global weather forecasting model.

Videos:

510km nonstop – Ross Edgley: I have read several of his books and it is the first time I watch a full interview. Still I am not clear what his dark side is.

A man with few friends or not circle at all – Jordan B Peterson: I need to watch this more often

TPUv6, Alphafold, OOB design, OpenInterpreter, Walkie-Talkies, Zero Trust SSH, Videos, Finger Strength

Google TPUv6 Analysis: “… cloud infrastructure and which also is being tuned up by Google and Nvidia to run Google’s preferred JAX framework (written in Python) and its XLA cross-platform compiler, which speaks both TPU and GPU fluently.” So I guess this is a cross-compiler for CUDA?

“The A3 Ultra instances will be coming out β€œlater this year,” and they will include Google’s own β€œTitanium” offload engine paired with Nvidia ConnectX-7 SmartNICs, which will have 3.2 Tb/sec of bandwidth interconnecting GPUs in the cluster using Google’s switching tweaks to RoCE Ethernet.” So again custom ethernet tweaks for RoCE, I hope it makes to the UEC? Not sure I understand having a Titanium offload and a connectx-7, are they not the same?

Alphafold: It is open to be used. Haven’t read properly the license.

OOB Design:

Open Interpreter: The next step in LLMs is to control/interact with your system.

In my laptop fails because I have the free version πŸ™ need to try a different one, but looks promising!

open-interpreter main$ interpreter --model gpt-3.5-turbo

●

Welcome to Open Interpreter.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

β–Œ OpenAI API key not found

To use gpt-4o (recommended) please provide an OpenAI API key.

To use another language model, run interpreter --local or consult the documentation at docs.openinterpreter.com.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

OpenAI API key: ********************************************************************************************************************************************************************


Tip: To save this key for later, run one of the following and then restart your terminal.
MacOS: echo 'export OPENAI_API_KEY=your_api_key' >> ~/.zshrc
Linux: echo 'export OPENAI_API_KEY=your_api_key' >> ~/.bashrc
Windows: setx OPENAI_API_KEY your_api_key

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

β–Œ Model set to gpt-3.5-turbo

Open Interpreter will require approval before running code.

Use interpreter -y to bypass this.

Press CTRL-C to exit.

> what is my os?
Traceback (most recent call last):

Walkie-Talkies: Out of James Bond world.

Zero Trust SSH. From Cloudflare. And this video I watched some months ago (and it is already 4y).

Finger Strength: I follow similar protocol, although not everyday, for warm up and I think it works. I am not getting that super results but at least my fingers are stronger…. and I am not getting injuries!!!! \o/

Cisco AI/ML DC Infra Challenges: I am not quiet fan of Cisco products but this is a good overview.

Key points:

  • Create different networks (inter-GPU, front-end, storage, mgmt),
  • Inter-GPU:
    • – non-blocking, rails-optimized (fig.3)
  • Inter-GPU challenges:
  • – Packet loss: Use PFC +ECN (flow aware)
  • – Network delay: “Rich” QoS – proprietary QoS to handle mice flows. Needs good telemetry
  • – Network congestion: Some kind of communication switch-NIC
  • – Non-uniform utilization: Most vendors have something proprietary here, some dynamic LB and static-pinning?
  • – Simultaneous Elephant flows with large bursts: dynamic buffer protection (proprietary)

Videos:

  • Raoul Pal: Crypto Investment. His company. Go long run, invest a bit you can lose
  • Scott Galloway: Interesting his political analysis. Trump won and it seems Latins voted massively for him.
  • Bruce Dickinson: I read Bruce’s books some years ago so I was surprised to see him in a podcast. Need to finish it.
  • Eric Schmidt: I read one of his books some time ago so again, surprised to find him in a podcast. Still think Google has become evil and most of the good things he says are gone.
  • Javier Milei: I am not economist but it “seems” things are improving in Argentina. He is a character nonetheless. Need to finish it.
  • Matthew McConaughey: His book was really refreshing, and seeing him talking is the same. Raw, real.
  • Alex Honnold: You have to try hard if you want to do hard things.

SemiAnalysis – 100k cluster

This is site that a friend shared with me some months ago. And it is PURE gold from my point of view. They share a lot info free but not all, you have to subscribe/pay for the whole report. I would pay for it if my job were in that “business”

This is the link for a 100k GPU cluster.

It covers all details for building such infrastructure up to the network/hardware side. So from power distribution, cooling, racking, network design, etc. All is there.

It is something to read slowly to try to digest all the info.

This report for electrical systems (p1) shows the power facilities can be as big as the datacenter itself! So it is not rear to read hyperscalers want nuclear reactors.

MS GB200 rack, Malaysia DC boom, Oracle DCs, FuriosaAI, OCP concrete, IBM Mainframe Telum II, NotebookLM youtube summary, EdgeShark, RSA Quantum, OCP24 Meta

It seems Malaysia is getting a DC boom, but it based on coal???

This is a MS NVIDIA GB200 based rack. I am quite impressed with the cooling systems being twice as big as the compute rack! And yes, MS is sticking with IB for AI networking.

I didnt know that Oracle OCI was getting that big in the DC/AI business. And they were related to xAI. Their biggest DC is 800 megwatts… and a new one will have three nuclear reactors??

FuriosaAI: A new AI accelerator in the market. Good: cheap, less power. Bad: memory size.

OCP concrete: Interesting how far can go the OCP consortium.

IBM Mainframe Telum II: You think the mainframes business doesnt exist. Well, it is not. Honestly, at some point, I would like to fully understand the differences between a “standard” CPU and a mainframe CPU.

NotoebookLM: It seems it is possible to make summary of youtube videos! (and free)

EdgeShark: wireshark for containers. This has to be good for troubleshooting

22-bit RSA broken with Quantum computer: I think Quantum computing is the underdog in the current “all-is-AI” world. Schneier says we are ok.

OCP24 Meta AI: It is interesting comparing the Catalina rack with the one from MS above. The MS has the power rack next to it but FB doesnt show it, just mention Orv4 supports 140kW and it is liquid cooled. I assume that will be next to Catalina like MS design. And AMD GPU are getting into the mix with NVIDIA. It mentions Disaggregated Scheduled Fabric (DSF), with more details here. And here from STH more pictures.