Skip to content

Instantly share code, notes, and snippets.

@trozet

trozet/gist:4f54ec9f3fbdb6bc17c5515360cde1c3

Created January 18, 2024 02:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save trozet/4f54ec9f3fbdb6bc17c5515360cde1c3 to your computer and use it in GitHub Desktop.
Save trozet/4f54ec9f3fbdb6bc17c5515360cde1c3 to your computer and use it in GitHub Desktop.
Download ZIP
ndp ocp mac change
Raw
gistfile1.txt
pkt received on worker node:
01:30:41.176263 M 00:07:35:c0:23:cd ethertype IPv6 (0x86dd), length 88: (flowlabel 0x8e949, hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::207:35ff:fec0:23cd > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fd2e:6f44:5dd8:c956::18, Flags [override]
destination link-address option (2), length 8 (1): 00:07:35:c0:23:cd
datapath flow:
recirc_id(0xd),in_port(1),ct_state(-new-est-rel+trk),ct_mark(0),eth(src=00:07:35:c0:23:cd,dst=33:33:00:00:00:01),eth_type(0x86dd),ipv6(src=fe80::207:35ff:fec0:23cd,dst=ff02::1,proto=58,hlimit=255,frag=no),icmpv6(type=136,code=0), packets:27611, bytes:2374546, used:0.002s, actions:2,check_pkt_len(size=1414,gt(sample(sample=100.0%,actions(meter(3),userspace(pid=4294967295,controller(reason=1,dont_send=0,continuation=0,recirc_id=25194,rule_cookie=0x25862262,controller_id=0,max_len=65535))))),le(drop)
mac is not changing:
[root@master-0 ~]# ovn-sbctl --no-leader-only find mac_binding logical_port=rtoe-GR_worker-0.ostest.test.metalkube.org ip=fd2e\\:6f44\\:5dd8\\:c956\\:\\:18
_uuid : e429f7bf-8926-42ad-b88c-2c951168a7fd
datapath : 5d879988-cebc-4aea-a3a9-cf1dc176289e
ip : "fd2e:6f44:5dd8:c956::18"
logical_port : rtoe-GR_worker-0.ostest.test.metalkube.org
mac : "00:07:35:c0:23:c9"
timestamp : 1705532264322
### ofproto trace, but I forgot to set nw_ttl and nd_ttl, clusterbot was reaped right after this command
[root@worker-0 ~]# ovs-appctl ofproto/trace br-ex "in_port=1,icmp6,icmpv6_type=136,nd_target=fd2e:6f44:5dd8:c956::18,dl_src=00:07:35:c0:23:cd,dl_dst=33:33:00:00:00:01,ipv6_src=fe80::207:35ff:fec0:23cd,ipv6_dst=ff02::1"
Flow: icmp6,in_port=1,vlan_tci=0x0000,dl_src=00:07:35:c0:23:cd,dl_dst=33:33:00:00:00:01,ipv6_src=fe80::207:35ff:fec0:23cd,ipv6_dst=ff02::1,ipv6_label=0x00000,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no,icmp_type=136,icmp_code=0,nd_target=fd2e:6f44:5dd8:c956::18,nd_sll=00:00:00:00:00:00,nd_tll=00:00:00:00:00:00
bridge("br-ex")
---------------
0. ipv6,in_port=1, priority 50, cookie 0xdeff105
ct(table=1,zone=64000)
drop
-> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 1.
-> Sets the packet to an untracked state, and clears all the conntrack fields.
Final flow: unchanged
Megaflow: recirc_id=0,eth,icmp6,in_port=1,dl_dst=33:33:00:00:00:01,nw_frag=no
Datapath actions: ct(zone=64000),recirc(0xd)
===============================================================================
recirc(0xd) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================
Flow: recirc_id=0xd,ct_state=new|trk,ct_zone=64000,eth,icmp6,in_port=1,vlan_tci=0x0000,dl_src=00:07:35:c0:23:cd,dl_dst=33:33:00:00:00:01,ipv6_src=fe80::207:35ff:fec0:23cd,ipv6_dst=ff02::1,ipv6_label=0x00000,nw_tos=0,nw_ecn=0,nw_ttl=0,nw_frag=no,icmp_type=136,icmp_code=0,nd_target=fd2e:6f44:5dd8:c956::18,nd_sll=00:00:00:00:00:00,nd_tll=00:00:00:00:00:00
bridge("br-ex")
---------------
thaw
Resuming from table 1
1. icmp6,icmp_type=136, priority 14, cookie 0xdeff105
FLOOD
bridge("br-int")
----------------
0. in_port=7,vlan_tci=0x0000/0x1000, priority 100, cookie 0xdbf8b584
set_field:0x11->reg13
set_field:0x13->reg11
set_field:0x14->reg12
set_field:0x14->metadata
set_field:0x1->reg14
resubmit(,8)
8. metadata=0x14, priority 50, cookie 0x61e736b6
set_field:0/0x1000->reg10
resubmit(,73)
73. reg0=0x2, priority 0
drop
move:NXM_NX_REG10[12]->NXM_NX_XXREG0[111]
-> NXM_NX_XXREG0[111] is now 0
resubmit(,9)
9. metadata=0x14, priority 0, cookie 0x7b13790d
resubmit(,10)
10. metadata=0x14, priority 0, cookie 0xd74625ff
resubmit(,11)
11. metadata=0x14, priority 0, cookie 0x9c6902f6
resubmit(,12)
12. metadata=0x14, priority 0, cookie 0xf33625d5
resubmit(,13)
13. ipv6,reg14=0x1,metadata=0x14, priority 110, cookie 0xb5bfa54e
resubmit(,14)
14. metadata=0x14, priority 0, cookie 0x8c855215
resubmit(,15)
15. metadata=0x14, priority 65535, cookie 0x69fefa80
resubmit(,16)
16. metadata=0x14, priority 65535, cookie 0x5f076257
resubmit(,17)
17. metadata=0x14, priority 0, cookie 0x43a694d2
resubmit(,18)
18. metadata=0x14, priority 0, cookie 0xe0e56a5f
resubmit(,19)
19. metadata=0x14, priority 0, cookie 0xc612e10c
resubmit(,20)
20. metadata=0x14, priority 0, cookie 0x6a689319
resubmit(,21)
21. metadata=0x14, priority 0, cookie 0xed8f13e5
resubmit(,22)
22. metadata=0x14, priority 0, cookie 0xc07ecf7a
resubmit(,23)
23. metadata=0x14, priority 0, cookie 0xb99c3df2
resubmit(,24)
24. metadata=0x14, priority 0, cookie 0x63db46a0
resubmit(,25)
25. metadata=0x14, priority 0, cookie 0x4886ef92
resubmit(,26)
26. metadata=0x14, priority 0, cookie 0x8b14e046
resubmit(,27)
27. reg14=0x1,metadata=0x14, priority 100, cookie 0x5ee6d516
resubmit(,28)
28. metadata=0x14, priority 0, cookie 0xd3be213b
resubmit(,29)
29. metadata=0x14, priority 0, cookie 0xdbe40bd3
resubmit(,30)
30. metadata=0x14, priority 0, cookie 0x31d6c4c1
resubmit(,31)
31. metadata=0x14, priority 0, cookie 0x63a3a90e
resubmit(,32)
32. metadata=0x14, priority 0, cookie 0x5a81fcea
resubmit(,33)
33. metadata=0x14,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, priority 70, cookie 0x9427b4a3
set_field:0x8000->reg15
resubmit(,37)
37. priority 0
resubmit(,39)
39. priority 0
resubmit(,40)
40. reg15=0x8000,metadata=0x14, priority 100, cookie 0xfa6459e1
set_field:0x12->reg13
set_field:0x2->reg15
resubmit(,41)
41. priority 0
set_field:0->reg0
set_field:0->reg1
set_field:0->reg2
set_field:0->reg3
set_field:0->reg4
set_field:0->reg5
set_field:0->reg6
set_field:0->reg7
set_field:0->reg8
set_field:0->reg9
resubmit(,42)
42. metadata=0x14, priority 0, cookie 0xb23f4513
resubmit(,43)
43. ipv6,reg15=0x2,metadata=0x14, priority 110, cookie 0x805065eb
ct_clear
resubmit(,44)
44. metadata=0x14, priority 0, cookie 0xfbcb6414
resubmit(,45)
45. metadata=0x14, priority 65535, cookie 0xe7bd4014
resubmit(,46)
46. metadata=0x14, priority 65535, cookie 0x6f78a56
resubmit(,47)
47. metadata=0x14, priority 0, cookie 0xaa05a980
resubmit(,48)
48. metadata=0x14, priority 0, cookie 0xd09c8f14
resubmit(,49)
49. metadata=0x14, priority 0, cookie 0xb5d66170
resubmit(,50)
50. metadata=0x14,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, priority 100, cookie 0x5672d6f8
set_field:0/0x8000000000000000000000000000->xxreg0
resubmit(,51)
51. metadata=0x14, priority 0, cookie 0xcf672b4a
resubmit(,64)
64. priority 0
resubmit(,65)
65. reg15=0x2,metadata=0x14, priority 100, cookie 0xa0c0a1f4
clone(ct_clear,set_field:0->reg11,set_field:0->reg12,set_field:0->reg13,set_field:0x15->reg13,set_field:0x10->reg11,set_field:0x13->metadata,set_field:0x2->reg14,set_field:0->reg10,set_field:0->reg15,set_field:0->reg0,set_field:0->reg1,set_field:0->reg2,set_field:0->reg3,set_field:0->reg4,set_field:0->reg5,set_field:0->reg6,set_field:0->reg7,set_field:0->reg8,set_field:0->reg9,resubmit(,8))
ct_clear
set_field:0->reg11
set_field:0->reg12
set_field:0->reg13
set_field:0x15->reg13
set_field:0x10->reg11
set_field:0x13->metadata
set_field:0x2->reg14
set_field:0->reg10
set_field:0->reg15
set_field:0->reg0
set_field:0->reg1
set_field:0->reg2
set_field:0->reg3
set_field:0->reg4
set_field:0->reg5
set_field:0->reg6
set_field:0->reg7
set_field:0->reg8
set_field:0->reg9
resubmit(,8)
8. reg14=0x2,metadata=0x13,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, priority 50, cookie 0xa9c6a716
check_pkt_larger(1414)->OXM_OF_PKT_REG4[1]
set_field:0x735c023c50000000000000000/0xffffffffffff0000000000000000->xxreg0
resubmit(,9)
9. metadata=0x13, priority 0, cookie 0x626512e
set_field:0x4/0x4->xreg4
resubmit(,10)
10. reg9=0x4/0x4,metadata=0x13, priority 100, cookie 0x81be5249
resubmit(,11)
11. ipv6,reg9=0x2/0x3,reg14=0x2,metadata=0x13, priority 150, cookie 0x25862262
controller(userdata=00.00.00.14.00.00.00.00.00.19.00.18.80.01.09.10.00.00.00.00.00.00.00.01.00.00.00.00.00.00.00.01.00.19.00.18.80.01.09.10.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.02.00.19.00.10.80.00.06.06.00.07.35.c0.23.c5.00.00.00.1c.00.18.00.80.00.00.00.00.00.00.80.00.34.10.80.00.36.10.00.00.00.00.00.19.00.18.80.00.34.10.fd.2e.6f.44.5d.d8.c9.56.00.00.00.00.00.00.00.17.00.19.00.10.00.01.3a.01.ff.00.00.00.00.00.00.00.00.19.00.10.80.00.3a.01.02.00.00.00.00.00.00.00.00.19.00.10.80.00.3c.01.00.00.00.00.00.00.00.00.ff.ff.00.30.00.00.23.20.00.25.00.00.00.00.00.00.00.03.00.10.00.00.00.15.00.00.00.00.00.00.05.78.00.04.00.04.00.00.00.00.00.05.00.08.00.00.00.02.ff.ff.00.10.00.00.23.20.00.0e.ff.f8.08.00.00.00,meter_id=2)
set_field:0x735c023c50000000000000000/0xffffffffffff0000000000000000->xxreg0
resubmit(,9)
9. metadata=0x13, priority 0, cookie 0x626512e
set_field:0x4/0x4->xreg4
resubmit(,10)
10. reg9=0x4/0x4,metadata=0x13, priority 100, cookie 0x81be5249
resubmit(,11)
11. ipv6,metadata=0x13,dl_dst=33:33:00:00:00:00/ff:ff:00:00:00:00,ipv6_dst=ff00::/8, priority 82, cookie 0xf75cf173
drop
set_field:0x11->reg13
set_field:0x1->reg15
resubmit(,41)
41. reg10=0/0x1,reg14=0x1,reg15=0x1,metadata=0x14, priority 100, cookie 0xdbf8b584
drop
set_field:0x8000->reg15
Final flow: unchanged
Megaflow: recirc_id=0xd,ct_state=+new-est-rel+trk,ct_mark=0,eth,icmp6,in_port=1,dl_src=00:07:35:c0:23:cd,dl_dst=33:33:00:00:00:01,ipv6_src=fe80::/10,ipv6_dst=ff02::1,nw_ttl=0,nw_frag=no,icmp_type=0x88/0xff
Datapath actions: 2,check_pkt_len(size=1414,gt(sample(sample=100.0%,actions(meter(3),userspace(pid=4294967295,controller(reason=1,dont_send=1,continuation=0,recirc_id=32107,rule_cookie=0x25862262,controller_id=0,max_len=65535))))),le(drop))
#### steps to reproduce
1. launch cluster bot dualstack: launch 4.12.45 metal,dualstack
2. when cluster comes up create a pod on worker0, force it to send traffic to the ipv6 address of worker1. This will establish the mac binding for the address. I went a bit further and enabled ICNI for the pod, and set the gw ip to be ipv6 address of worker1.
3. verify the mac address is there in ovn: ovn-sbctl --no-leader-only find mac_binding logical_port=rtoe-GR_worker-0.ostest.test.metalkube.org ip=fd2e\\:6f44\\:5dd8\\:c956\\:\\:18
4. launch a debug pod on worker1, and then take down its primary interface on br-ex. This will sever the connection to the node permanently, but thats what we want.
5. create another pod on worker2:
[trozet@fedora network_logs]$ cat ~/basic-clone.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: client2
labels:
pod-name: client2
role: webserver
#app: spk-coredns
spec:
hostNetwork: true
containers:
- name: client2
image: fedora
#command:
# - /sbin/init
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 3000000; done;" ]
imagePullPolicy: IfNotPresent
ports:
- name: dns-tcp
containerPort: 53
protocol: TCP
- name: dns-udp
containerPort: 9999
protocol: UDP
# securityContext:
#sysctls:
#- name: net.ipv6.route.max_size
#value: 2048
# securityContext:
# sysctls:
# - name: net.ipv4.tcp_sack
# value: "0"
securityContext:
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
nodeSelector:
#kubernetes.io/hostname: ovn-worker2
# test: trozet1
# nodeSelector:
kubernetes.io/hostname: worker-2.ostest.test.metalkube.org
6. exec into client2, yum install libndp
7. also exec into ovnkube-node pod on worker0, and begin tcpdumping for icmp6 traffic
8. in client2, run an unsolicited advertisement in a loop: while true; do ndptool -t na -U -i br-ex -T fd2e:6f44:5dd8:c956::18 send; done
9. verify the packet is making it to worker0, try to debug why it is being dropped by OVN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment