Skip to content

Instantly share code, notes, and snippets.

@trozet

trozet/gist:025d8afe714ef3c724d063bcfe1b4ac6

Last active January 11, 2024 19:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save trozet/025d8afe714ef3c724d063bcfe1b4ac6 to your computer and use it in GitHub Desktop.
Save trozet/025d8afe714ef3c724d063bcfe1b4ac6 to your computer and use it in GitHub Desktop.
Download ZIP
kube proxy session affinity
Raw
gistfile1.txt
#### setup, client curling a service with session affinity that is backed by server and server-sdn pods
[trozet@fedora test]$ oc get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 172.30.0.1 <none> 443/TCP 46m
my-service1 ClusterIP 172.30.189.139 <none> 1337/UDP,80/TCP 5m5s
openshift ExternalName <none> kubernetes.default.svc.cluster.local <none> 41m
[trozet@fedora test]$ oc get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
client 1/1 Running 0 16m 10.129.2.12 ip-10-0-82-66.ec2.internal <none> <none>
ip-10-0-82-66ec2internal-debug 1/1 Running 0 14m 10.0.82.66 ip-10-0-82-66.ec2.internal <none> <none>
server 1/1 Running 0 17m 10.131.0.19 ip-10-0-20-253.ec2.internal <none> <none>
server-sdn 1/1 Running 0 6m57s 10.129.2.13 ip-10-0-82-66.ec2.internal <none>
#### initial curls stick to server-sdn
/proc/net/xt_recent/KUBE-SEP-OZ2MS2W3WHYLO3FE
sh-5.1# cat /proc/net/xt_recent/KUBE-SEP-OZ2MS2W3WHYLO3FE
src=10.129.2.12 ttl: 64 last_seen: 4297597633 oldest_pkt: 6 4297433512, 4297467220, 4297596311, 4297596756, 4297597207, 4297597633
sh-5.1# iptables -L -t nat -n -v |grep my-service1
0 0 KUBE-MARK-MASQ all -- * * 10.129.2.13 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-2Y5D3ZEW2UO45WBZ side: source mask: 255.255.255.255 udp to:10.129.2.13:1337
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah2 */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255 tcp to:10.131.0.19:80
0 0 KUBE-MARK-MASQ all -- * * 10.129.2.13 0.0.0.0/0 /* default/my-service1:blah2 */
6 360 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-OZ2MS2W3WHYLO3FE side: source mask: 255.255.255.255 tcp to:10.129.2.13:80
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255 udp to:10.131.0.19:1337
0 0 KUBE-SVC-IDCYZEXVTJSZKTPT udp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
6 360 KUBE-SVC-G2ZQY3LF2A3MNAZ6 tcp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-MARK-MASQ tcp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
5 300 KUBE-SEP-OZ2MS2W3WHYLO3FE all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.129.2.13:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-OZ2MS2W3WHYLO3FE side: source mask: 255.255.255.255
0 0 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255
1 60 KUBE-SEP-OZ2MS2W3WHYLO3FE all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.129.2.13:80 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */
0 0 KUBE-MARK-MASQ udp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
0 0 KUBE-SEP-2Y5D3ZEW2UO45WBZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.129.2.13:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-2Y5D3ZEW2UO45WBZ side: source mask: 255.255.255.255
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255
0 0 KUBE-SEP-2Y5D3ZEW2UO45WBZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.129.2.13:1337 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */
<none>
##### delete server-sdn
sh-5.1# iptables -L -t nat -n -v |grep my-service1
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah2 */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255 tcp to:10.131.0.19:80
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255 udp to:10.131.0.19:1337
0 0 KUBE-SVC-IDCYZEXVTJSZKTPT udp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
0 0 KUBE-SVC-G2ZQY3LF2A3MNAZ6 tcp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-MARK-MASQ tcp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255
0 0 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */
0 0 KUBE-MARK-MASQ udp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */
#### initiate client curl again
sh-5.1# iptables -L -t nat -n -v |grep my-service1
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah2 */
4 240 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255 tcp to:10.131.0.19:80
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255 udp to:10.131.0.19:1337
0 0 KUBE-SVC-IDCYZEXVTJSZKTPT udp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
4 240 KUBE-SVC-G2ZQY3LF2A3MNAZ6 tcp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-MARK-MASQ tcp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
3 180 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255
1 60 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */
0 0 KUBE-MARK-MASQ udp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */
#### now hits other server, add server-sdn back
sh-5.1# iptables -L -t nat -n -v |grep my-service1
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah2 */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255 tcp to:10.131.0.19:80
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255 udp to:10.131.0.19:1337
0 0 KUBE-MARK-MASQ all -- * * 10.129.2.14 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-SU5DTMGP2CMFCSFK side: source mask: 255.255.255.255 udp to:10.129.2.14:1337
0 0 KUBE-MARK-MASQ all -- * * 10.129.2.14 0.0.0.0/0 /* default/my-service1:blah2 */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-TUIWK7LV6Y27FYQZ side: source mask: 255.255.255.255 tcp to:10.129.2.14:80
0 0 KUBE-SVC-IDCYZEXVTJSZKTPT udp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
0 0 KUBE-SVC-G2ZQY3LF2A3MNAZ6 tcp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-MARK-MASQ tcp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-SEP-TUIWK7LV6Y27FYQZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.129.2.14:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-TUIWK7LV6Y27FYQZ side: source mask: 255.255.255.255
0 0 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255
0 0 KUBE-SEP-TUIWK7LV6Y27FYQZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.129.2.14:80 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */
0 0 KUBE-MARK-MASQ udp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
0 0 KUBE-SEP-SU5DTMGP2CMFCSFK all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.129.2.14:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-SU5DTMGP2CMFCSFK side: source mask: 255.255.255.255
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255
0 0 KUBE-SEP-SU5DTMGP2CMFCSFK all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.129.2.14:1337 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */
sh-5.1# cat /proc/net/xt_recent/KUBE-SEP-4OO6XJTATFURZMQT
src=10.129.2.12 ttl: 64 last_seen: 4297811267 oldest_pkt: 4 4297796545, 4297810383, 4297810847, 4297811267
#### curl again, client still uses other server
sh-5.1# iptables -L -t nat -n -v |grep my-service1
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah2 */
4 240 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255 tcp to:10.131.0.19:80
0 0 KUBE-MARK-MASQ all -- * * 10.131.0.19 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255 udp to:10.131.0.19:1337
0 0 KUBE-MARK-MASQ all -- * * 10.129.2.14 0.0.0.0/0 /* default/my-service1:blah */
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah */ recent: SET name: KUBE-SEP-SU5DTMGP2CMFCSFK side: source mask: 255.255.255.255 udp to:10.129.2.14:1337
0 0 KUBE-MARK-MASQ all -- * * 10.129.2.14 0.0.0.0/0 /* default/my-service1:blah2 */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 */ recent: SET name: KUBE-SEP-TUIWK7LV6Y27FYQZ side: source mask: 255.255.255.255 tcp to:10.129.2.14:80
0 0 KUBE-SVC-IDCYZEXVTJSZKTPT udp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
4 240 KUBE-SVC-G2ZQY3LF2A3MNAZ6 tcp -- * * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-MARK-MASQ tcp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah2 cluster IP */ tcp dpt:80
0 0 KUBE-SEP-TUIWK7LV6Y27FYQZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.129.2.14:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-TUIWK7LV6Y27FYQZ side: source mask: 255.255.255.255
4 240 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-4OO6XJTATFURZMQT side: source mask: 255.255.255.255
0 0 KUBE-SEP-TUIWK7LV6Y27FYQZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.129.2.14:80 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-4OO6XJTATFURZMQT all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah2 -> 10.131.0.19:80 */
0 0 KUBE-MARK-MASQ udp -- !tun0 * 0.0.0.0/0 172.30.189.139 /* default/my-service1:blah cluster IP */ udp dpt:1337
0 0 KUBE-SEP-SU5DTMGP2CMFCSFK all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.129.2.14:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-SU5DTMGP2CMFCSFK side: source mask: 255.255.255.255
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-ROU3D7MTIRX6MEVO side: source mask: 255.255.255.255
0 0 KUBE-SEP-SU5DTMGP2CMFCSFK all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.129.2.14:1337 */ statistic mode random probability 0.50000000000
0 0 KUBE-SEP-ROU3D7MTIRX6MEVO all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/my-service1:blah -> 10.131.0.19:1337 */
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment