Creating core file for analysis with OCP and golang

gistfile1.txt

0. Use ovn-k master leader pod for the following process

1. Install dependencies (gdb)
In order to be able to generate a core file of the ovnkube-master process, you will need the gcore binary from the gdb package. The easiest way to get this package is by spawning a toolbox on the node in question and by installing gdb:
# toolbox
# yum install gdb -y
One may have to change the toolbox image (https://docs.openshift.com/container-platform/4.10/support/gathering-cluster-data.html?extIdCarryOver=true&sc_cid=701f2000001OH7JAAW#starting-an-alternative-image-with-toolbox_gathering-cluster-data)

2. Generating a gcore of the current running ovnkube-master process

# pid=$(ps ax | grep '/usr/bin/ovnkube' | grep '\--init-master' | awk '{print $1}')
Make sure that the process ID is indeed of the correct process:
[root@openshift-master-0 /]# ps u $pid
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       12338  0.2  0.2 1869120 76388 ?       Ssl  Feb22  16:55 /usr/bin/ovnkube --init-master openshift-master-0 (...)

Generate a core file of the process on the /host file system:
# gcore -o /host/var/tmp/ovnkube-master $pid

This will confirm the generation of a relatively large file (likely several GB):
# Saved corefile /host/var/tmp/ovnkube-master.12338

Hence, gzip the file for transfer and take the md5sum of the file:
# gzip /host/var/tmp/ovnkube-master.$pid
# md5sum /host/var/tmp/ovnkube-master.$pid.gz

Now, transfer the file to your local system, e.g. with oc debug/node <node name>.
After transfer, take the md5sum of the new file and make sure that the sums are identical:
$ oc debug node/openshift-master-0 -- cat /host/var/tmp/ovnkube-master.12338.gz > ovnkube-master.12338.gz
$ md5sum ovnkube-master.12338.gz

3. Obtaining the current ovnkube-master binary for core file analysis
Copy the current ovnkube binary from one of the master pods to your local system. This file is needed during the later analysis of the core file:
$ oc cp -n openshift-ovn-kubernetes ${master_pod}:/usr/bin/ovnkube ovnkube