Simon Krenger

One of the newer features in Kubernetes (1.30 and later) is the Kubelet Checkpoint API. This new API allows users to create a stateful copy of a running container, a functionality which is often used for forensics or for debugging.

In Kubernetes installations where this feature is enabled, a checkpoint can be created by accessing the respective Kubelet API via curl or similar. In the following example I am also using the Kubernetes API /proxy endpoint (the same can also be done on the Node locally via localhost:10250/checkpoint/...):

$ curl -k -X POST --header "Authorization: Bearer $TOKEN" "$KUBERNETES_API_URL/api/v1/nodes/$NODE_NAME/proxy/checkpoint/$NAMESPACE_NAME/$POD_NAME/$CONTAINER_NAME"
{"items":["/var/lib/kubelet/checkpoints/checkpoint-fedora-74d79dd7f4-csrmg_skrenger-container-2024-12-12T12:56:19Z.tar"]}

At their core, containers are just Linux processes that are namespaced. This means in practice, many containers still run as processes on the same host machine. While namespacing processes using cgroups creates very good boundaries between processes, the isolation is still not perfect.

So when working with a lot of different namespaces in Kubernetes and you only know the “oc project” command from OpenShift, you start to miss an easy way to change namespaces in Kubernetes.

The official documentation to switch namespaces proposes something like this:

$ kubectl config set-context $(kubectl config current-context) --namespace=<insert-namespace-name-here>

Not something that I want to type regularly. First I tried to create a BASH alias or something, which did not work. So I looked around for BASH functions. I found that Jon Whitcraft proposed a nice BASH function in a GitHub issue. I lightly modified this and placed this in my own .bashrc file:

function kubectlns() {
  ctx=`kubectl config current-context`
  ns=$1

  # verify that the namespace exists
  ns=`kubectl get namespace $1 --no-headers --output=go-template={{.metadata.name}} 2>/dev/null`
  if [ -z "${ns}" ]; then
    echo "Namespace (${1}) not found, using default"
    ns="default"
  fi

  kubectl config set-context ${ctx} --namespace="${ns}"
}

So to change your namespace, use something like this:

$ kubectlns simon
Context "kubernetes-admin@kubernetes" modified.

Nice and short.

So when using NodeSelectors in OpenShift, you’ll also have to set labels on your nodes. You can find more information on labeling nodes in the OpenShift documentation. Here is how you can add or remove a label from a node or pod:

To add a label to a node or pod:

# oc label node node001.krenger.ch mylabel=myvalue
# oc label pod mypod-34-g0f7k mylabel=myvalue

To remove a label (in the example “mylabel”) from a node or pod:

# oc label node node001.krenger.ch mylabel-
# oc label pod mypod-34-g0f7k mylabel-

You can also use oc label -h to see more options for the oc label command.

So in any larger container orchestrator installation, be it Kubernetes or OpenShift, you will encounter pods that crash regularly and enter the “CrashLoopBackOff” status.

$ oc get pod --all-namespaces 
NAMESPACE         NAME                       READY     STATUS             RESTARTS   AGE
[..]
my-project-1      helloworld-11-9w3ud        1/1       Running            0          7h
my-project-2      myapp-simon-43-7macd       0/1       CrashLoopBackOff   3774       9h

Note the container that has status “CrashLoopBackOff” and 3774 restarts.