Simon Krenger

One of the newer features in Kubernetes (1.30 and later) is the Kubelet Checkpoint API. This new API allows users to create a stateful copy of a running container, a functionality which is often used for forensics or for debugging.

In Kubernetes installations where this feature is enabled, a checkpoint can be created by accessing the respective Kubelet API via curl or similar. In the following example I am also using the Kubernetes API /proxy endpoint (the same can also be done on the Node locally via localhost:10250/checkpoint/...):

$ curl -k -X POST --header "Authorization: Bearer $TOKEN" "$KUBERNETES_API_URL/api/v1/nodes/$NODE_NAME/proxy/checkpoint/$NAMESPACE_NAME/$POD_NAME/$CONTAINER_NAME"
{"items":["/var/lib/kubelet/checkpoints/checkpoint-fedora-74d79dd7f4-csrmg_skrenger-container-2024-12-12T12:56:19Z.tar"]}

I like to use the Prometheus node_exporter to get metrics about my hardware. However some hardware (such as my X300M-STX mainboard) exposes sensors with some rather nonsensical values:

[..]
node_hwmon_temp_celsius{chip="platform_nct6775_656",sensor="temp13"} 49.75
node_hwmon_temp_celsius{chip="platform_nct6775_656",sensor="temp15"} 3.892313987e+06
node_hwmon_temp_celsius{chip="platform_nct6775_656",sensor="temp16"} 3.892313987e+06
[..]

To ignore such values, node_exporter only allowed the exclusion of complete chips / devices using --collector.hwmon.chip-exclude. However, in newer versions of node_exporter you’ll be able to exclude (or explicitly include) single sensors on a sensor-level using the following command line option:

--collector.hwmon.sensor-exclude="platform_nct6775_656;temp1[5,6]"

The argument is a regex that is matched against the device name and the sensor. Separate the chip name and the sensor name using “;“.

As my little home server I have an Asrock DeskMini X300 with an AMD Ryzen 7 5700G (16 cores) and 64GB of memory. A nice low powered home server to play around with. Out of the box, the DeskMini comes with one 1 Gbit network interface (a Realtek chipset). Since most of my devices are connected via WiFi anyway, this was more than enough until now. But then, modernity arrived in my part of the world and we now have 10Gbit fiber internet, great!

10Gbit internet sounds awesome, however devices connected via WiFi will only ever see a real-world maximum of around 700 Mbits/sec via WiFi 6. But maybe my little DeskMini could use all that 10Gbit? Unfortunately, the DeskMini motherboard does not have any of the usual PCIe expansion slots apart from SATA and M.2 slots. So I decided to try the “IOCREST M.2 to Single 10G Ethernet Network Adapter (IO-M2F107-GLAN)” adapter (AliExpress link here), to see if that would work.

NVIDIA provides the Linux4Tegra (L4T) distribution as an image for use with the NVIDIA Jetson Nano. However, once you upgrade the whole system, strange problems will pop up, one of which I have described here: NVIDIA Docker “permission denied: unknown.” on Jetson Nano.

When applying a popular solution described here by adding a new repository to your L4T installation, this will result in interesting error messages such as the following when trying to run L4T-ML containers:

docker run  --rm --runtime nvidia -it nvcr.io/nvidia/l4t-ml:r32.7.1-py3 python3 -c "import torch"

[..]
libcurand.so.10: cannot open shared object file: No such file or directory

I recently bought an NVIDIA Jetson Nano Developer Kit to fiddle around with things like MicroShift or TensorFlow. The board is typically used with L4T (Linux for Tegra) based on Ubuntu 18.04. Fedora can also be installed, although not all drivers (for example for the GPU) are available yet. So after properly updating the system with the latest packages, when starting a container using the nvidia runtime, I got the following error:

docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-ml:r32.6.1-py3
[..]
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.

When working with JSON data, I typically use jq to mangle the data. I keep this post as a reference for myself on how to remove an element from a JSON list or array using jq.

Given we have the following array:

$ echo '{"hello": "world", "myarray": ["a", "b", "c"]}' | jq
{
  "hello": "world",
  "myarray": [
    "a",
    "b",
    "c"
  ]
}

To remove an element from the array, use the del function with the select function to select a single element:

jq 'del(.myarray[] | select(. == "b"))'

So when applying this to the above array, we can remove “b” from the array like so:

$ echo '{"hello": "world", "myarray": ["a", "b", "c"]}' | jq 'del(.myarray[] | select(. == "b"))'
{
  "hello": "world",
  "myarray": [
    "a",
    "c"
  ]
}

Due to COVID-19, like many others I am currently working from home and as a result I took the chance to update my home office. Working with a small laptop screen for months is not optimal, so I went the ultra-wide route and got myself a Dell U3818DW monitor.

Since I did not find a lot of information about running this monitor with Linux, here is a quick overview. To summarize, everything works out-of-the-box.

For editing YAML, be it for OpenShift / Kubernetes or Ansible, having your editor set up right can help to avoid common mistakes. So here is the minimalistic config in my ~/.vimrc to make working with YAML files easier. I am sure there are even more plugins or settings available, but this minimal set of commands works fine for me:

set ts=2
set sts=2
set sw=2
set expandtab

syntax on
filetype indent plugin on

set ruler

For a few years now I have been using the pass password manager. It is a wonderfully simple way to manage passwords using PGP to encrypt passwords in text files. The same files can then be placed in a git repository, which makes replicating passwords easy.

For different reasons I am now migrating to gopass, a Go implementation of pass with a few additional features. I am using Homebrew to install gopass on my machine: brew install gopass. Theoretically, gopass should work out-of-the-box and is compatible with the old pass utility. So I was quite surprised to see an error message like this:

$ gopass github
Entry 'github' not found. Starting search...
Found exact match in 'github.com/simonkrenger'
gpg: decryption failed: No secret key

Error: failed to retrieve secret 'github.com/simonkrenger': Failed to decrypt

Strange. But decrypting the password file directly using PGP works fine:

$ gpg -d ~/.password-store/github.com/simonkrenger.gpg
[..]

If the above command using gpg does not work, check your keys using gpg --list-keys and gpg --list-secret-keys. Especially when migrating to GPG2, sometimes keys do not get imported into the new keyrings. In case you need to import the old keyring into the new format like so:

$ gpg --import ~/.gnupg/pubring.gpg
$ gpg --import ~/.gnupg/secring.gpg

But even after importing the keys, I still received gpg: decryption failed: No secret key. So after searching around I found that I need to set the GPG_TTY variable:

$ export GPG_TTY=$(tty)

It seems that not setting the GPG_TTY environment variable leads to the error above. Which is quite misleading. After setting this environment variable (and adding it to the .bash_profile), gopass works as expected.

If you have worked with remote Linux servers before, I am guessing you already encountered machines that just don’t want to reboot. This is typically due screwed-up network mounts or stuck processes, so the server will hang during shutdown. But it turns out that there are other ways to reboot a server.

One of these is the “Magic SysRq key“. To reboot a server using the SysRq trigger in the kernel, use the following two commands. First, enable the trigger:

echo 1 > /proc/sys/kernel/sysrq

Then, reboot the server the magic way by typing

echo b > /proc/sysrq-trigger

Note that this will reboot the server without unmounting or syncing the filesystems! There are also other options available via the SysRq trigger, some of them are listed in the Wikipedia article above.