Simon Krenger

Kubernetes uses etcd as the persistent store for API data. As etcd is a distributed key-value store, we can also use command line tools to query this store. The examples in this post are for OpenShift 3.x.

Apart from just using get, there is also the possibility to perform the following actions on certain keys:

• put to write to a key – unless you know what you are doing, don’t touch the Kubernetes data in etcd, as this will manifest in very strange Kubernetes behaviour.
• del to delete a key – also, this may break your Kubernetes cluster by introducing inconsistencies.
• watch to keep a watch on an object. This is very helpful to track changes on a certain object.

The get action is probably the most helpful functionality for in-depth API debugging directly within etcd.

Dump all keys from etcd

On an OpenShift Container Platform Master, execute the following commands to first set the correct environment variables and then to retrieve all keys with the prefix / using the etcdctl3 get command:

source /etc/etcd/etcd.conf
etcdctl3 --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS --write-out=simple get / --prefix --keys-only

This will result in a list of all the available keys in the store:

/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/bundlebindings.automationbroker.io
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/bundleinstances.automationbroker.io
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/bundles.automationbroker.io
/kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com
[..]
/kubernetes.io/apiservices/v1.
/kubernetes.io/apiservices/v1.apps
/kubernetes.io/apiservices/v1.apps.openshift.io
/kubernetes.io/apiservices/v1.authentication.k8s.io
/kubernetes.io/apiservices/v1.authorization.k8s.io
/kubernetes.io/apiservices/v1.authorization.openshift.io
/kubernetes.io/apiservices/v1.autoscaling
/kubernetes.io/apiservices/v1.batch
[..]
/kubernetes.io/statefulsets/openshift-monitoring/prometheus-k8s
/openshift.io/deploymentconfigs/default/docker-registry
/openshift.io/deploymentconfigs/default/registry-console
/openshift.io/deploymentconfigs/default/router
[..]

Note that depending on the size of your cluster and the amount of objects in your etcd, this might put some load on your etcd cluster. So be careful when dumping all the keys.

Get a key from etcd

When you want to check the content of a certain key, use the get <key> command to retrieve the value of a single key in JSON format (specified by the write-out option):

source /etc/etcd/etcd.conf
etcdctl3 --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS --write-out=json get /openshift.io/deploymentconfigs/default/router

Which will show a JSON representation of the content of that key:

{"header":{"cluster_id":6288023921786686208,"member_id":14443325126483823971,"revision":1693101,"raft_term":49},"kvs":[{"key":"L29wZW5zaGlmdC5pby9kZXBsb3ltZW50Y29uZmlncy9kZWZhdWx0L3JvdXRlcg==","create_revision":2866,"mod_revision":542876,"version":12,"value":"azhzAAooChRhcHBzLm9wZW5[..]"}],"count":1}

The actual Kubernetes object is stored in the Base64-encoded value field of the returned JSON data. You can use base64 -d to decode the value field further. This will then show you the serialised protobuf Kubernetes object, which in turn can be decoded using the protoc command line tool.

Watch a key in etcd

Using the watch command, we can watch for changes. In the following example, watch for changes on the “router” DeploymentConfig in the default namespace:

source /etc/etcd/etcd.conf
etcdctl3 --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS --write-out=json watch /openshift.io/deploymentconfigs/default/router