How kubectl uses Kubernetes API

• August 19, 2019
• tuxotron
• Kubernetes

• Facebook
• Twitter
• Reddit
• LinkedIn

As you probably already know, any type of query or command that you run against Kubernetes, it is done by sending an API request to a component called API server. This component lives in the master node/s.

The most common way to interact with a Kubernetes cluster, although you have several graphical options, it is through a command line tool called kubectl.

This tool provides a quite extended number of options, but in this entry I’m going to focus on verbosity, which is a very handy option if we want to learn more how kubectl interacts with the component mentioned previously: the API server.

Any command we run though kubectl, we can ask to obtain a more verbose output by adding the -v or –v option to it. This option also gets the level of verbosity we would like to get out of our command as a parameter. Such level is specified by a number between 0 and 9 inclusive, and each level provides a certain degree of verbosity as you can see in the following image:

For instance, if we run the following command:

kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7bb7cd8db5-8z6t8   1/1     Running   0          33s

We get the pods running in the default workspace.

Now, if we add to the previous command the option -v=5:

kubectl get pods -v=5
I0819 17:02:54.174578   30833 get.go:564] no kind "Table" is registered for version "meta.k8s.io/v1beta1" in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:30"
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7bb7cd8db5-8z6t8   1/1     Running   0          26s

Besides the same result we obtained previously, we can also see some extra information. Levels between 0 and 5 will give you back some extra information about what’s going on while kubectl runs. This could be very helpful for debugging purposes. But in this post, I want to focus on levels 6 to 9, in which kubectl will also provide information about the resources called (API) and the information sent and received (headers and body) from those calls.

Let’s run again the previous command, but changing the verbosity level to 6:

kubectl get pods -v=6
...
I0819 17:11:39.565753   30923 round_trippers.go:438] GET https://192.168.99.110:8443/api/v1/namespaces/default/pods?limit=500 200 OK in 12 milliseconds
...

As you can see here, we see now some extra information about the calls made to the API server, in this case a GET request to https://192.168.99.110:8443/api/v1/namespaces/default/pods?limit=500. Here you can also see the limit parameter issued by 'kubectl, so if you run a get pods and only get 500 pods back, now you know where the limitation is coming from ;)

Let’s try now with verbosity level 7:

kubectl get pods -v=7
...
I0819 17:22:29.600084   31029 round_trippers.go:416] GET https://192.168.99.110:8443/api/v1/namespaces/default/pods?limit=500
I0819 17:22:29.600108   31029 round_trippers.go:423] Request Headers:
I0819 17:22:29.600118   31029 round_trippers.go:426]     Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json
I0819 17:22:29.600132   31029 round_trippers.go:426]     User-Agent: kubectl/v1.15.2 (darwin/amd64) kubernetes/f627830
I0819 17:22:29.612086   31029 round_trippers.go:441] Response Status: 200 OK in 11 milliseconds
...

As you can see here, the difference between levels 6 and 7, is that in 6 we only see the resources called, and 7 we also see the HTTP headers of such calls.

In levels 8 and 9, besides the headers, we also get the body content. The difference between these two, 8 and 9, is that in 9 the content is not truncated while in 8 it is. Let’s see an example:

kubectl get pods -v=8
...
I0819 17:22:22.188395   31000 request.go:947] Response Body: {"kind":"Table","apiVersion":"meta.k8s.io/v1beta1","metadata":{"selfLink":"/api/v1/namespaces/default/pods","resourceVersion":"70162"},"columnDefinitions":[{"name":"Name","type":"string","format":"name","description":"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names","priority":0},{"name":"Ready","type":"string","format":"","description":"The aggregate readiness state of this pod for accepting traffic.","priority":0},{"name":"Status","type":"string","format":"","description":"The aggregate status of the containers in this pod.","priority":0},{"name":"Restarts","type":"integer","format":"","description":"The number of times the containers in this pod have been restarted.","priority":0},{"name":"Age","type":"string [truncated 2611 chars]
...

Let’s check out now a different command with a little bit more complexity:

kubectl describe pod nginx-7bb7cd8db5-8z6t8 -v=6
...
I0819 17:26:27.770772   31121 round_trippers.go:438] GET https://192.168.99.110:8443/api/v1/namespaces/default/pods/nginx-7bb7cd8db5-8z6t8 200 OK in 12 milliseconds
I0819 17:26:27.777728   31121 round_trippers.go:438] GET https://192.168.99.110:8443/api/v1/namespaces/default/pods/nginx-7bb7cd8db5-8z6t8 200 OK in 2 milliseconds
I0819 17:26:27.786906   31121 round_trippers.go:438] GET https://192.168.99.110:8443/api/v1/namespaces/default/events?fieldSelector=involvedObject.name%3Dnginx-7bb7cd8db5-8z6t8%2CinvolvedObject.namespace%3Ddefault%2CinvolvedObject.uid%3D9e77227d-cc08-4365-aeab-c0bbbfc1c1d8 200 OK in 2 milliseconds
...

Here you can observe that describe requires more than one call to the API server.

And lastly, let’s create a deployment using the run command:

kubectl run nginx2 --image nginx -v=8
...
I0819 17:29:23.727063   31398 round_trippers.go:416] GET https://192.168.99.110:8443/apis/apps/v1?timeout=32s
I0819 17:29:23.727097   31398 round_trippers.go:423] Request Headers:
...
I0819 17:29:23.749539   31398 request.go:947] Request Body: {"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"nginx2","creationTimestamp":null,"labels":{"run":"nginx2"}},"spec":{"replicas":1,"selector":{"matchLabels":{"run":"nginx2"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"run":"nginx2"}},"spec":{"containers":[{"name":"nginx2","image":"nginx","resources":{}}]}},"strategy":{}},"status":{}}
I0819 17:29:23.749618   31398 round_trippers.go:416] POST https://192.168.99.110:8443/apis/apps/v1/namespaces/default/deployments
I0819 17:29:23.749631   31398 round_trippers.go:423] Request Headers:
I0819 17:29:23.749638   31398 round_trippers.go:426]     Content-Type: application/json
I0819 17:29:23.749645   31398 round_trippers.go:426]     User-Agent: kubectl/v1.15.2 (darwin/amd64) kubernetes/f627830
...

In this case we see that kubectl not only makes a GET request, but also a POST request, and because we are using the verbosity level 8, we can also see the body content of that POST request, as well as the responses back from GET and POST.

This is a very nice way to see how kubectl uses the Kubernetes API behind the scene and also an interactive way to learn more about the this API, besides obviously consulting its documentation.

Happy Hacking!

• kubectl