As Kubernetes celebrated its 10th birthday bash on the 6th of June, I decided to write a post about 10 interview questions that I would ask a potential employee to gauge their skills on Kubernetes and its eco-system.

1. Why do need Gateway API when there is Ingress?

Ingress is used to manage external traffic into the cluster. Rather than creating induvidual LBs for each service, Ingress allows us to use one load balancer to serve traffic to all the services. That being said, there are many drawbacks in Ingress implementation. Ingress only supports L7 HTTP traffic. It does not support other L7 protocols or L4 protocols like TCP and UDP. This opens room for many problems. Moreover there is no support for advanced features like rate-limiting, A/B testing etc. This makes customers use vendor-specific annotations which leads to vendor lock-in. Ingress does not also support Header based routing. All these drawbacks made Ingress inferior product. Hence Gateway API is introduced

Gateway API is modern, extensible traffic management solution in K8s which support both L4 and L7 protocols. GatewayAPI completely embraces the extensibility of K8s by introducing new Custom Resources like Gateway , GatewayClass and HTTPRoute. This establishes a clear set of boundary for each CR making them replacable. It supports all advanced features like request mirroring, fine grained traffic metrics etc. GatewayAPI also sets standard set of definition files for API objects so one vendor can be replcaed easily with other vendors. This is a pain point in the Ingress' case.

2. How do you manage secrets in K8s?

Secrets are an interesting aspect of Kubernetes. The inbuilt Kubernetes secrets are not encrypted at rest but just encoded. So using inbuilt secrets is a huge security vulnerability. If your etcd cluster data is compromised then all your secrets can be accessed in a plain text format. One way to prevent this is to encrypt ectd data at rest.

One can use cloud-specific secret stores to manage secrets and use cloud-native tools like External Secrets Operator, secret store CSI driver etc. There are also alternatives like Hashicorp Vault along with vault sidecar container that directly inserts secrets into pod by skipping K8s secrets as a whole. This is the most secure way of managing secrets.

3. How do you debug applications in K8s?

Debugging apps in K8s environment is a challenging task as the pods will not/should not have debugging tools like Curl or nc etc(rightfully so for security purposes). This can be addressed by using ephemeral containers like busybox using kubectl debug command. This helps with troubleshooting pods, services etc and understanding where the issue lies

kubectl debug -it ephemeral-demo --image=busybox:1.28 --target=ephemeral-demo

Apart from that you can analyze K8s events, use kubectl describe to get detailed understanding of state of application. You can analyize logs using kubectl logs or port-forward for further debugging.

4. What are Custom Resources?

While using K8s features, you can also extend them by developing new CRDs as per requirement. You can create new CRDs and Custom Controllers to manage them using frameworks like OperatorSDK, Kubebuilder etc. This feature makes K8s extensible and programmable.

5. When do you not deploy in Kubernetes?

Though K8s tries to appeal to wide variety of workloads there might be use-cases where K8s is not ideal solution for your problem like

1. If you have small number of identical applications.

2. If your applications are not optimized for the cloud.

3. If your app is still monolithic.

4. If you cannot keep-up with the k8s releases and migrations.

5. You don't need to scale or deploy new applications.

6. How do you scale K8s resources?

K8s is built by keeping scalability in mind. Ideally, K8s can scale up to 5000 nodes or 150,000 pods. There are lot of built-in features to scale pods horizontally and vertically like HPA and VPA. You can set the CPU and memory threshold and once these numbers are hit then HPA will scale the number of pods. There is KEDA(Kubernetes-based Event Driven Autoscaler) using which we can scale applications based on custom metrics.

When it comes to scaling a K8s cluster, there are tools like Cluster Autoscaler which can add nodes if there are un-schedulable pods but Cluster Autoscaler treats all the nodes same. That is where Karpenter comes into picture. You can setup nodePools for different application types or for different regions etc. Karpenter is a best solution out there for scaling cluster.

7. How do you handle logs in K8s?

For small clusters and where you do not need to retain logs for any purposes, kubectl logs command is just fine. But if you need to retain, aggregate and analyze logs then you have to look at logging solutions like Splunk, Logz etc. These are paid services. Apart from these you can use stacks like EFK, ELK and Promtail, Grafana and Loki where one tool collects the logs and forwards them(Fluentd, Logstash and Promtail) one acts as a database (Elasticsearch, Loki) where as other tool is used for querying, visualization and alerts.

8. Why do you need to use ServiceMesh?

Service mesh can be a great infrastructure add-on to the K8s cluster. Service Meshes like Istio, Cilium and Linkerd can offer lot of features out of box like mTLS, canary deployment, progressive deployment, traffic splitting, A/B testing and circuit breaking etc. Service meshes increase observability, security and reliability of apps in the cluster.
The caveat is that the service mesh comes with lot of complexity and architectural overhead to the cluster. If your workloads are simple and do not need additional latency(though minimal) or complexity, you can stay away from them.

9. How do you ensure security of K8s cluster?

Though security is a vast topic, I can try to be brief. Firstly, by using network policies, apps can restrict access to other namespaces. Using RBAC to make sure only users/service accounts with proper permissions are accessing the resources. Using Policy-as-a-Code tools like OPA or Kyverno to set guardrails around apps getting deployed. mTLS for app to app communication. Setting threat monitoring tools like Falco and PSA for early detection and mitigation. Setting up seperate clusters for apps handling sensitive data to make them PCI-DSS, HIPPA and GDPR compatible.

If I am using managed K8s like EKS or GKE, using cloud security features like KMS, Golden AMIs etc.

10. How do you handle Stateful applications in K8s?

Traditionally Kubernetes has a bad reputation for stateful applications like PostgreSQL because popular belief is that K8s is only meant for stateless workloads. That is not true at all. StatefulSets are used to maintain the identity of pods across restarts and redeployments. However, I would not use storage in K8s nodes to create StorageClass or PVs because of the ephemeral nature of nodes, Using something like EBS volume to create SC using CSI driver and creating PVC from it is recommended.

Alternatively, there are many OS solutions like Cloud Native PG and StackGres which offer better approach to the problem. Running databases on Kubernetes is still an evolving topic which can see many changes in coming days.