AI inference is the "doing" part of artificial intelligence. It's the moment a trained model stops learning and starts working, turning its knowledge into real-world results.

We all use cutting-edge frontier models in our day-to-day use like Gemini, Claude etc. In our personal life we don’t interact with open-source, purpose built models like Qwen, Llama etc that often who have smaller parameters and often fine-tuned for specific tasks. But for enterprises, there is a growing demand to use these type of models internally. Whether it is to serve models fine-tuned on proprietary data, or for agentic purposes like tool-calling at scale, understanding-intent etc. This creates a unique use-case of using cloud-native frameworks like KServe and llm-d to put forward best-practices to serve AI inference

LLM-D

llm-d: a Kubernetes-native high-performance distributed LLM inference framework

llm-d is an open-source inference framework which offers battle-tested configurations to start our model deployment journey. Rather than using vanilla Kubernetes to run vLLM pods in accelerated hardware, llm-d offers better performance metrics by solving problems like KV-Cache offloading, separation of prefill and decode phases(Prefill/Decode Disaggregation), Intelligent inference routing etc. These features helps us in serving LLM models at scale. One example is, by separating prefill and decode phases, we can deploy prefill pods in compute-bound hardware and decode pods in memory-bound hardware. Also by tweaking the PD ratio of pods, you can achieve smaller TTFT(time to first token) and TPOT(time per output token). The llm-d framework have many well-lit paths to solve specific problems depending on the type of QPS(queries per sec) workloads and models.

Architecture

llm-d architecture have some novel components like GIE(GatewayAPI Inference Extension) and EndPointPicker(EPP) which uses filtering and scoring techniques to enable Gateway API to route traffic dynamically to the next best pod.

Here is a detailed infographic on inference scheduling

Testing Inference Scheduling

I have deployed a GKE cluster using Nvidia L4 instance and some general compute for all other pods. Firstly you need to enable GKE Gateway class in-order for llm-d to create Gateway during helm install. Here is a how-to for that. Once it is done, you need to install monitoring stack to observe the metrics on vLLM pods. llm-d github page have a bash script to deploy kube-prometheus-stack. The I have installed the helmcharts to deploy modelserver(ms) pods in L4 GPU instance.

llm-d-inference-scheduler   gaie-inference-scheduling-epp-5dc59b4767-dv8tx                    1/1     Running   0          51m
llm-d-inference-scheduler   ms-inference-scheduling-llm-d-modelservice-decode-5b7bb867kxmff   0/2     Pending   0          27m
llm-d-inference-scheduler   ms-inference-scheduling-llm-d-modelservice-decode-5b7bb867zxgnm   2/2     Running   0          32m
llm-d-inference-scheduler   ms-inference-scheduling-llm-d-modelservice-decode-77f5d7b89bc7n   0/2     Pending   0          31m

Then I have deployed httproute to connect my InferencePool to the Gateway.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-d-inference-scheduling
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: infra-inference-scheduling-inference-gateway
  rules:
    - backendRefs:
      - group: inference.networking.k8s.io
        kind: InferencePool
        name: gaie-inference-scheduling
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /

Then I could run some load tests on the single vLLM pod that I managed to deploy through the external gateway ip.

Conclusion

All the technologies like llm-d, KServe and GIE are still going through rapid progression. LLM inference poses unique challenges which can be tricky to solve. In the future where there will be massive consumption of tokens by applications, you cannot pay per-token price while it is just an additional feature in your application(imagine booking uber using voice). In those circumstances, frameworks like llm-d is the best alternative.

If you think about it for a minute, once in a lifetime there will be a major invention that profoundly impacts the way we do things. Till late 1990s we all used to shop at physical stores and then internet happened which brought in e-commerce. Now with the AI phenomenon, the way we shop is changing again. Every major innovation brings in little process improvements, give us choice.

We have been provisioning infrastructure since the advent of internet. Firstly, there used to be physical hardware which you procure and set it up manually and run software on it. Then came cloud which brought in IaC tools like Terraform, CF etc. This is an improvement on the existing process. You can quickly setup infrastructure as you need instead of doing clickOps but also have a lot of drawbacks like configuration drift over time, managing statefiles etc. But then Kubernetes happened…

The more and more adoption of Kubernetes led to a plethora of new cloud native tools which offers a little process improvement on the existing tools and applications. Crossplane is one such tool which offers a better way of infra creation and management. It prevents configuration drift and allows us to create abstractions for self service. This helps us shift left so developers can provision the infra they need and manage it as required. Here is a X post contrasting differences between Crossplane and Terraform

https://x.com/ianmiell/status/1788973776996028813

In this article let’s quickly go through the basic terms used in Crossplane. There are 5 main components

Composite Resource(XR) - It can be defined as a bunch of different resources packaged as one resource. If you need IAM Role, Policy and S3 bucket for your application to access that bucket then all three resources can be combined to make one XR.

Composite Resource Definition(XRD) - Composite resource definitions (XRDs) define the schema for a custom API. Users create composite resources (XRs) using the API schema defined by an XRD.

Compositions - Compositions have the actual definition of the things to create that XR is requesting for. It holds all the logic.

Providers - Crossplane providers are extensions that enable Crossplane to manage infrastructure and resources on external services, like cloud providers. Think of them as the bridge that connects your Kubernetes cluster to a third-party API.

Functions - Functions enable you to dynamically populate resources based on the XR. You can use different composition functions to configure what Crossplane does when someone creates or updates a composite resource (XR).

Here is an image from Crossplane docs to better understand how these work

The reason to use Crossplane with ArgoCD in a gitops based approach is that Git manages the commit history so you can quickly refer the changes that are being made to any crossplane resource. ArgoCD can help you visualize the infrastructure you are creating and also allows you to setup sync waves etc. There are lot more benefits than what is stated here but you can catch my drift.

Crossplane in action

Here is the github repo used in this blog

Let’s take a scenario where a dev team needs new EKS cluster for some testing. Usually these requests go through a Jira for DevOps team who create a cluster based on their specification and gives it to them. This comes with a lot of friction because dev team do not want to ask ops team to shutdown the cluster for weekend because they have to raise another jira again on monday and have to wait till the cluster is created again. What if we let dev team create the cluster as required but do not want to deal with HCL or low level components like VPC, Subnet etc.

Here are the building blocks of EKS cluster as platform for developers.

Firstly the dev team have to create the required networking to deploy EKS cluster on top of it. So they will push the eksnetworking XR file in the given Github repository

Using this XR, dev team has ability to configure region, CIDR range, number of subnets etc. Yet it is very guardrailed in terms of what parameters are required and how they have to go in etc. This is controlled by Composite Resource Definition(XRD). You can see the required sections and enum below.

Now The composition is where all the required resources will be configured. It uses one or more functions to fetch the values given in XR to create the resources. KCL is popular language of choice but you can use go-template or Python as well. Here is the composition used.

Remember, XRD and Composition must always be owned by ops team who dictates how an XR can be created and what resources will be created as a part of it. Here is a quick snapshot of created resources from ArgoCD

This gives an overall idea about the resources that are being created and the overall health of your XR. Now you have your networking now to create EKS controlplane. It also follows a similar pattern of creating XRD and Composition. Here is the ekscluster XR for reference

apiVersion: srujanpakanati.com/v1alpha1
kind: EKSCluster
metadata:
  name: my-eks-cluster
  namespace: default
spec:
  parameters:
    clusterName: my-eks-cluster
    region: us-east-2
    kubernetesVersion: "1.33"
    accessList:
      - name: developer-role
        roleARN: "arn:aws:iam::xxxxxxxxx:role/eks-dev-role-to-test-crossplane" # Replace with your IAM Role ARN
        policies:
          - "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
          - "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
    addons:
      - name: vpc-cni
        version: "v1.20.1-eksbuild.3"
      - name: coredns
        version: "v1.12.3-eksbuild.1"
      - name: kube-proxy
        version: "v1.33.3-eksbuild.6"
    vpcId: "vpc-07ddbab7b7c6a6fef" # Replace with your VPC ID
    subnetIds:
      - "subnet-0c7420944a320ddff" # Replace with your subnet IDs
      - "subnet-095925a4561365bf5"
  crossplane:
    compositionSelector:
      matchLabels:
        provider: aws
        workload: ekscluster

Here we can see the components getting created in real time in ArgoCD

Once the controlplane gets created, we can go ahead with NodeGroup creation. Here is my nodegroup XR

apiVersion: srujanpakanati.com/v1alpha1
kind: EKSNodeGroup
metadata:
  name: my-nodegroup
  namespace: default
spec:
  parameters:
    clusterName: cluster-my-eks-cluster # This must match the name of your EKSCluster resource
    region: us-east-2
    nodeGroupName: my-managed-nodes
    instanceTypes:
      - "t3.medium"
    scalingConfig:
      minSize: 1
      maxSize: 3
      desiredSize: 2
    subnetIds: 
      - subnet-0c7420944a320ddff
  crossplane:
    compositionSelector:
      matchLabels:
        provider: aws
        workload: eksnodegroup

Here is the same XR in ArgoCD.

If we take step back here a minute and see what we have created, not only created a cluster and nodegroup. We also created a template in which as many number of clusters can be created as required. 0→1 is done now 1→n is easy. Its not just new controlplanes, you can add nodegroups to the existing EKS clusters. This way we can create self-serving platforms for developer teams so that you prevent the infra related bottlenecks.

Conclusion

Crossplane and ArgoCD are quintessential tools in Cloud Native era. Teams iterate in cycles, these tools offer a significant improvement in that cycle pushing developer experience and cost efficiency further. The returns will be multi-fold and the wildest thing is that these tools are open-source. If your workloads runs on Kubernetes then it is time to adopt appropriate tools for the job.

While all our existing workloads work well with Ingress and we don’t want to touch what is working just fine. Everyone have to migrate from Ingress to Gateway API at some point. I always felt that ingress is too rigid and complex with a lot of nesting YAML and bulky annotation section. Gateway API splits the responsibilities of different people in a team and dedicates a YAML component for them like “GatewayClass“, “Gateway“ and “HTTPRoute“ etc.

Why Ingress sucks?

Ingress is an initial solution to the problem where majority did not have a clarity on where the Kubernetes project is going to endup. It became GA in 1.18 which is very early into Kubernetes and also lacked lot of features natively. There are many other drawbacks with Ingress that we can discuss later

GatewayAPI

Gateway API is truly non opinionated and loosely coupled way of routing outside traffic to your services. Developer, Cluster Administrator and Infrastructure Engineer have their own tasks to do and YAMLs to manage. This enables a great support for multi-tenancy and portability from one vendor to other. Here is an example on how Envoy Gateway API can be tested.

First you need to helm install envoy gateway and CRDs

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.5.0 -n envoy-gateway-system --create-namespace

This deploys envoy gateway deployment with following config

➜  ~ k get cm envoy-gateway-config -n envoy-gateway-system -o yaml 
apiVersion: v1
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    extensionApis: {}
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    logging:
      level:
        default: info
    provider:
      kubernetes:
        rateLimitDeployment:
          container:
            image: docker.io/envoyproxy/ratelimit:3e085e5b
          patch:
            type: StrategicMerge
            value:
              spec:
                template:
                  spec:
                    containers:
                    - imagePullPolicy: IfNotPresent
                      name: envoy-ratelimit
        shutdownManager:
          image: docker.io/envoyproxy/gateway:v1.5.0
      type: Kubernetes
kind: ConfigMap

This configuration essentially defines controllerName and images that needs to be used to for purposes of shutdown and rate limiting.

Following this, you need to install quickstart.yaml to get all the necessary components for the Gateway and application. Practically speaking, GatewayClass and Gateway will already be installed as bootstrap before applications get installed. Here is the GatewayClass and Gateway YAML files.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80

Here in the GatewayClass, the controllerName must match with the one we have provided to EnvoyGateway. In that way, you can use different controllers for different GatewayClasses in the same cluster. The gatewayClassName in Gateway must match with the GatewayClass.

Next, installing application related components. After installing the entire quickstart.yaml, it creates below components

➜  ~ kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.5.0/quickstart.yaml -n default

gatewayclass.gateway.networking.k8s.io/eg created
gateway.gateway.networking.k8s.io/eg created
serviceaccount/backend created
service/backend created
deployment.apps/backend created
httproute.gateway.networking.k8s.io/backend created

You can see that the Gateway has spun up new pod(deployment) and service in envoy namespace

➜  ~ k get pods,svc -n envoy-gateway-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/envoy-default-eg-e41e7b31-55bf69f99d-pl6dl   2/2     Running   0          2m37s
pod/envoy-gateway-667545bc7d-dpmmz               1/1     Running   0          45m

NAME                                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                            AGE
service/envoy-default-eg-e41e7b31   LoadBalancer   10.96.222.26    172.18.0.7    80:30823/TCP                                       2m37s
service/envoy-gateway               ClusterIP      10.96.174.248   <none>        18000/TCP,18001/TCP,18002/TCP,19001/TCP,9443/TCP   45m

In the default namespace

➜  ~ k get pods,svc -n default             
NAME                           READY   STATUS    RESTARTS   AGE
pod/backend-869c8646c5-vrrtl   1/1     Running   0          5m2s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/backend      ClusterIP   10.96.220.59   <none>        3000/TCP   5m2s
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP    48m

Here is the httpRoute for this backend app

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

To test the traffic, we can port-forward the gateway service and curl it

➜  ~ curl --verbose --header "Host: www.example.com" http://localhost:8888/get

* Host localhost:8888 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8888...
* Connected to localhost (::1) port 8888
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
Handling connection for 8888
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Wed, 03 Sep 2025 15:34:15 GMT
< content-length: 467
<

So, in order to present a mental map on how the traffic is flowing, here is a diagram

Conclusion

GatewayAPI is maturing with more vendors offering it as a solution. There are good migration guides as well. GatewayAPI is a key step towards easier, superior and de-coupled networking solution.

When it comes to BRICS as an organization, it is never either a political, economic or military organization. There is no common culture, ideology, political or economic systems between member countries. Two out of five are authoritarian regimes with communist ideologies. There are border disputes between India-China and Russia-China. China has grown into economic powerhouse so there is no possibility of creating an alternate economic institutions like common currency or payment systems without accepting Chinese hegemony. All the other countries have an opinion that it is better to deal with a known evil rather than an unknown one. In essence, no one country is interested in putting efforts to make a uniform clay lump, mould it and heat it to make strong BRICS.

BRICS is just a bunch of misfits having tea once a year to discuss their hardships with Western systems.

Here comes Trump, who started out as “using tariffs to re-industrialize USA” to now “Using tariffs to release Bolsonaro and to stop wars“. This is started by Biden when he weaponized SWIFT payment system and seized Russian assets, now continued by Trump. USA, rather being a neutral marketplace, it is choosing sides while undoing years of American progress in creating global financial institutions like Petrodollar and SWIFT payment system. No one trying to de-dollarize world economy except USA itself.

When President Trump said that he would end Russia-Ukraine war, everyone thought that he would do it by negotiations but not by strong arming India to stop buying Russian oil. USA has to make up its mind on who their biggest enemy is. Because just imagine for a second, USA cannot stop China from buying Russian oil. If India stops buying it, it would raise the global crude prices while forcing Russia to sell its oil at a more discounted price to China. China would and can buy more oil than it needs and has a capacity to make additional facilities to store and process crude oil at a breakneck pace. India cannot afford oil in the international market and results in stifled growth and high inflation while its competitor China buys cheap oil at 50 cents on the dollar.

There is no wrath like a Modi scorned.

Due to all these misadventures by Trump, now BRICS has a purpose to fulfill, an agenda to achieve. You cannot keep bullying countries and expect nothing to happen. You cannot keep insulting visiting heads of states without any repercussions. Modi is being forced to shake hands with China at upcoming SCO Summit due to unprovoked provocation by Trump. Remember, India did not even respond to “Elephant-Dragon Tango“ invite by Xi to India’s counterpart Draupadi Murmu on April 1st 2025. There is no other choice left with BRICS nations other than to retaliate against tariffs in some shape or form. Their options can vary from all the BRICS nations having reciprocal tariffs either 50% or half of it 25%. Or can also put tariffs on American service exports to those nations. Trump not only shaped the clay brick but also heated it enough to harden it and handed it over to China only to get hit in the face.

An object at rest will remain at rest, and an object in motion will remain in motion at a constant velocity unless acted upon by a net external force.

This is Newton’s first law of motion which also applies in geo-politics. There is a thing called status-quo which you would want to change if it is not working for you. Unluckily for USA, its President want to have the only thing working for US to be removed. As one starts to breath even before their conscience fully develops, one will only realize that the air is most important thing to live only when one enters into medium which is lack of air. Sometimes people do not actually realize what is working for them until they lack it.

Dynamic Resource Allocation(DRA) is a new feature in Kubernetes to address the pain point of managing assignment of hardware devices like GPUs in a better fashion. This can be understood as higher level generalization of storage allocation pattern in K8s using apis like StorageClass, PV and PVC. Similar to them, you’ll have DeviceClass, ResourceClaimTemplate and ResourceClaim. I have tried this feature with the help of dra-example-driver created by kubernetes-sigs

The Need for DRA

Since the AI boom, there is an increased focus in K8s to address the painpoints of AI/ML workloads. One such problem is k8s ability to allocate GPUs and schedule pods in the node that have GPU, resource sharing between pods etc. This feature is not just limited to GPUs but also for other hardware like network interfaces for which one has to rely on solutions like multus.

The DRA is a step in right direction to address the gap in the synchrony of hardware and pod that uses it.

How it works?

Here is how Kubernetes documentation defines the APIs

ResourceClaim

Describes a request for access to resources in the cluster, for use by workloads. For example, if a workload needs an accelerator device with specific properties, this is how that request is expressed. The status stanza tracks whether this claim has been satisfied and what specific resources have been allocated.

ResourceClaimTemplate

Defines the spec and some metadata for creating ResourceClaims. Created by a user when deploying a workload. The per-Pod ResourceClaims are then created and removed by Kubernetes automatically.

DeviceClass

Contains pre-defined selection criteria for certain devices and configuration for them. DeviceClasses are created by a cluster administrator when installing a resource driver. Each request to allocate a device in a ResourceClaim must reference exactly one DeviceClass.

ResourceSlice

Used by DRA drivers to publish information about resources (typically devices) that are available in the cluster.

DeviceTaintRule

Used by admins or control plane components to add device taints to the devices described in ResourceSlices.

So basically, when the device driver is installed, it also comes with its own DeviceClass(like StorageClass) and also ResourceSlice which manages the list of resources of that DeviceClass that are available. The pods will have reference to either ResourceClaim or ResourceClaimTemplate.

Testing DRA

So I have created kind cluster using latest version 1.33. We need to explicitly enable DRA feature. Here is the kind config

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  DynamicResourceAllocation: true
containerdConfigPatches:
# Enable CDI as described in
# https://tags.cncf.io/container-device-interface#containerd-configuration
- |-
  [plugins."io.containerd.grpc.v1.cri"]
    enable_cdi = true
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          runtime-config: "resource.k8s.io/v1beta1=true"
    scheduler:
        extraArgs:
          v: "1"
    controllerManager:
        extraArgs:
          v: "1"
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        v: "1"
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        v: "1"

Once the cluster is created, we need to install the helmchart dra-example-driver. This creates the daemon-set

➜  kind k get daemonsets.apps -n dra-example-driver 
NAME                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
dra-example-driver-kubeletplugin   1         1         1       1            1           <none>          8h

you can now see the device class by listing k get deviceclasses command and you can also list the resourceslice available for that deviceclass

➜  dra-example-driver git:(main) kubectl get resourceslice -o yaml
apiVersion: v1
items:
- apiVersion: resource.k8s.io/v1beta1
  kind: ResourceSlice
  metadata:
    creationTimestamp: "2025-05-25T20:02:27Z"
    generateName: kind-worker-gpu.example.com-
    generation: 1
    name: kind-worker-gpu.example.com-tkbqm
    ownerReferences:
    - apiVersion: v1
      controller: true
      kind: Node
      name: kind-worker
      uid: 3c2af9d8-c6e1-449d-8869-85bc68131687
    resourceVersion: "1010"
    uid: ffeeda3f-b43b-4be9-bb8f-e6937284907b
  spec:
    devices:
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 6
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-121b8219-b8d6-015c-b2eb-1e320ee07510
        capacity:
          memory:
            value: 80Gi
      name: gpu-6
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 7
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-f270be4e-7cd6-da75-39e7-b707122f9b70
        capacity:
          memory:
            value: 80Gi
      name: gpu-7
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 0
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-4cbf87f3-433e-6717-5588-c33e6886832f
        capacity:
          memory:
            value: 80Gi
      name: gpu-0
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 1
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-58bd415e-dee8-f0a5-ca03-02d000554b1a
        capacity:
          memory:
            value: 80Gi
      name: gpu-1
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 2
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-6ab67185-8eff-3a23-32fd-75bfbe37b488
        capacity:
          memory:
            value: 80Gi
      name: gpu-2
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 3
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-6b77fb80-2d68-809d-4bf1-285e5f47dcc5
        capacity:
          memory:
            value: 80Gi
      name: gpu-3
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 4
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-417d66cd-4546-0786-59a3-ef7eb54c564d
        capacity:
          memory:
            value: 80Gi
      name: gpu-4
    - basic:
        attributes:
          driverVersion:
            version: 1.0.0
          index:
            int: 5
          model:
            string: LATEST-GPU-MODEL
          uuid:
            string: gpu-f0fdf728-dccb-f484-bbf5-33f63a90b820
        capacity:
          memory:
            value: 80Gi
      name: gpu-5
    driver: gpu.example.com
    nodeName: kind-worker
    pool:
      generation: 1
      name: kind-worker
      resourceSliceCount: 1
kind: List
metadata:
  resourceVersion: ""

It lists all the GPUs available in the cluster. Now I have tried running the pods listed in the repo and observed their behaviour
For example gpu-test-1

Two pods, one container each

Each container asking for 1 distinct GPU

Once I installed the pod, here is the list of resourceclaims(RC) and RCTs that I can see

➜  demo git:(main) k get resourceclaim
NAME             STATE                AGE
pod0-gpu-724sp   allocated,reserved   106s
pod1-gpu-kvs27   allocated,reserved   106s
➜  demo git:(main) k get resourceclaimtemplates.resource.k8s.io 
NAME         AGE
single-gpu   2m4s

Here are the logs from one of the pod showing the GPU it was allocated with

➜  demo git:(main) k logs pod0 
declare -x DRA_RESOURCE_DRIVER_NAME="gpu.example.com"
declare -x GPU_DEVICE_6="gpu-6"
declare -x GPU_DEVICE_6_RESOURCE_CLAIM="0b66bdc4-7112-4eb7-a371-6df9e0a08167"
declare -x GPU_DEVICE_6_SHARING_STRATEGY="TimeSlicing"
declare -x GPU_DEVICE_6_TIMESLICE_INTERVAL="Default"
declare -x HOME="/root"
declare -x HOSTNAME="pod0"
declare -x KUBERNETES_NODE_NAME="kind-worker"
declare -x KUBERNETES_PORT="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP_ADDR="10.96.0.1"
declare -x KUBERNETES_PORT_443_TCP_PORT="443"
declare -x KUBERNETES_PORT_443_TCP_PROTO="tcp"
declare -x KUBERNETES_SERVICE_HOST="10.96.0.1"
declare -x KUBERNETES_SERVICE_PORT="443"
declare -x KUBERNETES_SERVICE_PORT_HTTPS="443"
declare -x OLDPWD
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
declare -x PWD="/"
declare -x SHLVL="1"

Similarly I also tried other definition files as well and everything worked as expected.

Conclusion

Though this is just an example testing, I believe that DRA will work in real-world just as expected. Though DRA is still in its initial stages, a lot of new features can be added and all cloud providers will come up with new device classes to make them available for pods.

Disclaimer: I am an individual working in tech sector. I never academically pursued immigration or economics. Data maybe incorrect, arguments maybe flawed but intent is genuine. AI maybe used to read the source material to write this post but no part of this blog is AI written(As one can realize from the poor grammar😜).

Free Privileged Speech

Yes, we all can agree that ability to immigrate to a country is a privilege and not a right. US have always been a diverse country where immigrants can thrive with hard work and perseverance to become a cornerstone of the nation. Be it Italians, Irish, Chinese, Arabs and Indians. But I always thought free speech is a right for everyone who is legally present. As Emma Lazarus’ iconic poem engraved on the Statue of Liberty goes

Not like the brazen giant of Greek fame,
With conquering limbs astride from land to land;
Here at our sea-washed, sunset gates shall stand
A mighty woman with a torch, whose flame
Is the imprisoned lightning, and her name
Mother of Exiles. From her beacon-hand
Glows world-wide welcome; her mild eyes command
The air-bridged harbor that twin cities frame.
“Keep, ancient lands, your storied pomp!” cries she
With silent lips. “Give me your tired, your poor,
Your huddled masses yearning to breathe free,
The wretched refuse of your teeming shore.
Send these, the homeless, tempest-tost to me,
I lift my lamp beside the golden door!”

Maybe this has to be updated again to “yearning to breathe free as long as you do not criticize government with that freedom“. Ironically Emma is a Jewish American. With the current acts of government, If a privilege can be revoked for exercising a right, Isn’t right a privilege now?

The L Day

Coming on to a lighter topic, I have always enjoyed President Trump’s press meets. He is candid, fun and whimsical. I always believed he have best interests at heart but his approaches are far from ideal. On April 2nd he have liberated Americans from the shackles for free trade by imposing reciprocal sanctions on all the countries including uninhabited islands like Heard and McDonald Islands. Which questions the amount of thought went into the matter at hand given the stakes. I have always believed that tariffs are important tool in principle to build and strengthen local industries which pays exponentially by keeping money inside the economy. But this sort of implementation begs the question, Will this work?

I would prefer to buy an American made hat at $50 which may earn $5 in net income to an American worker than $25 cap which is made in China and which may earn $0.5 to a Chinese worker because..

• Money earned by an American will become a income to at least 6 other people. Imagine Company → worker → restaurant employee → gas station employee → coffeeshop employee → etc

• Chinese government takes major chunk of the sale which in turn uses it for global dominance, spyware masqueraded as weather balloons or spy vessels masquerading as research ships etc.

• CCP promises it’s people some sort of global economic dominance as it used to be before the “Century of Humiliation“. This forces CCP to save face in front of its people by doing things like “Belt and Road“ and “EV supply chain dominance“ etc.

• My concern is that the lack of democracy will take country in a wrong path for a prolonged periods of time and when people starts becoming wary of its government, they can try to keep false sense of nationalism intact by going into war with other nations. Imagine “Battle of Falklands“, “Kuwait war“ etc. CCP have quite few targets like Taiwan, India, Japan, Philippines and Russia.

I have always thought that American government has left its workers high and dry in the pursuit of cheaper products and using its consumers as a bargaining chip in the geo-political global dominance game. This has led to awful demises of many industries like auto, manufacturing etc. rendering majority of American middle class to rely on service sector. Instead of competing in production at global markets, USA has gave up on its pursuit of raising a generation of highly skilled workforce, automation and innovation. It’s a sad reality where the nation that pioneered assembly line have given up on its industry for cheaper alternatives.

An Indian Perspective

I have lived in india for 25 years which have very high tariffs on electronics and luxury goods, sometimes as much as 100%. I can only justify tariffs in India as a way of raising income and putting a smaller orifice for the flow of Chinese products with whom India is at odds since Indo-China war of 1962. There is a popular way of categorizing Indian consumers into 3 groups. Europe, Brazil and Africa. Europe category is small maybe 5cr who are not much worried about tariffs since the purchases are discretionary in nature and they buy the luxury goods anyway for their comfort and to garner respect. African section of Indian economy comprises of around 100cr which have nothing to do with the products of foreign origin. The Brazilian section is the middle class who are stuck between rock and hard place. They wants to(sometimes have to) buy electronics or luxury goods will suffer due to these high tariffs and often try to emigrate to western nations for better quality of life and higher purchase parity.

None of these tariffs have moved any production to India till recent days. Due to raise in middle class population accessibility to cheaper skilled labor and with government initiatives like “Make in India“ and “PLI“, there is a growth in production. Iphones have become cheaper in India when compared to USA.

Problems in current implementation

On a bigger picture, USA does not want to be global policeman anymore and it is trading soft power for prosperity which leaves a void and leads to an emergence of new global order. As many problems there are with current global order, The USA have played a key role in relatively peaceful world since WW2 and I am personally grateful for that. The genZ are struggling to make ends meet and wants to make decent living. USA’s changing priorities can help a lot with that.

There are many problems with current tariff implementation like…

• Trade deficit should not be a prime proponent to tariffs.Things like Italian cheese, Scotch and Champagne will always have better quality at cheaper price.

• There are no long term plans for tariffs. Are they going to be here after Trump? Can companies gamble on moving production back to USA? Is declining population and struggling generation a concern?

• Tariffs are not industry targeted. For example you can try to move auto and semiconductor industries back by piggybacking on CHIPS Act investments.

• Skilled labor is always a concern. The current higher education system is far from ideal to address this problem. Are there any program linked incentives for universities?

An Ideal way

The best way in my opinion to reduce trade deficit and make US an industrial powerhouse again is by categorizing deficit in 3 groups

Government should not treat tariffs as a source of income(please do not get addicted) and should pump the money raised as different forms of incentives to their respective industries. Imagine tax rebates, incentivizing universities for industry specific programs and income tax discounts for highly skilled workers participating in inter-state migration.

More tariffs collected for an industry → More money gets pumped as incentives → Faster progression to target

Conclusion

I am and always grateful to this country. The people are patriotic, hardworking, warm and always treated me with humility. I hate to see them struggle where their government wastes money on pointless pursuits and lack of clearer vision. You simply cannot perform surgery with an axe. It takes dexterity, perseverance and most of all… time. As the old Indian saying goes, I would like to quote it in the spirit of globalization.

वसुधैव कुटुम्बम् ~ World is one big family

A happy accident

I started watching a Youtube video from sentdex about his experience using openhands and then I was intrigued by it. As I followed Devin saga and their ridiculous $500/month price I was excited to try openhands as it solves the same use-case of using a coding agent to accelerate your dev work. I tried it and it is delightfully good at doing what you are asking it to do. If you can articulate clearly(as the case with every LLM) what needs to be done. It can do it for you.

Openhands in action

As you can see from the video. It can get a lot of things done. We have a choice to use whichever LLM we want to solve the tasks. I am using gemini2.0-flash-exp because it is free and I will not get rate limited. The ideal candidate would be claude-sonnet 3.5. The agent would be as good as the LLM we are using. You can create a github bot and provide token here and you can ask it to raise PRs so that you can review them later. Openhands would be an excellent sidekick for every dev out there. Me being a Devops engineer would do anything to alleviate their pain points(poor devs😆) by equipping them with the best possible tooling with least additional costs.

Reading tea leaves

Okay, my hunch is that in the future, opensource models will be as good as closed source ones like R1 over o1. And every cloud provider out there will charge more for inference by hosting these models(both open and close source) than compared to self-hosting. Right now every company is experimenting with these models as base layer and creating applications on top of them and cloud providers are providing foundational models for dirt cheap. Once the applications reach the scale and cloud providers raise the prices, that’s when companies realize that self-hosting models would be a better option and ends up doing significant code changes and rewiring. If companies explore options to self host and use from the get-go. They can save tons in coming years.

We are in the subscription era both as consumers and as businesses. Imagine you run a company where you chose datadog and splunk over otel/prometheus, ELK stack and Grafana. During the upward trajectory everything looks fine. What if your business plateaued and you are looking to cut costs? What if you are in a downward trajectory and have to cut costs? You cannot hire devs then to rewrite apps so you can cut costs. You should not grab leaves after your hands are burnt(poorly translated idiomatic expression in Telugu).

I know, I digress a lot😅. Let’s get back to OpenHands

My use-case

So currently you can run OpenHands locally using the docker run command.

sudo docker run -it --rm --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.24-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
    docker.all-hands.dev/all-hands-ai/openhands:0.24

But what if I wanted to run it as a deployment in my Kubernetes cluster and serve developers from there? If I run models in the same cluster using llmariner(An excellent opensource solution to self-host, finetune and train genai models. Please do check it out). It would cut my latency by a lot. So I raised a bug in the OpenHands github page requesting for a pod definition file and here is the response I got from maintainers.

Hi @HighonAces , we don't have an open source version of deploying OpenHands on a Kubernetes cluster, but at All Hands we have a (paid) solution for deploying OpenHands to larger teams. If you'd be interested in having us help you deploy to a team please jump on the OpenHands slack and ping me and Rob*** and we could discuss more.

Honestly, I have no qualms about it. They developed a product. The developers from All Hands might have contributed a ton and they have every right to steer the path of OpenHands project as they see fit.

The kubehustle

So I took things into my own hands. The challenging thing about deploying OpenHands in K8s is that everytime you initiate an agent session on the browser, it creates a new docker container to serve as an agent. This is something different from typical usecases. It also directly mounts /var/run/docker.sock on the container which is strict no-go from security perspective. So it took me multiple attempts to get it right. And here is the pod definiton.

apiVersion: v1
kind: Pod
metadata:
  name: openhands-app-v2  # Changed to avoid conflict
spec:
  volumes:
    - name: docker-socket
      hostPath:
        path: /var/run/docker.sock
        type: Socket
    - name: openhands-state
      persistentVolumeClaim:
        claimName: openhands-state-pvc

  securityContext:
    fsGroup: 42420

  containers:
    - name: openhands-app
      image: docker.all-hands.dev/all-hands-ai/openhands:0.24
      imagePullPolicy: Always
      securityContext:
        privileged: true
      ports:
        - containerPort: 3000
      env:
        - name: SANDBOX_RUNTIME_CONTAINER_IMAGE
          value: "docker.all-hands.dev/all-hands-ai/runtime:0.24-nikolaik"
        - name: LOG_ALL_EVENTS
          value: "true"
        - name: SANDBOX_HOST
          value: "172.17.0.1"  # Replace with your host’s Docker bridge IP
        - name: SANDBOX_PORT
          value: "32315"       # Explicitly set the port
      volumeMounts:
        - name: docker-socket
          mountPath: /var/run/docker.sock
        - name: openhands-state
          mountPath: /openhands-state  # Adjusted to a typical path

  # Optional: Use hostNetwork to simplify access
  hostNetwork: true

This definition runs single container and it does not address the security concerns. If I were to run this in my company, I would run it on isolated nodes to decrease the threat surface. I was also working on another solution where we can run dind(Docker in Docker) as a sidecar container to mitigate security risks but it is not working right now.

apiVersion: v1
kind: Pod
metadata:
  name: openhands-app-dind
spec:
  volumes:
    - name: docker-run
      emptyDir: {}  # Mount at /var/run for socket
    - name: openhands-state
      persistentVolumeClaim:
        claimName: openhands-state-pvc-v2  # Updated PVC name for v2

  securityContext:
    fsGroup: 42420

  hostAliases:
    - ip: "127.0.0.1"
      hostnames:
        - "host.docker.internal"  # Map host.docker.internal to localhost

  containers:
    # Main application container
    - name: openhands-app
      image: docker.all-hands.dev/all-hands-ai/openhands:0.24
      imagePullPolicy: Always
      ports:
        - containerPort: 3000
      env:
        - name: SANDBOX_RUNTIME_CONTAINER_IMAGE
          value: "docker.all-hands.dev/all-hands-ai/runtime:0.24-nikolaik"
        - name: LOG_ALL_EVENTS
          value: "true"
        - name: DOCKER_HOST
          value: "unix:///var/run/docker.sock"  # Use DinD’s socket
        - name: SANDBOX_PORT
          value: "32315"      # Match the error’s port (verify if correct)
      volumeMounts:
        - name: docker-run
          mountPath: /var/run
        - name: openhands-state
          mountPath: /openhands-state

    # DinD sidecar container
    - name: dind
      image: docker:dind
      securityContext:
        privileged: true
      args:
        - "--host=unix:///var/run/docker.sock"  # Disable TCP
      volumeMounts:
        - name: docker-run
          mountPath: /var/run
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      resources:
        requests:
          memory: "512Mi"
          cpu: "500m"
        limits:
          memory: "1Gi"
          cpu: "1"

This is not working as expected. Even if it works, you would be trading latency with security as we will be running nested runtimes. Another concern is that Kubernetes has moved from Docker as de-facto runtime to containerd. I have to test the solution with containerd and update it.

Outro

The future looks bright for humanity as we are ushering into the AI era. Increased productivity always results in increased quality of life for everyone. I wish Openhands become de-faco agent in developer world.

Hope everyone is doing well. We are ushering into a new era where using AI tools are becoming part and parcel of everyone’s lives. There are promises of increased efficiency or fears of job loss etc. We often read the news of CEOs saying that 50% of new code being written by AI or developers getting 30% more efficient or Jansen Huang statement on coding becoming obsolete. I personally thought how is this going to affect me until it did.

This weekend I got a chance to use Github copilot and it is darn amazing. Being a Devops Engineer I am not a very good programmer because my daily work doesn’t require me to code and it is very difficult to catch up with the changes in programming trends or languages etc. I recently learnt golang but I cannot keep my go skills on the edge because I write a piece of code and then nothing for weeks or months altogether. Typically my pace is unbearably slow. I have to open 50 tabs to write 10 lines of code. I can easily get distracted or lose interest altogether. In general reading code is very easy but writing has a very steep learning curve. Github copilot changed everything for me. I can ask for a function and it is aware of the context and the file structure and writes very accurate function for my needs and also tells me what changes do I need to make.

Project Fundata

If you are a long-term investor, you need to have access to company’s financial data like income statement and cashflow etc. This is called as Fundamental Data. I used to use TIKR which costs $20/month. The idea is to get data from alphavantage API for free and put it in MongoDB and visualize it using Grafana or something equivalent by writing queries. I have written a similar README and asked copilot for project structure. It read my readme file and gave the project structure exactly like I wanted.

cmd/myapp/main.go: Entry point of the application.
internal/api/handler.go: Handles HTTP requests.
internal/db/mongo.go: MongoDB connection and operations.
internal/models/data.go: Data models.
internal/services/alpha_vantage.go: Interactions with the AlphaVantage API.
pkg/utils/utils.go: Utility functions.
go.mod and go.sum: Go modules files.
README.md: Project documentation.

And it also gave this test to make me understand what goes where.

In this example, I have not told it how the struct should look like but it took it from my readme and understood what I am trying to do. This is from my readme

Currently I have subscribed to TIKR which costs me $20/month to get fundamental data like earnings, cashflow etc. This project is to bring that cost down by having a self-implemented solution using AlphaVantage API and my impeccable coding skills in golang with the help of github copilot.

• Write a application which takes a post request with a json payload like {"symbol": "META", "function": "cash_flow", "api_key":"xxxxx"}

• Store the data in mongo-db which will be deployed in the same cluster

• Use Grafana to visualize the data stored in mongo-db

Initially for testing I have put everything in the same file. When I gave this prompt, it gave me two different files to tidy-up the code.

Ensure that the handler function in handler.go is exported (i.e., its name should start with an uppercase letter):

It gave me that there is a vulnerability in using hard-coded credentials

Initially the struct is not accurate because it did not know what incoming data looks like(duh!) So I gave another prompt with the json structure then it gave me accurate struct.

I never wrote code to connect with MongoDB in Golang. This is the first time I am seeing this and it worked without any error.

I can ask for explanations.

Make sure to replace "your_database_name" with the actual name of your MongoDB database.

This code updates the GetDividendInformation function to call the insertDividendData function, which inserts the dividend data into the MongoDB collection.

And then I asked it to get URI from secret.

Summary

I could never dreamt of writing an entire project in 2 hours. Github copilot can make us extremely efficient and language agnostic. This opens up all new possibilities

Core Components

Rather than devs raising a jira to create cloud resources, what if they can create themselves with a simple yaml file? Just as you write a definition file for pod what if you can write a definition file for a database?. Let see how it can be done but before that I have to introduce you to some concepts and definitions. This table briefly summarizes the definitons. Please go through the below table.

Crossplane Functions

There are multiple functions that are supported by Crossplane. Functions are essential to write concise steps and remove redundancy. They also allow us to use conditionals and loops when creating resources(Imagine creating multiple subnets for a VPC). Composition can be written as steps of functions, each step is used to create different Managed Resources. There are many programming languages which you can use to write functions like Python and Golang. Along with those, there configuration languages such as KCL and CUE which can be a great help to write compositions.

An Example

Enough with the theory but let’s see how we can leverage these in creating infrastructure. Let us take a scenario where I want to let users create VPC with subnets. Here is how my composition will look like

And here is my CompositeResourceDefinition(XRD) to serve this composition. It lays out an OpenAPI format to call the XRD like required parameters and their format.

These two files lays out the template for resource creation. Now I can create multiple Composite Resources or Composite Claims(namespace scoped) using the above two components. If I wanted to create a claim file in default namespace, here is how it will look like

Here is my claim file, things to be noted here are compositionRef is not necessary. I can use something like compositionSelector and get appropriate composition with matching labels. So only I apply these definitions, I can see all the resources reflected in AWS. I can also track the resource creation from CLI using commands like crossplane beta trace(CLI must be installed)

➜  network crossplane beta trace dbnetworks.db.srujanpakanati.com storage-network 
NAME                                       SYNCED   READY   STATUS
DBNetwork/storage-network (default)        True     False   Waiting: Claim is waiting for composite resource to become Ready
└─ XDBNetwork/storage-network-c5sqw        True     False   Creating: ...s-east-2b-192-168-64-0-18-private, and storage-network-c5sqw-vpc
   ├─ Subnet/storage-network-c5sqw-2zscc   True     False   Creating
   ├─ Subnet/storage-network-c5sqw-lcf4k   True     True    Available
   └─ VPC/storage-network-c5sqw-qk9wj      True     True    Available

➜  network crossplane beta trace dbnetworks.db.srujanpakanati.com storage-network 
NAME                                       SYNCED   READY   STATUS
DBNetwork/storage-network (default)        True     False   Waiting: Claim is waiting for composite resource to become Ready
└─ XDBNetwork/storage-network-c5sqw        True     False   Creating: ...s-east-2b-192-168-64-0-18-private, and storage-network-c5sqw-vpc
   ├─ Subnet/storage-network-c5sqw-2zscc   True     True    Available
   ├─ Subnet/storage-network-c5sqw-lcf4k   True     True    Available
   └─ VPC/storage-network-c5sqw-qk9wj      True     True    Available



➜  network k describe vpc storage-network-c5sqw-qk9wj 
Name:         storage-network-c5sqw-qk9wj
Namespace:    
Labels:       crossplane.io/claim-name=storage-network
              crossplane.io/claim-namespace=default
              crossplane.io/composite=storage-network-c5sqw
Annotations:  crossplane.io/composition-resource-name: storage-network-c5sqw-vpc
              crossplane.io/external-create-pending: 2024-12-07T21:01:18Z
              crossplane.io/external-create-succeeded: 2024-12-07T21:01:18Z
              crossplane.io/external-name: vpc-094642de4b2a114f6
API Version:  ec2.aws.upbound.io/v1beta1
Kind:         VPC
Metadata:
  Creation Timestamp:  2024-12-07T21:01:16Z
  Finalizers:
    finalizer.managedresource.crossplane.io
  Generate Name:  storage-network-c5sqw-
  Generation:     3
  Owner References:
    API Version:           db.srujanpakanati.com/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  XDBNetwork
    Name:                  storage-network-c5sqw
    UID:                   753881fe-b9fe-4953-8007-1a14114b520f
  Resource Version:        42765
  UID:                     e8fee79f-4d34-4078-af3f-f14cff6f1518
Spec:
  Deletion Policy:  Delete
  For Provider:
    Cidr Block:            192.168.0.0/16
    Enable Dns Hostnames:  true
    Enable Dns Support:    true
    Instance Tenancy:      default
    Region:                us-east-2
    Tags:
      Name:                         storage-network-c5sqw
      Crossplane - Kind:            vpc.ec2.aws.upbound.io
      Crossplane - Name:            storage-network-c5sqw-qk9wj
      Crossplane - Providerconfig:  default
  Init Provider:
  Management Policies:
    *
  Provider Config Ref:
    Name:  default
Status:
  At Provider:
    Arn:                                   arn:aws:ec2:us-east-2:135227014767:vpc/vpc-094642de4b2a114f6
    assignGeneratedIpv6CidrBlock:          false
    Cidr Block:                            192.168.0.0/16
    Default Network Acl Id:                acl-051e434241f655bd1
    Default Route Table Id:                rtb-0ea107329a3ddc22b
    Default Security Group Id:             sg-0c3510da7998ad5ae
    Dhcp Options Id:                       dopt-016e84c858d19736f
    Enable Dns Hostnames:                  true
    Enable Dns Support:                    true
    Enable Network Address Usage Metrics:  false
    Id:                                    vpc-094642de4b2a114f6
    Instance Tenancy:                      default
    ipv6AssociationId:                     
    ipv6CidrBlock:                         
    ipv6CidrBlockNetworkBorderGroup:       
    ipv6IpamPoolId:                        
    ipv6NetmaskLength:                     0
    Main Route Table Id:                   rtb-0ea107329a3ddc22b
    Owner Id:                              135227014767
    Tags:
      Name:                         storage-network-c5sqw
      Crossplane - Kind:            vpc.ec2.aws.upbound.io
      Crossplane - Name:            storage-network-c5sqw-qk9wj
      Crossplane - Providerconfig:  default
    Tags All:
      Name:                         storage-network-c5sqw
      Crossplane - Kind:            vpc.ec2.aws.upbound.io
      Crossplane - Name:            storage-network-c5sqw-qk9wj
      Crossplane - Providerconfig:  default
  Conditions:
    Last Transition Time:  2024-12-07T21:01:33Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-12-07T21:01:18Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
    Last Transition Time:  2024-12-07T21:01:30Z
    Reason:                Success
    Status:                True
    Type:                  LastAsyncOperation
Events:
  Type    Reason                   Age    From                                          Message
  ----    ------                   ----   ----                                          -------
  Normal  CreatedExternalResource  5m50s  managed/ec2.aws.upbound.io/v1beta1, kind=vpc  Successfully requested creation of external resource


➜  network k get dbnetworks.db.srujanpakanati.com                                 
NAME              SYNCED   READY   CONNECTION-SECRET   AGE
storage-network   True     True                       14m

As you can see that finally my network components have come up. To put everything in a nutshell
XRD → lays down the schema of the API

Composition → A blueprint to create composite resources for that API (Like a Class)

Claim → Like an object which gets a physical copy of that Composition

Please note that I have written the composition to create SQLInstances hence the API extension dbnetworks.db.srujanpakanati.com. It is just a naming convention but nothing else

Conclusion

This is a very basic use-case of Crossplane. There are lot many features like importing existing resources, managementPolicies, writeConnectionSecretToRef and using ExternalSecretStore etc. Crossplane will be a good transition from Terraform if you are already on Kubernetes.

Disclaimer: I penned this blog post long ago but releasing it on the occasion of the Kalki 2898A.D OTT release

It's been almost 5 days since I watched the movie but some elements remain in my mind reverberating at a constant hum. Hence I am putting my review in words so that I can get some closure(in a positive way😜) with the movie.

The Review

The movie starts at the end of Mahabharata where Lord Krishna confronts Aswathama on the battlefield of Kurukshetra. The visuals are exceptionally stunning in these sequences. The stories I have heard and read my entire life take a two-dimensional form in front of my eyes and are more flamboyant than I expected. The shot where the white horse runs behind Lord Krishna horripilated me. And as all good things come to an end, these scenes end and the movie takes us 6000 years in a time machine.

The dystopian world of Kashi city is well-built. The production design is thorough but the first half falls flat a few minutes after Prabhas' introduction. The unending fight sequences, The cameos that make you say "But why??🫤" and The song with Disha Patani in The Complex test your patience and make you wonder when the main story starts or did we get scammed again(Remember Adipurush?). I feel that the reason for this slow pace is due to the way the story has to split between the first and second parts. Some more informative scenes about the world of Kashi (like weapons or economy or food etc)could have been added instead of pointless cameos. But once Mr.Amitabh enters, the story shifts gears.

Before Dr.Amitabh, we must talk about Dr.Kamal Hasan. He looked in a way like a Buddhist monk who was in the process of becoming Sokushinbutsu. His appearance and performance are a sight for sore eyes in the first half. Especially when he says "భయపడకు మరొప్రపంచం వస్తుంది" is eargasm for Sri Sri fanatics. The fight sequence between Amitabh and Prabhas was very well choreographed leading to the end of the first half on a high note.

The city of Shambala shows a stark difference from Kashi. Director Nag Ashwin took good care of building something different from Kashi. In the second half, the song slowed screenplay again but the climax war sequences are a visual treat. It looked something like from Hollywood. Prabhas getting revealed as Karna is something unexpected.

Overall, the movie definitely have some flaws but it is a step in right direction. For international audience who does not know anything about The Mahabharata, this movie would not click for them. But this movie opens pathways for bigger movies to come. All the actors are aptly cast except for Vijay Devarakonda. Arjuna being an important character in the epic, the audience were expecting someone more nuanced. Music by Santosh Narayanan is Lit!! but I felt like he left a lot of opportunities on the table.

Glorifying Karna

Prabhas being cast as Karna has reinvigorated the discussion about whether Karna is good or bad. The Mahabharata being epic as it is has all the grey characters in it. Sometimes we tend to look only at some aspects of a person and judge rather than looking at everything as a whole. Karna is one such character. All his life was driven by hatred and envy towards Pandavas because he believed he belonged somewhere he could never be. I agree that Kunti shouldn't have abandoned him but she is also not planning to give birth that morning either. As fate would have it she read mantra to test its effectiveness in inviting Lord Surya without thinking of unintended consequences.

Coming back to our discussion, Karna, being raised by a stable family, and has everything he needs but he has grown up to be an angry, envious man. Everything that he learnt or achieved, he did it out of spite with Pandavas. One cannot simply be added as one of "Dushta Chatushtaya" without any reason. The Vyas Mahabharata is very clear about it. There were lot of other versions of Mahabharata like one by Villiputhurar which glorifies Karna. Members of Tamil intellectual society took the story of Karna and tried to portray it as higher caste atrocities on lower caste individuals like Karna. The Tamil industry has made the movie Karnan taking Villiputhurar's Mahabharata as a reference and NTR made Daana Veera Shoora Karna which also has these misrepresentations because of Kondaveeti Venkatakavi.

My problem here is there is Kalki continues the trend of whitewashing Karna. One generation is already disillusioned by watching movies like Karnan and DVSK, there is no need for the other generation to go on the same path. What if people in 2898AD believes that Hitler was actually a good leader who brought back Germany from WW1 with electrifying growth and prosperity and gave quotes like "Arbeit macht frei"(Work makes one free) and genocide never happened??

One who fails to understand history is doomed to repeat it. I am all for the creative liberties that are being taken and they have every right to do so. Our society should be aware of truth from lies.

Ref:

https://swarajyamag.com/culture/why-karna-was-not-the-hero-modern-retellings-of-mahabharata-make-him-out-to-be

https://talageri.blogspot.com/2021/10/karna-and-yudhisthira-in-mahabharata.html

https://www.tamilbrahmins.com/threads/was-karna-good-or-bad.19842/

In order to setup K8s cluster as SH environment, you need to have Action Runner Controller installed. It is a K8s operator to manage SHR lifecycle. It comes with its own CRDs.

ARC Installation

In order to install ARC you need to figure out authentication method first. I have used GitHubApp approach. So you will create an app and give required permissions and add the App to required repositories.

ARC needs to be authenticated with GitHub. In order to do that you need to create secret with App details so ARC can pick these up during installation.

➜  k8s-gha k get secrets -n actions controller-manager -o json |jq '.data | keys'
[
  "github_app_id",
  "github_app_installation_id",
  "github_app_private_key"
]

Now you can install ARC in the cluster using Helm

➜  ~ helm install actions-runner-controller actions-runner-controller/actions-runner-controller --namespace=actions --version=0.22.0 -f runner-controller.yaml
NAME: actions-runner-controller
LAST DEPLOYED: Sun Jul 14 16:36:39 2024
NAMESPACE: actions
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace actions -l "app.kubernetes.io/name=actions-runner-controller,app.kubernetes.io/instance=actions-runner-controller" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace actions $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace actions port-forward $POD_NAME 8080:$CONTAINER_PORT

Here is how the runner-controller file looks like

replicaCount: 1
webhookPort: 9443
syncPeriod: 1m
defaultScaleDownDelay: 5m
enableLeaderElection: true

authSecret:
  enabled: true
  create: false
  name: "controller-manager"

image:
  repository: "summerwind/actions-runner-controller"
  actionsRunnerRepositoryAndTag: "summerwind/actions-runner:ubuntu-20.04"
  dindSidecarRepositoryAndTag: "docker:dind"
  pullPolicy: IfNotPresent

serviceAccount:
  create: true

service:
  type: ClusterIP
  port: 443

certManagerEnabled: true

logFormat: text

githubWebhookServer:
  enabled: false

Now you are all set-up 🎉.

Below are the CRDs that comes with ARC

➜  k8s-gha k get crds |grep summer
horizontalrunnerautoscalers.actions.summerwind.dev    2024-07-14T21:36:26Z
runnerdeployments.actions.summerwind.dev              2024-07-14T21:36:27Z
runnerreplicasets.actions.summerwind.dev              2024-07-14T21:36:28Z
runners.actions.summerwind.dev                        2024-07-14T21:36:29Z
runnersets.actions.summerwind.dev                     2024-07-14T21:36:32Z

This follows the same pattern of K8s workloads where runners are equivalent to Pods and runnerdeployments are Deployments. You need to create a runnerdeployment with the image name, labels so GHA can pick up runners from this deployment based on the label

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    karpenter.sh/do-not-evict: "true"
  name: self-hosted-runner-deployment
  namespace: actions
spec:
  template:
    spec:
      repository: HighonAces/actions-1
      image: summerwind/actions-runner:ubuntu-20.04
      resources:
        requests:
          cpu: 1500m
          memory: 2000Mi

You also need to create HorizontalRunnerAutoscaler to scale this deployment.

apiVersion: v1
items:
- apiVersion: actions.summerwind.dev/v1alpha1
  kind: HorizontalRunnerAutoscaler
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"actions.summerwind.dev/v1alpha1","kind":"HorizontalRunnerAutoscaler","metadata":{"annotations":{},"name":"self-hosted-runner-deployment-autoscaler","namespace":"actions"},"spec":{"maxReplicas":30,"minReplicas":0,"scaleTargetRef":{"kind":"RunnerDeployment","name":"self-hosted-runner-deployment"},"scaleUpTriggers":[{"duration":"30m","githubEvent":{"workflowJob":{}}}]}}
    creationTimestamp: "2024-07-14T21:52:23Z"
    generation: 2
    name: self-hosted-runner-deployment-autoscaler
    namespace: actions
    resourceVersion: "9689"
    uid: 41291b4a-cbd8-4af2-bfe3-02c2b29d8262
  spec:
    maxReplicas: 30
    minReplicas: 2
    scaleTargetRef:
      kind: RunnerDeployment
      name: self-hosted-runner-deployment
    scaleUpTriggers:
    - duration: 30m
      githubEvent:
        workflowJob: {}
  status:
    desiredReplicas: 2
    lastSuccessfulScaleOutTime: "2024-07-14T22:28:46Z"
kind: List
metadata:
  resourceVersion: ""

This continuously keeps 2 replicas running as I have not given any metrics definition. By making use of PercentageRunnersBusy, you can scaleup or scaledown. Here is the metrics definition from ARC documentation

  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.25'
    scaleUpFactor: '2'
    scaleDownFactor: '0.5'

Once you have min number of runners running you can see the corresponding pods in cluster and same in GH runners page.

These runners will pickup the jobs whenever you have actions configured with this parameter runs-on: self-hosted-linux.

Conclusion

This setup is rather easy and customizable. It is a very good alternative for the GHR and I cannot wait to see what future holds for K8s based Self hosted runners.

Source: https://medium.com/simform-engineering/how-to-setup-self-hosted-github-action-runner-on-kubernetes-c8825ccbb63c

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller

https://github.com/actions/actions-runner-controller

KEDA is Kubernetes-based Event Driven Autoscaler. It can be used to scale any container based resource like Deployments, StatefulSets etc whenever an event occurs and then scale back to zero when it is not necessary or it can be used to create Kubernetes Jobs during events. It can be used to scale both Horizontally and Vertically.

Need for KEDA

KEDA is an important player to maximise your usage efficiency. You simply cannot keep your deployments running in the hope of traffic/event occurs. You must be able to create resources at the minute of need and must be able to terminate them as soon as their purpose is served. This is especially useful for Jobs. KEDA can ingest metrics from any sources(currently 68 scalers available). Here are some of the examples.

• Vertical scaling of pods based on avg resource usage metrics from Prometheus or Datadog.

• Scaling based on ALB metrics.

• Scaling jobs based on Kafka or ActiveMQ queues.

• Scaling based on query results from databases like MongoDb, DynamoDb and MySQL etc.

• Scaling Github runners based number of jobs pending in GHA.

These are some of the examples but you can use any event to scale your resources.

KEDA Architecture

Lets first see all the CRDs that KEDA offers

➜  keda k get crds |grep keda
cloudeventsources.eventing.keda.sh                    2024-07-12T02:20:03Z
clustertriggerauthentications.keda.sh                 2024-07-12T02:20:03Z
scaledjobs.keda.sh                                    2024-07-12T02:20:03Z
scaledobjects.keda.sh                                 2024-07-12T02:20:03Z
triggerauthentications.keda.sh                        2024-07-12T02:20:03Z

ScaledObjects

ScaledObjects defines what resource you want to scale and based on what triggers. The resource would be typically Deployment or StatefulSet.

spec:
  scaleTargetRef:
    apiVersion:    {api-version-of-target-resource}         # Optional. Default: apps/v1
    kind:          {kind-of-target-resource}                # Optional. Default: Deployment
    name:          {name-of-target-resource}                # Mandatory. Must be in the same namespace as the ScaledObject

Triggers is the external source based on which you want to scale the said object. ScaledJobs are similar to ScaledObjects. The only difference is you create/scale Jobs based on the external source.

TriggerAuthentication

As you define external triggers, you also need a way to authenticate to that source. For this purpose you need TriggerAuthentication. ClusterTriggerAuthentication is a similar one with cluster-scope.

CloudEventSource

CloudEventSource resource can be used in KEDA for subscribing to events that are emitted to the user’s defined CloudEvent sink.

Here's how all works together. ScaledObject first checks if TriggerAuthentication works or not and if could able to get required metrics from the external source. Also checks if the object that it needs to be scaled is defined or not. Then it will create a Horizontal Pod Autoscaler(HPA) for the object and starts passing metrics to HPA based on the data from external source.

Demo: Using MongoDB to scale Deployment

Here I am trying to scale deployment based on MongoDb query results. I have created a deployment for nginx with one replica, pretty standard

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx-dep
  name: nginx-dep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-dep
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-dep
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}

My target is to scaledown this deployment to zero when my mongodb document has "enabled" value is set as "no" and scales it whenever the value is "yes". So my scaled object definition looks something like

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: nginx-so
  namespace: default
spec:
  scaleTargetRef:
    name: nginx-dep
  cooldownPeriod:  30
  minReplicaCount: 0
  maxReplicaCount: 5

  triggers:
  - type: mongodb
    metadata:
      dbName: "drinks"
      collection: "softdrinks"
      query: '{ "enabled": "yes" }'
      queryValue: "1"
    authenticationRef: 
      name: mongodb-trigger
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata: 
  name: mongodb-trigger
spec: 
  secretTargetRef:
    - parameter: connectionString
      name: mongodb-secret
      key: connect

You need to create mongodb secret with the key connect and value should be mongodb connection string mongodb+srv://:@******.*****.mongodb.net/<dbname>

apiVersion: v1
kind: Secret
metadata: 
  name: mongodb-secret
type: Opaque
data: 
  connect: bW9uZ29kYitzcnY6Ly86QCoqKioqKi4qKioqKi5tb25nb2RiLm5ldC8K==

Now we have everything in place. Once you create ScaledObject and triggerauthentications, SO will create a HPA and converts the query metric to HPA understandable metric.

Here I have changed value {"enabled": "no"}. Now when you describe ScaledObject you will see

➜  keda k get so nginx-so                
NAME       SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS   AUTHENTICATION    READY   ACTIVE   FALLBACK   PAUSED    AGE
nginx-so   apps/v1.Deployment   nginx-dep         0     5     mongodb    mongodb-trigger   True    False    False      Unknown   28m

Name:         nginx-so
Namespace:    default
Labels:       scaledobject.keda.sh/name=nginx-so
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2024-07-13T06:44:57Z
  Finalizers:
    finalizer.keda.sh
  Generation:        1
  Resource Version:  144637
  UID:               9b385df8-88a4-4d0a-bc68-4e7ffea7f8f7
Spec:
  Cooldown Period:    30
  Max Replica Count:  5
  Min Replica Count:  0
  Scale Target Ref:
    Name:  nginx-dep
  Triggers:
    Authentication Ref:
      Name:  mongodb-trigger
    Metadata:
      Collection:   softdrinks
      Db Name:      drinks
      Query:        { "enabled": "yes" }
      Query Value:  1
    Type:           mongodb
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is not performed because triggers are not active
    Reason:   ScalerNotActive
    Status:   False
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
    Status:   Unknown
    Type:     Paused
  External Metric Names:
    s0-mongodb-softdrinks
  Health:
    s0-mongodb-softdrinks:
      Number Of Failures:  0
      Status:              Happy
  Hpa Name:                keda-hpa-nginx-so
  Last Active Time:        2024-07-13T07:01:28Z
  Original Replica Count:  0
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason                      Age   From           Message
  ----     ------                      ----  ----           -------
  Warning  KEDAScalerFailed            29m   keda-operator  failed to parsing mongoDB metadata, because of missing required field in scaler config: no host given
  Warning  ScaledObjectCheckFailed     29m   keda-operator  failed to ensure HPA is correctly created for ScaledObject
  Normal   KEDAScalersStarted          29m   keda-operator  Scaler mongodb is built.
  Normal   KEDAScalersStarted          29m   keda-operator  Started scalers watch
  Normal   ScaledObjectReady           29m   keda-operator  ScaledObject is ready for scaling
  Normal   KEDAScaleTargetActivated    28m   keda-operator  Scaled apps/v1.Deployment default/nginx-dep from 0 to 1, triggered by mongoDBScaler
  Normal   KEDAScaleTargetDeactivated  12m   keda-operator  Deactivated apps/v1.Deployment default/nginx-dep from 1 to 0

The statements
Message: Scaling is not performed because triggers are not active. and Normal KEDAScaleTargetDeactivated 12m keda-operator Deactivated apps/v1.Deployment default/nginx-dep from 1 to 0 shows that the trigger is not active so scaled to zero.

HPA reflects the same

➜  keda k get hpa
NAME                REFERENCE              TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-nginx-so   Deployment/nginx-dep   <unknown>/1 (avg)   1         5         0          32m
➜  keda k describe hpa keda-hpa-nginx-so 
Name:                                              keda-hpa-nginx-so
Namespace:                                         default
Labels:                                            app.kubernetes.io/managed-by=keda-operator
                                                   app.kubernetes.io/name=keda-hpa-nginx-so
                                                   app.kubernetes.io/part-of=nginx-so
                                                   app.kubernetes.io/version=2.14.0
                                                   scaledobject.keda.sh/name=nginx-so
Annotations:                                       <none>
CreationTimestamp:                                 Sat, 13 Jul 2024 01:44:58 -0500
Reference:                                         Deployment/nginx-dep
Metrics:                                           ( current / target )
  "s0-mongodb-softdrinks" (target average value):  <unknown> / 1
Min replicas:                                      1
Max replicas:                                      5
Deployment pods:                                   0 current / 0 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededGetScale   the HPA controller was able to get the target's current scale
  ScalingActive   False   ScalingDisabled     scaling is disabled since the replica count of the target is zero
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range

Now if I change enabled to yes

Name:         nginx-so
Namespace:    default
Labels:       scaledobject.keda.sh/name=nginx-so
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2024-07-13T06:44:57Z
  Finalizers:
    finalizer.keda.sh
  Generation:        1
  Resource Version:  146171
  UID:               9b385df8-88a4-4d0a-bc68-4e7ffea7f8f7
Spec:
  Cooldown Period:    30
  Max Replica Count:  5
  Min Replica Count:  0
  Scale Target Ref:
    Name:  nginx-dep
  Triggers:
    Authentication Ref:
      Name:  mongodb-trigger
    Metadata:
      Collection:   softdrinks
      Db Name:      drinks
      Query:        { "enabled": "yes" }
      Query Value:  1
    Type:           mongodb
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is performed because triggers are active
    Reason:   ScalerActive
    Status:   True
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
    Status:   Unknown
    Type:     Paused
  External Metric Names:
    s0-mongodb-softdrinks
  Health:
    s0-mongodb-softdrinks:
      Number Of Failures:  0
      Status:              Happy
  Hpa Name:                keda-hpa-nginx-so
  Last Active Time:        2024-07-13T07:19:58Z
  Original Replica Count:  0
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason                      Age                From           Message
  ----     ------                      ----               ----           -------
  Warning  KEDAScalerFailed            35m                keda-operator  failed to parsing mongoDB metadata, because of missing required field in scaler config: no host given
  Warning  ScaledObjectCheckFailed     35m                keda-operator  failed to ensure HPA is correctly created for ScaledObject
  Normal   KEDAScalersStarted          35m                keda-operator  Scaler mongodb is built.
  Normal   KEDAScalersStarted          35m                keda-operator  Started scalers watch
  Normal   ScaledObjectReady           35m                keda-operator  ScaledObject is ready for scaling
  Normal   KEDAScaleTargetDeactivated  18m                keda-operator  Deactivated apps/v1.Deployment default/nginx-dep from 1 to 0
  Normal   KEDAScaleTargetActivated    34s (x2 over 33m)  keda-operator  Scaled apps/v1.Deployment default/nginx-dep from 0 to 1, triggered by mongoDBScaler

and HPA looks like

Name:                                              keda-hpa-nginx-so
Namespace:                                         default
Labels:                                            app.kubernetes.io/managed-by=keda-operator
                                                   app.kubernetes.io/name=keda-hpa-nginx-so
                                                   app.kubernetes.io/part-of=nginx-so
                                                   app.kubernetes.io/version=2.14.0
                                                   scaledobject.keda.sh/name=nginx-so
Annotations:                                       <none>
CreationTimestamp:                                 Sat, 13 Jul 2024 01:44:58 -0500
Reference:                                         Deployment/nginx-dep
Metrics:                                           ( current / target )
  "s0-mongodb-softdrinks" (target average value):  1 / 1
Min replicas:                                      1
Max replicas:                                      5
Deployment pods:                                   1 current / 1 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-mongodb-softdrinks(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: nginx-so,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

Source:

https://devtron.ai/blog/introduction-to-kubernetes-event-driven-autoscaling-keda/#keda-architecture-and-components

https://medium.com/@mohammadsaquib.ee/mongodb-powered-autoscaling-harnessing-keda-to-scale-applications-dynamically-based-on-database-f38a68e71db6

https://keda.sh/docs/2.14/scalers/mongodb/

1. Why do need Gateway API when there is Ingress?

Ingress is used to manage external traffic into the cluster. Rather than creating induvidual LBs for each service, Ingress allows us to use one load balancer to serve traffic to all the services. That being said, there are many drawbacks in Ingress implementation. Ingress only supports L7 HTTP traffic. It does not support other L7 protocols or L4 protocols like TCP and UDP. This opens room for many problems. Moreover there is no support for advanced features like rate-limiting, A/B testing etc. This makes customers use vendor-specific annotations which leads to vendor lock-in. Ingress does not also support Header based routing. All these drawbacks made Ingress inferior product. Hence Gateway API is introduced

Gateway API is modern, extensible traffic management solution in K8s which support both L4 and L7 protocols. GatewayAPI completely embraces the extensibility of K8s by introducing new Custom Resources like Gateway , GatewayClass and HTTPRoute. This establishes a clear set of boundary for each CR making them replacable. It supports all advanced features like request mirroring, fine grained traffic metrics etc. GatewayAPI also sets standard set of definition files for API objects so one vendor can be replcaed easily with other vendors. This is a pain point in the Ingress' case.

2. How do you manage secrets in K8s?

Secrets are an interesting aspect of Kubernetes. The inbuilt Kubernetes secrets are not encrypted at rest but just encoded. So using inbuilt secrets is a huge security vulnerability. If your etcd cluster data is compromised then all your secrets can be accessed in a plain text format. One way to prevent this is to encrypt ectd data at rest.

One can use cloud-specific secret stores to manage secrets and use cloud-native tools like External Secrets Operator, secret store CSI driver etc. There are also alternatives like Hashicorp Vault along with vault sidecar container that directly inserts secrets into pod by skipping K8s secrets as a whole. This is the most secure way of managing secrets.

3. How do you debug applications in K8s?

Debugging apps in K8s environment is a challenging task as the pods will not/should not have debugging tools like Curl or nc etc(rightfully so for security purposes). This can be addressed by using ephemeral containers like busybox using kubectl debug command. This helps with troubleshooting pods, services etc and understanding where the issue lies

kubectl debug -it ephemeral-demo --image=busybox:1.28 --target=ephemeral-demo

Apart from that you can analyze K8s events, use kubectl describe to get detailed understanding of state of application. You can analyize logs using kubectl logs or port-forward for further debugging.

4. What are Custom Resources?

While using K8s features, you can also extend them by developing new CRDs as per requirement. You can create new CRDs and Custom Controllers to manage them using frameworks like OperatorSDK, Kubebuilder etc. This feature makes K8s extensible and programmable.

5. When do you not deploy in Kubernetes?

Though K8s tries to appeal to wide variety of workloads there might be use-cases where K8s is not ideal solution for your problem like

1. If you have small number of identical applications.

2. If your applications are not optimized for the cloud.

3. If your app is still monolithic.

4. If you cannot keep-up with the k8s releases and migrations.

5. You don't need to scale or deploy new applications.

6. How do you scale K8s resources?

K8s is built by keeping scalability in mind. Ideally, K8s can scale up to 5000 nodes or 150,000 pods. There are lot of built-in features to scale pods horizontally and vertically like HPA and VPA. You can set the CPU and memory threshold and once these numbers are hit then HPA will scale the number of pods. There is KEDA(Kubernetes-based Event Driven Autoscaler) using which we can scale applications based on custom metrics.

When it comes to scaling a K8s cluster, there are tools like Cluster Autoscaler which can add nodes if there are un-schedulable pods but Cluster Autoscaler treats all the nodes same. That is where Karpenter comes into picture. You can setup nodePools for different application types or for different regions etc. Karpenter is a best solution out there for scaling cluster.

7. How do you handle logs in K8s?

For small clusters and where you do not need to retain logs for any purposes, kubectl logs command is just fine. But if you need to retain, aggregate and analyze logs then you have to look at logging solutions like Splunk, Logz etc. These are paid services. Apart from these you can use stacks like EFK, ELK and Promtail, Grafana and Loki where one tool collects the logs and forwards them(Fluentd, Logstash and Promtail) one acts as a database (Elasticsearch, Loki) where as other tool is used for querying, visualization and alerts.

8. Why do you need to use ServiceMesh?

Service mesh can be a great infrastructure add-on to the K8s cluster. Service Meshes like Istio, Cilium and Linkerd can offer lot of features out of box like mTLS, canary deployment, progressive deployment, traffic splitting, A/B testing and circuit breaking etc. Service meshes increase observability, security and reliability of apps in the cluster.
The caveat is that the service mesh comes with lot of complexity and architectural overhead to the cluster. If your workloads are simple and do not need additional latency(though minimal) or complexity, you can stay away from them.

9. How do you ensure security of K8s cluster?

Though security is a vast topic, I can try to be brief. Firstly, by using network policies, apps can restrict access to other namespaces. Using RBAC to make sure only users/service accounts with proper permissions are accessing the resources. Using Policy-as-a-Code tools like OPA or Kyverno to set guardrails around apps getting deployed. mTLS for app to app communication. Setting threat monitoring tools like Falco and PSA for early detection and mitigation. Setting up seperate clusters for apps handling sensitive data to make them PCI-DSS, HIPPA and GDPR compatible.

If I am using managed K8s like EKS or GKE, using cloud security features like KMS, Golden AMIs etc.

10. How do you handle Stateful applications in K8s?

Traditionally Kubernetes has a bad reputation for stateful applications like PostgreSQL because popular belief is that K8s is only meant for stateless workloads. That is not true at all. StatefulSets are used to maintain the identity of pods across restarts and redeployments. However, I would not use storage in K8s nodes to create StorageClass or PVs because of the ephemeral nature of nodes, Using something like EBS volume to create SC using CSI driver and creating PVC from it is recommended.

Alternatively, there are many OS solutions like Cloud Native PG and StackGres which offer better approach to the problem. Running databases on Kubernetes is still an evolving topic which can see many changes in coming days.

EFK stands for ElasticSearch, Fluentd and Kibana. It is similar to the ELK stack where we are replacing Logstash with Fluentd. This is a free, open-source alternative to Splunk for log aggregation, processing and visualisation. I have been going through the KodeKloud course on EFK stack but there you are using local volume for persistent storage which goes against the Kubernetes principles. Here I am trying to deploy the EFK stack on EKS cluster using EBS add-on.

Here is a small primer on tools that the stack is based upon

Elasticsearch

It is a distributed NoSQL database and search and analytics engine based on Apache Lucene. I have been using elasticsearch to store logs since the beginning of my career.

Fluentd

Fluentd is a lightweight log-forwarder and indexer. It is deployed as Daemonset and collects all the logs and forwards them to elasticsearch.

Kibana

Kibana is a querying and log-visualization dashboard. It can be used to query logs and application monitoring.

Creating EKS cluster

I have used eksctl command to create the cluster. There are more efficient ways like writing clusterConfig for more verbose and consistent cluster creation. But stupid me always do Ctrl+R to see and predict which cluster I have created previously😆

eksctl create cluster \
  --name test-efk-stack \
  --region us-east-1 \
  --version 1.29 \
  --with-oidc \
  --node-type t3.medium \
  --nodes 2 \
  --managed

Creating an IAM role and associating it with SA

eksctl create iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster test-efk-stack \
  --region us-east-1 \
  --attach-policy-arn $POLICY_ARN \
  --role-only \
  --role-name $ROLE_NAME \
  --approve

Now I am installing the EBS addon to the cluster

eksctl create addon \                                            
  --name "aws-ebs-csi-driver" \
  --cluster $EKS_CLUSTER_NAME \
  --region=us-east-1 \
  --service-account-role-arn $ACCOUNT_ROLE_ARN \
  --force

The EBS csi driver is now installed.

➜  ~ k get pods -n kube-system 
NAME                                  READY   STATUS    RESTARTS   AGE
aws-node-bpfkn                        2/2     Running   0          21m
aws-node-nbwm6                        2/2     Running   0          21m
coredns-54d6f577c6-4ghdt              1/1     Running   0          28m
coredns-54d6f577c6-hm5dj              1/1     Running   0          28m
ebs-csi-controller-86b8d8bb96-256dg   6/6     Running   0          3m1s
ebs-csi-controller-86b8d8bb96-kfzhb   6/6     Running   0          3m1s
ebs-csi-node-5jqfm                    3/3     Running   0          3m1s
ebs-csi-node-6rgjg                    3/3     Running   0          3m1s
kube-proxy-wl4z7                      1/1     Running   0          21m
kube-proxy-wxcls                      1/1     Running   0          21m

Note: I have followed parts of this blog post to create cluster

Installing Elasticsearch

We install the database first and then we will install Fluentd and finally Kibana.

You can install using the Helm chart provided in this blog and I tried the same. But the problem is that in default provisions 3 pods and each pod provisions 30GB which is unnecessary. I can use my own values but I went in another way and installed Stateful set and Service directly using this github repo. I had to change some things around but it finally started to work.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: efk
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch:8.5.1
          resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
          ports:
            - containerPort: 9200
              name: rest
              protocol: TCP
            - containerPort: 9300
              name: inter-node
              protocol: TCP
          volumeMounts:
            - name: data
              mountPath: /usr/share/elasticsearch/data
          env:
            - name: cluster.name
              value: k8s-logs
            - name: network.host
              value: 0.0.0.0
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: discovery.seed_hosts
              value: "es-cluster-0.elasticsearch"
            - name: discovery.type
              value: single-node
            - name: xpack.license.self_generated.type
              value: "trial"
            - name: xpack.security.enabled
              value: "true"
            - name: xpack.monitoring.collection.enabled
              value: "true"
            - name: ES_JAVA_OPTS
              value: "-Xms256m -Xmx256m"
            - name: ELASTIC_PASSWORD
              value: "elasticpassword"
      initContainers:
        - name: fix-permissions
          image: busybox
          command:
            ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
          securityContext:
            privileged: true
          volumeMounts:
            - name: data
              mountPath: /usr/share/elasticsearch/data
        - name: increase-vm-max-map
          image: busybox
          command: ["sysctl", "-w", "vm.max_map_count=262144"]
          securityContext:
            privileged: true
        - name: increase-fd-ulimit
          image: busybox
          command: ["sh", "-c", "ulimit -n 65536"]
          securityContext:
            privileged: true
  volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app: elasticsearch
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp2
        resources:
          requests:
            storage: 5Gi

kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: efk
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node
  type: LoadBalancer

Installing Kibana and Fluentd

Kibana can be installed using simple Helm command

helm install kibana --set service.type=LoadBalancer elastic/kibana -n efk

After successfully installing Kibana, I have installed Fluentd and modified the config file as well.

Conclusion

EFK is a powerful opensource alternative for Splunk. This cost-effective solution works for the small to mid-level enterprises who have restricted budgets. In future articles we'll explore scaling and securing EFK stack.

Managing secrets in Kubernetes is tricky as the "Secrets" are not really Secrets but just encoded configMaps. You cannot store Secrets definitions in Helm on source code or you cannot rotate these secrets. So there must be a better way of managing secrets. You might already have secrets somewhere else like Vault or AWS SecretsManager that you want to access in the cluster. One way of addressing this issue is by using External Secrets Operator(ESO)

ESO Architecture

ESO extends Kubernetes functionality using Custom Resources. Firstly you can install ESO in cluster using Helm

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \

This installs bunch of CRDs from ESO

➜  ESOperator k get crds |grep external-secrets
acraccesstokens.generators.external-secrets.io          2024-05-20T16:34:03Z
clusterexternalsecrets.external-secrets.io              2024-05-20T16:34:03Z
clustersecretstores.external-secrets.io                 2024-05-20T16:34:03Z
ecrauthorizationtokens.generators.external-secrets.io   2024-05-20T16:34:03Z
externalsecrets.external-secrets.io                     2024-05-20T16:34:03Z
fakes.generators.external-secrets.io                    2024-05-20T16:34:03Z
gcraccesstokens.generators.external-secrets.io          2024-05-20T16:34:03Z
githubaccesstokens.generators.external-secrets.io       2024-05-20T16:34:03Z
passwords.generators.external-secrets.io                2024-05-20T16:34:03Z
pushsecrets.external-secrets.io                         2024-05-20T16:34:03Z
secretstores.external-secrets.io                        2024-05-20T16:34:03Z
vaultdynamicsecrets.generators.external-secrets.io      2024-05-20T16:34:03Z
webhooks.generators.external-secrets.io                 2024-05-20T16:34:03Z

Of these the important ones to understand the architecture are

SecretStore secretstores.external-secrets.io

SecretStore is like a centralized location where all your secrets are located like Vault, AWS Secrets Manager, Azure Key Vault etc. All the providers are listed in their documentation. You can configure it using the resource definition file. For AWS SM I have created new user and gave it permissions for the SM and configured the SecretStore with the below definition file

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

Once you configure it, you have to make sure it is Ready

➜  ESOperator k get secretstores.external-secrets.io        
NAME                 AGE     STATUS   CAPABILITIES   READY
secretstore-sample   3h42m   Valid    ReadWrite      True

Now I can access all the secrets from SM in the cluster. To get one secret, you need to write definition for externalSecret

ExternalSecret externalsecrets.external-secrets.io

As the name suggests, the external secret is a secret from external source. It depends on the secretstore to provide the secret. I have used below definition file to fetch secret from the AWSSM named DB_CREDENTIALS

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 10m
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: kube-secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: DB_CREDENTIALS

Here target.name is the name of secret that you want to be created. As you can see that the secret is created here.

NAME           TYPE     DATA   AGE
awssm-secret   Opaque   2      4h32m
kube-secret    Opaque   2      3h52m

The thing is the secret is not encrypted. You have to do etcd encryption at rest to protect all the data. Using these two can make your cluster secure and robust.

Conclusion

I can see that ESO is extensible and can be used in various scenarios. This is a compelling alternative to vault sidecar injection. Especially when you take cost into consideration.

Istio is a service mesh for Kubernetes clusters. It offers many features like zero trust network using mTLS, Traffic management, Routing, observability using Kiali etc. Well how does this do it? By inserting a lightweight sidecar container as proxy in all the pods. So traffic flows to and from these proxy containers. In layman terms, if Kubernetes cluster is a town with paved roads, istio installs highways with traffic signals, CCTV cameras installed, and a centralized Traffic HQ.

Limitations with Istio

The problem with istio is that it needs dedicated resources for sidecar containers. Generally 128Mb memory and 100m CPU is given to the istio sidecars. The resource consumption is minimal, but when you add it to all the pods running in a cluster, it is a significant amount of resources.

The other problem is that traffic between pods has to do two hops i.e. to and from sidecar container, which adds to the latency, Yes minimal (~10-20ms) but still it adds up. This article explores some more of it.

What is Istio Ambient Mesh

On the surface, it is just a new way of proxying without injecting sidecar containers. This ups the istio's game bringing it equivalent to Cilium where Cilium uses eBPF for manipulating traffic. Istio now separates the concerns of L4 and L7 proxying by using two different ways to proxy traffic. This revolutionises the way of using Istio.

For L4 proxying, istio uses node proxy called ztunnel. It is a Daemonset which is deployed on every node and modifies the bidirectional traffic. It secures traffic using mTLS.

For L7 proxying, Istio uses waypoint proxy. It can be deployed as one for each service running in parallel with

Testing Istio in Ambient Mode

For this I am trying out the test setup for ambient mode written in Istio's blog

I have created a Kind cluster

➜  terraform_projects git:(main) ✗ kind create cluster --config=- <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ambient
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
Creating cluster "ambient" ...
 ✓ Ensuring node image (kindest/node:v1.29.2) 🖼 
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-ambient"
You can now use your cluster with:

kubectl cluster-info --context kind-ambient

Thanks for using kind! 😊
➜  terraform_projects git:(main) ✗ k get nodes
NAME                    STATUS   ROLES           AGE     VERSION
ambient-control-plane   Ready    control-plane   5m58s   v1.29.2
ambient-worker          Ready    <none>          5m34s   v1.29.2
ambient-worker2         Ready    <none>          5m34s   v1.29.2

Now to install istio, I am using istioctl

➜  ~ istioctl install --set profile=ambient

This will install the Istio 1.21.2 "ambient" profile (with components: Istio core, Istiod, CNI, and Ztunnel) into the cluster. Proceed? (y/N) y
✔ Istio core installed                                                                                                                                                                  
✔ Istiod installed                                                                                                                                                                      
✔ CNI installed                                                                                                                                                                         
✔ Ztunnel installed                                                                                                                                                                     
✔ Installation complete                                                                                                                                                                Made this installation the default for injection and validation.

If we check the pods installed in istio-system,

➜  ~ k get pods -n istio-system 
NAME                     READY   STATUS    RESTARTS   AGE
istio-cni-node-blxdv     1/1     Running   0          8m2s
istio-cni-node-k5q9p     1/1     Running   0          8m2s
istio-cni-node-nmg86     1/1     Running   0          8m2s
istiod-ff94b9d97-6tzfp   1/1     Running   0          8m17s
ztunnel-dsfdh            1/1     Running   0          7m28s
ztunnel-kwnz7            1/1     Running   0          7m28s
ztunnel-z78gf            1/1     Running   0          7m28s

Both ztunnel and istio CNI are installed as daemonSets. ztunnel is to intercept the traffic flowing between nodes and CNI intercepts traffic between pods in a same node and redirects it through ztunnel. Like forcing the traffic to implement the ambient mode.

To test it, I have installed bookinfo application, sleep and notsleep pods as well.

➜  ~ k apply -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/platform/kube/bookinfo.yaml
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created
➜  ~ k get pods
NAME                             READY   STATUS              RESTARTS   AGE
details-v1-cf74bb974-8bmhv       0/1     ContainerCreating   0          32s
productpage-v1-87d54dd59-2mxrt   1/1     Running             0          32s
ratings-v1-7c4bbf97db-cxmps      1/1     Running             0          32s
reviews-v1-5fd6d4f8f8-sfzv4      0/1     ContainerCreating   0          32s
reviews-v2-6f9b55c5db-9wh4k      0/1     ContainerCreating   0          32s
reviews-v3-7d99fd7978-jm5sr      0/1     ContainerCreating   0          32s
➜  ~ kubectl apply -f https://raw.githubusercontent.com/linsun/sample-apps/main/sleep/sleep.yaml
serviceaccount/sleep created
service/sleep created
deployment.apps/sleep created
➜  ~ kubectl apply -f https://raw.githubusercontent.com/linsun/sample-apps/main/sleep/notsleep.yaml


serviceaccount/notsleep created
service/notsleep created
deployment.apps/notsleep created

As the ambient mode is enabled to induvidual namespaces, and we haven't enabled it yet. You can see that I can hit other pods from sleep pod and get response

➜  ~ kubectl exec deploy/sleep -- curl -s http://productpage:9080/           
<!DOCTYPE html>
<html>
  <head>
    <title>Simple Bookstore App</title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">

After setting the ambient mode using kubectl label namespace default istio.io/dataplane-mode=ambient I could see that the traffic is now flowing through ztunnel

➜  ~ kubectl label namespace default istio.io/dataplane-mode=ambient
namespace/default labeled
➜  ~ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n1
<!DOCTYPE html>
➜  ~ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n5
<!DOCTYPE html>
<html>
  <head>
    <title>Simple Bookstore App</title>
<meta charset="utf-8">

Here are the ztunnel logs for the same

2024-05-17T20:25:18.143503Z     info    access  connection complete     src.addr=10.244.1.4:42786 src.workload=sleep-97576b68f-ftqsh src.namespace=default src.identity="spiffe://cluster.local/ns/default/sa/sleep" dst.addr=10.244.2.8:15008 dst.hbone_addr=10.244.2.8:9080 dst.service=productpage.default.svc.cluster.local dst.workload=productpage-v1-87d54dd59-jvj79 dst.namespace=default dst.identity="spiffe://cluster.local/ns/default/sa/bookinfo-productpage" direction="outbound" bytes_sent=79 bytes_recv=1918 duration="128ms"
2024-05-17T20:25:19.755188Z     info    access  connection complete     src.addr=10.244.1.4:42796 src.workload=sleep-97576b68f-ftqsh src.namespace=default src.identity="spiffe://cluster.local/ns/default/sa/sleep" dst.addr=10.244.2.8:15008 dst.hbone_addr=10.244.2.8:9080 dst.service=productpage.default.svc.cluster.local dst.workload=productpage-v1-87d54dd59-jvj79 dst.namespace=default dst.identity="spiffe://cluster.local/ns/default/sa/bookinfo-productpage" direction="outbound" bytes_sent=79 bytes_recv=1918 duration="5ms"
2024-05-17T20:25:21.033103Z     info    access  connection complete     src.addr=10.244.1.4:42810 src.workload=sleep-97576b68f-ftqsh src.namespace=default src.identity="spiffe://cluster.local/ns/default/sa/sleep" dst.addr=10.244.2.8:15008 dst.hbone_addr=10.244.2.8:9080 dst.service=productpage.default.svc.cluster.local dst.workload=productpage-v1-87d54dd59-jvj79 dst.namespace=default dst.identity="spiffe://cluster.local/ns/default/sa/bookinfo-productpage" direction="outbound" bytes_sent=79 bytes_recv=1918 duration="7ms"

Other than mTLS, you can set Authorization policies for different workloads.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: productpage-viewer
 namespace: default
spec:
 selector:
   matchLabels:
     app: productpage
 action: ALLOW
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep", "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]

➜  ~ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n1
<!DOCTYPE html>
➜  ~ kubectl exec deploy/notsleep -- curl -s http://productpage:9080/ | head -n1


command terminated with exit code 56

By using istio gateway, you can bring up waypoint proxy to do L7 loadbalancing. Which creates waypoint proxy pods which will not be attached to application pods but stay as stand-alone pods.

Trying on EKS

I also tried the same demo in the EKS cluster, ztunnel daemonset is not coming up.

➜  ~ k get pods -n istio-system 
NAME                      READY   STATUS    RESTARTS   AGE
istio-cni-node-26bn9      1/1     Running   0          5m27s
istio-cni-node-kzd82      1/1     Running   0          5m27s
istio-cni-node-x99n2      1/1     Running   0          5m27s
istiod-5888647857-d9vdh   0/1     Pending   0          5m22s
ztunnel-7qr8l             0/1     Pending   0          5m22s
ztunnel-pgpsn             0/1     Pending   0          5m22s
ztunnel-z2kvc             0/1     Running   0          5m22s

The logs show that the pods are failing with XDS client connection error

2024-05-17T17:33:58.934625Z     warn    xds::client:xds{id=29}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:34:13.939490Z     warn    xds::client:xds{id=30}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:34:28.943546Z     warn    xds::client:xds{id=31}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:34:43.947796Z     warn    xds::client:xds{id=32}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:34:58.950405Z     warn    xds::client:xds{id=33}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:35:13.955870Z     warn    xds::client:xds{id=34}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:35:28.960054Z     warn    xds::client:xds{id=35}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:35:43.964064Z     warn    xds::client:xds{id=36}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:35:58.968351Z     warn    xds::client:xds{id=37}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:36:13.972471Z     warn    xds::client:xds{id=38}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:36:28.975409Z     warn    xds::client:xds{id=39}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s
2024-05-17T17:36:43.979856Z     warn    xds::client:xds{id=40}  XDS client connection error: gRPC connection error:status: Unknown, message: "client error (Connect)", source: tcp connect error: Connection refused (os error 111), retrying in 15s

Conclusion

I believe this is Istio's attempt to stay relevant and adress all the customer concerns and it hit nail in the head. Looking forward to test it further.

Firstly, Zero Trust Network is where no network entity is trusted by default or with respect to its position in the network. Every workload, node, machine needs to continuously identify itself.

Shortcomings of traditional network architecture.

Traditionally IT companies used to have a network with firewall and all the entities within the network are trusted by default. This is also called as castle and moat architecture. We all have used VPNs to connect to the network. The problem with this approach is that once an attacker has access to the network. He have free reign over the network. He could access all the data he could. Now-a-days, the networks are spanned across different clouds and hence the traditional network architecture will not work.

Additional risks due to microservices

The emergence of microservices architecture means that the sensitive data now flows continuously between all the microservices, posing additional risks of spoofing attacks on microservices. You can have application-specific solutions to hold certificates of other workloads like Java truststore but a lack of coherent solutions across all the different workloads can be a problem.

Ephemeral nature of underlying hardware in the cloud-native ecosystem.

When using Kubernetes or similar container orchestration tools, the underlying nodes are changed and replaced continuously, so you cannot keep whitelisting the IP addresses or instanceIDs as the nodes change. This is not feasible and not practical.

mTLS and SPIFFE

As we all know the modern internet works on TLS certificates. mTLS is using TLS certs to mutually authenticate each other. mTLS is a proven way of implementing Zero Trust Network. Here you can have CA which issues certs to all the required workloads. The workloads can authenticate each other using TLS certificates.

SPIFFE is

SPIFFE, the Secure Production Identity Framework for Everyone, is a set of open-source standards for securely identifying software systems in dynamic and heterogeneous environments. Systems that adopt SPIFFE can easily and reliably mutually authenticate wherever they are running.

SPIFFE sets the standard of how an ZTN can be implemented. It works by having a workload API running on all the nodes and issuing SVID(Spiffe verifiable Identity Document) to the workloads which can use to authenticate themselves. These documents are short-lived, constantly rotated and cryptographically signed making them more secure.

SPIRE is

SPIRE is a production-ready implementation of the SPIFFE APIs that performs node and workload attestation in order to securely issue SVIDs to workloads, and verify the SVIDs of other workloads, based on a predefined set of conditions.

The SPIFFE project has created the standards and also an implementation of these standards so that the companies can implement mTLS. It works similarly to Hashicorp Vault or Istio where there is a SPIRE Server which is centrally located and there are SPIRE Agents which run in all the nodes. This part of the docs details how this entire process works. SPIFFE and SPIRE are primarily created for Cloud Native workloads but can be worked with traditional VMs as well. There are multiple ways to set up SPIRE server and agent, the easiest one will be having both inside the same Kubernetes cluster.

Conclusion

Though SPIFFE is not the only way it is a compelling solution to approach Zero Trust Network using mTLS in CloudNative environments

Stable Features

• Volume manger is refactored to get more information about how volumes are mounted after kubelet restart

• Prevent unauthorized volume mode conversion during volume restore . From 1.30 only the SA that have right permissions can make changes while restoring snapshot to PV. More information here

• Using .spec.schedulingGates you can now control when the pod is declared to be ready. This reduces burden on Scheduler to place an unplacable pod in a node due to resource constraints or similar issues. You can find more about SchedulingGates here.

• You can now use minDomains parameter in PodTopologySpread constaints as it graduates. PodTopologySpread is an interesting feature to distribute pods across the cluster/regions which comes with a great constraints and requires significant learning. Read about it here.

• Kubernetes now uses Go workspaces. This does not effect end users but effects developers.

Beta Features

• You can now get logs from induvidual nodes using features in kubelet like NodeLogQuery, enableSystemLogHandler and enableSystemLogQuery. See more information here.

• CRD validation ratcheting using feature gate CRDValidationRatcheting .

• Contextual logging. Using this the developers can add contextual details to logs with WithValues and WithName.

• Make Kubernetes aware of the LoadBalancer behaviour. Read more here.

There are ton of alpha features that are being added to the Uwubernetes 😂. The strong developer community and wide variety of use cases making Kubernetes de-facto way for the next two decades.

For me HAProxy is reminiscent of the days when I am still figuring out my way around software world. I started my career while working with HAProxy load-balancer. Back then we used to use automated shell scripts to dynamically configure backends for load-balancing. Both of us have came long way since then 😀

Yet another Ingress Controller

In the eco-system over crowded with multiple Ingress controllers, HAProxy comes with an enterprise focused approach and innovative approaches like rootless containers and QUIC protocol. Coming from a legacy of LBing in VMs, all the existing features are extended to the Ingress Controller as well. Here is the list of all config map options that are available.

Installation

The HAProxy Ingress controller installation is straightforward. Here I have created a simple EKS cluster and installed HAProxy using Helm

## Cluster
➜  ~ k get nodes
NAME                                       STATUS   ROLES    AGE     VERSION
ip-10-0-1-221.us-east-2.compute.internal   Ready    <none>   2m15s   v1.27.9-eks-5e0fdde
ip-10-0-2-143.us-east-2.compute.internal   Ready    <none>   2m21s   v1.27.9-eks-5e0fdde

## HAProxy installation
➜  ~ helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
  --create-namespace \
  --namespace haproxy-controller
NAME: haproxy-kubernetes-ingress
LAST DEPLOYED: Tue Mar 26 16:31:10 2024
NAMESPACE: haproxy-controller
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
HAProxy Kubernetes Ingress Controller has been successfully installed.

Controller image deployed is: "haproxytech/kubernetes-ingress:1.11.2".
Your controller is of a "Deployment" kind. Your controller service is running as a "NodePort" type.
RBAC authorization is enabled.
Controller ingress.class is set to "haproxy" so make sure to use same annotation for
Ingress resource.

Service ports mapped are:
  - name: http
    containerPort: 8080
    protocol: TCP
  - name: https
    containerPort: 8443
    protocol: TCP
  - name: stat
    containerPort: 1024
    protocol: TCP
  - name: quic
    containerPort: 8443
    protocol: UDP

## Components Installed

➜  learn_haproxy_ingress k get all -n haproxy-controller                                         
NAME                                              READY   STATUS      RESTARTS   AGE
pod/haproxy-kubernetes-ingress-6b9d5f976c-bvqfd   1/1     Running     0          18s
pod/haproxy-kubernetes-ingress-6b9d5f976c-v2lb4   1/1     Running     0          18s
pod/haproxy-kubernetes-ingress-crdjob-1-qzr5p     0/1     Completed   0          18s

NAME                                 TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                  AGE
service/haproxy-kubernetes-ingress   NodePort   10.110.187.218   <none>        80:32563/TCP,443:30875/TCP,443:30875/UDP,1024:32400/TCP,6060:31395/TCP   18s

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/haproxy-kubernetes-ingress   2/2     2            2           18s

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/haproxy-kubernetes-ingress-6b9d5f976c   2         2         2       18s

Testing with sample application

On a side note, I am trying the ingress controller in minikube so Ingress runs as a nodePort service. I have to portforward these ports using command

➜  learn_haproxy_ingress minikube service haproxy-kubernetes-ingress -n haproxy-controller --url
http://127.0.0.1:50314
http://127.0.0.1:50315
http://127.0.0.1:50316
http://127.0.0.1:50317
http://127.0.0.1:50318

Now you have to figure out which host port forwards to which service port in ingress. I found 50317 is HTTP(80) and 50315 is HTTPS(443). If you are trying this out, it may be same for you or might be different

For trying the Ingress, I have followed their book "HAProxy in Kubernetes - Supercharge Your Ingress Routing".

I have deployed sample application using below definition

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: app
  name: app
spec:
  replicas: 2
  selector:
    matchLabels:
      run: app
  template:
    metadata:
      labels:
        run: app
    spec:
      containers:
      - name: app
        image: jmalloc/echo-server
        ports:
        - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: app
  name: app
  annotations:
    haproxy.org/check: "enabled"
    haproxy.org/forwarded-for: "enabled"
    haproxy.org/load-balance: "roundrobin"
spec:
  selector:
    run: app
  ports:
  - name: port-1
    port: 80
    protocol: TCP
    targetPort: 8080

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: default
spec:
  rules:
  - host: foo.bar
  - http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: app
              port: 
                number: 80

Features

HAProxy Ingress Controller comes with all the features HAProxy has. It also has support to create TLS certs on the go for induvidual services as required.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to
use
    cert-manager.io/cluster-issuer:
letsencrypt-staging
  name: mysite-ingress
  namespace: default
spec: rules:
  - host: mysite.com
    http:
      paths:
      - path: /
        backend:
          serviceName: mysite-service
          servicePort: 80
  tls:
  - secretName: mysite-cert
    hosts:
    - mysite.com

It will allow you to define backend, global and default groups to have fine-grained access controls like setting up algorithm keep-alive and forwardfor etc.

Conclusion

HAProxy ingress is a compelling choice for the companies looking for enterprise supported tooling for safety and compliance and for companies who already uses HAProxy LB for their legacy systems.