Understanding KEDA

What is KEDA?

KEDA is Kubernetes-based Event Driven Autoscaler. It can be used to scale any container based resource like Deployments, StatefulSets etc whenever an event occurs and then scale back to zero when it is not necessary or it can be used to create Kubernetes Jobs during events. It can be used to scale both Horizontally and Vertically.

Need for KEDA

KEDA is an important player to maximise your usage efficiency. You simply cannot keep your deployments running in the hope of traffic/event occurs. You must be able to create resources at the minute of need and must be able to terminate them as soon as their purpose is served. This is especially useful for Jobs. KEDA can ingest metrics from any sources(currently 68 scalers available). Here are some of the examples.

  • Vertical scaling of pods based on avg resource usage metrics from Prometheus or Datadog.

  • Scaling based on ALB metrics.

  • Scaling jobs based on Kafka or ActiveMQ queues.

  • Scaling based on query results from databases like MongoDb, DynamoDb and MySQL etc.

  • Scaling Github runners based number of jobs pending in GHA.

These are some of the examples but you can use any event to scale your resources.

KEDA Architecture

Lets first see all the CRDs that KEDA offers

➜  keda k get crds |grep keda
cloudeventsources.eventing.keda.sh                    2024-07-12T02:20:03Z
clustertriggerauthentications.keda.sh                 2024-07-12T02:20:03Z
scaledjobs.keda.sh                                    2024-07-12T02:20:03Z
scaledobjects.keda.sh                                 2024-07-12T02:20:03Z
triggerauthentications.keda.sh                        2024-07-12T02:20:03Z

ScaledObjects

ScaledObjects defines what resource you want to scale and based on what triggers. The resource would be typically Deployment or StatefulSet.

spec:
  scaleTargetRef:
    apiVersion:    {api-version-of-target-resource}         # Optional. Default: apps/v1
    kind:          {kind-of-target-resource}                # Optional. Default: Deployment
    name:          {name-of-target-resource}                # Mandatory. Must be in the same namespace as the ScaledObject

Triggers is the external source based on which you want to scale the said object. ScaledJobs are similar to ScaledObjects. The only difference is you create/scale Jobs based on the external source.

TriggerAuthentication

As you define external triggers, you also need a way to authenticate to that source. For this purpose you need TriggerAuthentication. ClusterTriggerAuthentication is a similar one with cluster-scope.

CloudEventSource

CloudEventSource resource can be used in KEDA for subscribing to events that are emitted to the user’s defined CloudEvent sink.

Here's how all works together. ScaledObject first checks if TriggerAuthentication works or not and if could able to get required metrics from the external source. Also checks if the object that it needs to be scaled is defined or not. Then it will create a Horizontal Pod Autoscaler(HPA) for the object and starts passing metrics to HPA based on the data from external source.

Demo: Using MongoDB to scale Deployment

Here I am trying to scale deployment based on MongoDb query results. I have created a deployment for nginx with one replica, pretty standard

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx-dep
  name: nginx-dep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-dep
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-dep
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}

My target is to scaledown this deployment to zero when my mongodb document has "enabled" value is set as "no" and scales it whenever the value is "yes". So my scaled object definition looks something like

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: nginx-so
  namespace: default
spec:
  scaleTargetRef:
    name: nginx-dep
  cooldownPeriod:  30
  minReplicaCount: 0
  maxReplicaCount: 5

  triggers:
  - type: mongodb
    metadata:
      dbName: "drinks"
      collection: "softdrinks"
      query: '{ "enabled": "yes" }'
      queryValue: "1"
    authenticationRef: 
      name: mongodb-trigger
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata: 
  name: mongodb-trigger
spec: 
  secretTargetRef:
    - parameter: connectionString
      name: mongodb-secret
      key: connect

You need to create mongodb secret with the key connect and value should be mongodb connection string mongodb+srv://:@******.*****.mongodb.net/<dbname>

apiVersion: v1
kind: Secret
metadata: 
  name: mongodb-secret
type: Opaque
data: 
  connect: bW9uZ29kYitzcnY6Ly86QCoqKioqKi4qKioqKi5tb25nb2RiLm5ldC8K==

Now we have everything in place. Once you create ScaledObject and triggerauthentications, SO will create a HPA and converts the query metric to HPA understandable metric.

Here I have changed value {"enabled": "no"}. Now when you describe ScaledObject you will see

➜  keda k get so nginx-so                
NAME       SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS   AUTHENTICATION    READY   ACTIVE   FALLBACK   PAUSED    AGE
nginx-so   apps/v1.Deployment   nginx-dep         0     5     mongodb    mongodb-trigger   True    False    False      Unknown   28m
Name:         nginx-so
Namespace:    default
Labels:       scaledobject.keda.sh/name=nginx-so
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2024-07-13T06:44:57Z
  Finalizers:
    finalizer.keda.sh
  Generation:        1
  Resource Version:  144637
  UID:               9b385df8-88a4-4d0a-bc68-4e7ffea7f8f7
Spec:
  Cooldown Period:    30
  Max Replica Count:  5
  Min Replica Count:  0
  Scale Target Ref:
    Name:  nginx-dep
  Triggers:
    Authentication Ref:
      Name:  mongodb-trigger
    Metadata:
      Collection:   softdrinks
      Db Name:      drinks
      Query:        { "enabled": "yes" }
      Query Value:  1
    Type:           mongodb
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is not performed because triggers are not active
    Reason:   ScalerNotActive
    Status:   False
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
    Status:   Unknown
    Type:     Paused
  External Metric Names:
    s0-mongodb-softdrinks
  Health:
    s0-mongodb-softdrinks:
      Number Of Failures:  0
      Status:              Happy
  Hpa Name:                keda-hpa-nginx-so
  Last Active Time:        2024-07-13T07:01:28Z
  Original Replica Count:  0
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason                      Age   From           Message
  ----     ------                      ----  ----           -------
  Warning  KEDAScalerFailed            29m   keda-operator  failed to parsing mongoDB metadata, because of missing required field in scaler config: no host given
  Warning  ScaledObjectCheckFailed     29m   keda-operator  failed to ensure HPA is correctly created for ScaledObject
  Normal   KEDAScalersStarted          29m   keda-operator  Scaler mongodb is built.
  Normal   KEDAScalersStarted          29m   keda-operator  Started scalers watch
  Normal   ScaledObjectReady           29m   keda-operator  ScaledObject is ready for scaling
  Normal   KEDAScaleTargetActivated    28m   keda-operator  Scaled apps/v1.Deployment default/nginx-dep from 0 to 1, triggered by mongoDBScaler
  Normal   KEDAScaleTargetDeactivated  12m   keda-operator  Deactivated apps/v1.Deployment default/nginx-dep from 1 to 0

The statements
Message: Scaling is not performed because triggers are not active. and Normal KEDAScaleTargetDeactivated 12m keda-operator Deactivated apps/v1.Deployment default/nginx-dep from 1 to 0 shows that the trigger is not active so scaled to zero.

HPA reflects the same

➜  keda k get hpa
NAME                REFERENCE              TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-nginx-so   Deployment/nginx-dep   <unknown>/1 (avg)   1         5         0          32m
➜  keda k describe hpa keda-hpa-nginx-so 
Name:                                              keda-hpa-nginx-so
Namespace:                                         default
Labels:                                            app.kubernetes.io/managed-by=keda-operator
                                                   app.kubernetes.io/name=keda-hpa-nginx-so
                                                   app.kubernetes.io/part-of=nginx-so
                                                   app.kubernetes.io/version=2.14.0
                                                   scaledobject.keda.sh/name=nginx-so
Annotations:                                       <none>
CreationTimestamp:                                 Sat, 13 Jul 2024 01:44:58 -0500
Reference:                                         Deployment/nginx-dep
Metrics:                                           ( current / target )
  "s0-mongodb-softdrinks" (target average value):  <unknown> / 1
Min replicas:                                      1
Max replicas:                                      5
Deployment pods:                                   0 current / 0 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededGetScale   the HPA controller was able to get the target's current scale
  ScalingActive   False   ScalingDisabled     scaling is disabled since the replica count of the target is zero
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range

Now if I change enabled to yes

Name:         nginx-so
Namespace:    default
Labels:       scaledobject.keda.sh/name=nginx-so
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2024-07-13T06:44:57Z
  Finalizers:
    finalizer.keda.sh
  Generation:        1
  Resource Version:  146171
  UID:               9b385df8-88a4-4d0a-bc68-4e7ffea7f8f7
Spec:
  Cooldown Period:    30
  Max Replica Count:  5
  Min Replica Count:  0
  Scale Target Ref:
    Name:  nginx-dep
  Triggers:
    Authentication Ref:
      Name:  mongodb-trigger
    Metadata:
      Collection:   softdrinks
      Db Name:      drinks
      Query:        { "enabled": "yes" }
      Query Value:  1
    Type:           mongodb
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is performed because triggers are active
    Reason:   ScalerActive
    Status:   True
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
    Status:   Unknown
    Type:     Paused
  External Metric Names:
    s0-mongodb-softdrinks
  Health:
    s0-mongodb-softdrinks:
      Number Of Failures:  0
      Status:              Happy
  Hpa Name:                keda-hpa-nginx-so
  Last Active Time:        2024-07-13T07:19:58Z
  Original Replica Count:  0
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason                      Age                From           Message
  ----     ------                      ----               ----           -------
  Warning  KEDAScalerFailed            35m                keda-operator  failed to parsing mongoDB metadata, because of missing required field in scaler config: no host given
  Warning  ScaledObjectCheckFailed     35m                keda-operator  failed to ensure HPA is correctly created for ScaledObject
  Normal   KEDAScalersStarted          35m                keda-operator  Scaler mongodb is built.
  Normal   KEDAScalersStarted          35m                keda-operator  Started scalers watch
  Normal   ScaledObjectReady           35m                keda-operator  ScaledObject is ready for scaling
  Normal   KEDAScaleTargetDeactivated  18m                keda-operator  Deactivated apps/v1.Deployment default/nginx-dep from 1 to 0
  Normal   KEDAScaleTargetActivated    34s (x2 over 33m)  keda-operator  Scaled apps/v1.Deployment default/nginx-dep from 0 to 1, triggered by mongoDBScaler

and HPA looks like

Name:                                              keda-hpa-nginx-so
Namespace:                                         default
Labels:                                            app.kubernetes.io/managed-by=keda-operator
                                                   app.kubernetes.io/name=keda-hpa-nginx-so
                                                   app.kubernetes.io/part-of=nginx-so
                                                   app.kubernetes.io/version=2.14.0
                                                   scaledobject.keda.sh/name=nginx-so
Annotations:                                       <none>
CreationTimestamp:                                 Sat, 13 Jul 2024 01:44:58 -0500
Reference:                                         Deployment/nginx-dep
Metrics:                                           ( current / target )
  "s0-mongodb-softdrinks" (target average value):  1 / 1
Min replicas:                                      1
Max replicas:                                      5
Deployment pods:                                   1 current / 1 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from external metric s0-mongodb-softdrinks(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: nginx-so,},MatchExpressions:[]LabelSelectorRequirement{},})
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

Source:

https://devtron.ai/blog/introduction-to-kubernetes-event-driven-autoscaling-keda/#keda-architecture-and-components

https://medium.com/@mohammadsaquib.ee/mongodb-powered-autoscaling-harnessing-keda-to-scale-applications-dynamically-based-on-database-f38a68e71db6

https://keda.sh/docs/2.14/scalers/mongodb/

Did you find this article valuable?

Support Srujan Reddy by becoming a sponsor. Any amount is appreciated!