Using K8s Self Hosted Runners for GitHub Actions

Photo by Richy Great on Unsplash

Using K8s Self Hosted Runners for GitHub Actions

ยท

3 min read

As the popularity for GitHub Actions for CI increasing these days, so is the the bill for GitHub hosted Runners. As this article explains, the typical costs for GHR(GitHub hosted Runners) is 23x of AWS spot instances for the same compute. As it is comparatively easy๐Ÿ˜œ to setup a managed (or self-managed) K8s cluster with Karpenter and spot-instances nodepool setup for GHA than to use GHR. Moreover if you have to have complete control over the CI process for compliance purposes. It is a no brainer to use SHR.

In order to setup K8s cluster as SH environment, you need to have Action Runner Controller installed. It is a K8s operator to manage SHR lifecycle. It comes with its own CRDs.

ARC Installation

In order to install ARC you need to figure out authentication method first. I have used GitHubApp approach. So you will create an app and give required permissions and add the App to required repositories.

ARC needs to be authenticated with GitHub. In order to do that you need to create secret with App details so ARC can pick these up during installation.

โžœ  k8s-gha k get secrets -n actions controller-manager -o json |jq '.data | keys'
[
  "github_app_id",
  "github_app_installation_id",
  "github_app_private_key"
]

Now you can install ARC in the cluster using Helm

โžœ  ~ helm install actions-runner-controller actions-runner-controller/actions-runner-controller --namespace=actions --version=0.22.0 -f runner-controller.yaml
NAME: actions-runner-controller
LAST DEPLOYED: Sun Jul 14 16:36:39 2024
NAMESPACE: actions
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace actions -l "app.kubernetes.io/name=actions-runner-controller,app.kubernetes.io/instance=actions-runner-controller" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace actions $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace actions port-forward $POD_NAME 8080:$CONTAINER_PORT

Here is how the runner-controller file looks like

replicaCount: 1
webhookPort: 9443
syncPeriod: 1m
defaultScaleDownDelay: 5m
enableLeaderElection: true

authSecret:
  enabled: true
  create: false
  name: "controller-manager"

image:
  repository: "summerwind/actions-runner-controller"
  actionsRunnerRepositoryAndTag: "summerwind/actions-runner:ubuntu-20.04"
  dindSidecarRepositoryAndTag: "docker:dind"
  pullPolicy: IfNotPresent

serviceAccount:
  create: true

service:
  type: ClusterIP
  port: 443

certManagerEnabled: true

logFormat: text

githubWebhookServer:
  enabled: false

Now you are all set-up ๐ŸŽ‰.

Below are the CRDs that comes with ARC

โžœ  k8s-gha k get crds |grep summer
horizontalrunnerautoscalers.actions.summerwind.dev    2024-07-14T21:36:26Z
runnerdeployments.actions.summerwind.dev              2024-07-14T21:36:27Z
runnerreplicasets.actions.summerwind.dev              2024-07-14T21:36:28Z
runners.actions.summerwind.dev                        2024-07-14T21:36:29Z
runnersets.actions.summerwind.dev                     2024-07-14T21:36:32Z

This follows the same pattern of K8s workloads where runners are equivalent to Pods and runnerdeployments are Deployments. You need to create a runnerdeployment with the image name, labels so GHA can pick up runners from this deployment based on the label

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    karpenter.sh/do-not-evict: "true"
  name: self-hosted-runner-deployment
  namespace: actions
spec:
  template:
    spec:
      repository: HighonAces/actions-1
      image: summerwind/actions-runner:ubuntu-20.04
      resources:
        requests:
          cpu: 1500m
          memory: 2000Mi

You also need to create HorizontalRunnerAutoscaler to scale this deployment.

apiVersion: v1
items:
- apiVersion: actions.summerwind.dev/v1alpha1
  kind: HorizontalRunnerAutoscaler
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"actions.summerwind.dev/v1alpha1","kind":"HorizontalRunnerAutoscaler","metadata":{"annotations":{},"name":"self-hosted-runner-deployment-autoscaler","namespace":"actions"},"spec":{"maxReplicas":30,"minReplicas":0,"scaleTargetRef":{"kind":"RunnerDeployment","name":"self-hosted-runner-deployment"},"scaleUpTriggers":[{"duration":"30m","githubEvent":{"workflowJob":{}}}]}}
    creationTimestamp: "2024-07-14T21:52:23Z"
    generation: 2
    name: self-hosted-runner-deployment-autoscaler
    namespace: actions
    resourceVersion: "9689"
    uid: 41291b4a-cbd8-4af2-bfe3-02c2b29d8262
  spec:
    maxReplicas: 30
    minReplicas: 2
    scaleTargetRef:
      kind: RunnerDeployment
      name: self-hosted-runner-deployment
    scaleUpTriggers:
    - duration: 30m
      githubEvent:
        workflowJob: {}
  status:
    desiredReplicas: 2
    lastSuccessfulScaleOutTime: "2024-07-14T22:28:46Z"
kind: List
metadata:
  resourceVersion: ""

This continuously keeps 2 replicas running as I have not given any metrics definition. By making use of PercentageRunnersBusy, you can scaleup or scaledown. Here is the metrics definition from ARC documentation

  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.25'
    scaleUpFactor: '2'
    scaleDownFactor: '0.5'

Once you have min number of runners running you can see the corresponding pods in cluster and same in GH runners page.

These runners will pickup the jobs whenever you have actions configured with this parameter runs-on: self-hosted-linux.

Conclusion

This setup is rather easy and customizable. It is a very good alternative for the GHR and I cannot wait to see what future holds for K8s based Self hosted runners.

Source: https://medium.com/simform-engineering/how-to-setup-self-hosted-github-action-runner-on-kubernetes-c8825ccbb63c

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller

https://github.com/actions/actions-runner-controller

Did you find this article valuable?

Support Srujan Reddy by becoming a sponsor. Any amount is appreciated!

ย