As the popularity for GitHub Actions for CI increasing these days, so is the the bill for GitHub hosted Runners. As this article explains, the typical costs for GHR(GitHub hosted Runners) is 23x of AWS spot instances for the same compute. As it is comparatively easy😜 to setup a managed (or self-managed) K8s cluster with Karpenter and spot-instances nodepool setup for GHA than to use GHR. Moreover if you have to have complete control over the CI process for compliance purposes. It is a no brainer to use SHR.
In order to setup K8s cluster as SH environment, you need to have Action Runner Controller installed. It is a K8s operator to manage SHR lifecycle. It comes with its own CRDs.
ARC Installation
In order to install ARC you need to figure out authentication method first. I have used GitHubApp approach. So you will create an app and give required permissions and add the App to required repositories.
ARC needs to be authenticated with GitHub. In order to do that you need to create secret with App details so ARC can pick these up during installation.
➜ k8s-gha k get secrets -n actions controller-manager -o json |jq '.data | keys'
[
"github_app_id",
"github_app_installation_id",
"github_app_private_key"
]
Now you can install ARC in the cluster using Helm
➜ ~ helm install actions-runner-controller actions-runner-controller/actions-runner-controller --namespace=actions --version=0.22.0 -f runner-controller.yaml
NAME: actions-runner-controller
LAST DEPLOYED: Sun Jul 14 16:36:39 2024
NAMESPACE: actions
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace actions -l "app.kubernetes.io/name=actions-runner-controller,app.kubernetes.io/instance=actions-runner-controller" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace actions $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace actions port-forward $POD_NAME 8080:$CONTAINER_PORT
Here is how the runner-controller file looks like
replicaCount: 1
webhookPort: 9443
syncPeriod: 1m
defaultScaleDownDelay: 5m
enableLeaderElection: true
authSecret:
enabled: true
create: false
name: "controller-manager"
image:
repository: "summerwind/actions-runner-controller"
actionsRunnerRepositoryAndTag: "summerwind/actions-runner:ubuntu-20.04"
dindSidecarRepositoryAndTag: "docker:dind"
pullPolicy: IfNotPresent
serviceAccount:
create: true
service:
type: ClusterIP
port: 443
certManagerEnabled: true
logFormat: text
githubWebhookServer:
enabled: false
Now you are all set-up 🎉.
Below are the CRDs that comes with ARC
➜ k8s-gha k get crds |grep summer
horizontalrunnerautoscalers.actions.summerwind.dev 2024-07-14T21:36:26Z
runnerdeployments.actions.summerwind.dev 2024-07-14T21:36:27Z
runnerreplicasets.actions.summerwind.dev 2024-07-14T21:36:28Z
runners.actions.summerwind.dev 2024-07-14T21:36:29Z
runnersets.actions.summerwind.dev 2024-07-14T21:36:32Z
This follows the same pattern of K8s workloads where runners are equivalent to Pods and runnerdeployments are Deployments. You need to create a runnerdeployment with the image name, labels so GHA can pick up runners from this deployment based on the label
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
annotations:
karpenter.sh/do-not-evict: "true"
name: self-hosted-runner-deployment
namespace: actions
spec:
template:
spec:
repository: HighonAces/actions-1
image: summerwind/actions-runner:ubuntu-20.04
resources:
requests:
cpu: 1500m
memory: 2000Mi
You also need to create HorizontalRunnerAutoscaler to scale this deployment.
apiVersion: v1
items:
- apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"actions.summerwind.dev/v1alpha1","kind":"HorizontalRunnerAutoscaler","metadata":{"annotations":{},"name":"self-hosted-runner-deployment-autoscaler","namespace":"actions"},"spec":{"maxReplicas":30,"minReplicas":0,"scaleTargetRef":{"kind":"RunnerDeployment","name":"self-hosted-runner-deployment"},"scaleUpTriggers":[{"duration":"30m","githubEvent":{"workflowJob":{}}}]}}
creationTimestamp: "2024-07-14T21:52:23Z"
generation: 2
name: self-hosted-runner-deployment-autoscaler
namespace: actions
resourceVersion: "9689"
uid: 41291b4a-cbd8-4af2-bfe3-02c2b29d8262
spec:
maxReplicas: 30
minReplicas: 2
scaleTargetRef:
kind: RunnerDeployment
name: self-hosted-runner-deployment
scaleUpTriggers:
- duration: 30m
githubEvent:
workflowJob: {}
status:
desiredReplicas: 2
lastSuccessfulScaleOutTime: "2024-07-14T22:28:46Z"
kind: List
metadata:
resourceVersion: ""
This continuously keeps 2 replicas running as I have not given any metrics definition. By making use of PercentageRunnersBusy, you can scaleup or scaledown. Here is the metrics definition from ARC documentation
- type: PercentageRunnersBusy
scaleUpThreshold: '0.75'
scaleDownThreshold: '0.25'
scaleUpFactor: '2'
scaleDownFactor: '0.5'
Once you have min number of runners running you can see the corresponding pods in cluster and same in GH runners page.
These runners will pickup the jobs whenever you have actions configured with this parameter runs-on: self-hosted-linux.
Conclusion
This setup is rather easy and customizable. It is a very good alternative for the GHR and I cannot wait to see what future holds for K8s based Self hosted runners.