Information Technology plays a major role in today’s world to provide better solutions for the day to day problems faced by the different sectors in the market. Time is the most vital component of any process which should be monitored and managed in order to provide timely solutions. Continues improvement is a must and the companies strive towards scalable and fault-tolerant distributed systems. Proper QA processes should be in place to ensure a bug free application.
Software developers and solution providers have been working on innovative ways to automate the testing process to provide efficient solutions. The automated solutions require more resources to process. Specially, the tests depend on browsers.
If you or your team facing these issues now, please checkout the below solution which may have an answer to your current problems.
What is a virtual node?
Azure Kubernetes virtual node allow Azure Kubernetes Service to elastically provision additional pods without deploying VM computer nodes. Virtual node provisions Pods using Azure Container Instances that start in seconds.
https://docs.microsoft.com/en-us/azure/aks/virtual-nodes-portal
Why Selenium Grid on AKS and Virtual node?
There are so many ways to setup Selenium Grid, on the cloud. But if you already have a Azure Kubernetes cluster, why not run your tests on Azure Kubernetes Service. When you use virtual node, you only pay for execution time. On top of that you can scale up and scale down depending on demand using Horizontal Pod Autoscaler.
Virtual node run Pods on Azure Container Instances (ACI), so you don’t need to wait for the Kubernetes cluster autoscaler to deploy VM compute node to run the additional pods.
Let’s dive into setting up.
Prerequisites
- Lots of coffee
- Git / VS Code
- An Azure account with a subscription
- Azure CLI installed and configured
- kubectl: Kubernetes command line tool
Creating AKS with virtual nodes
Please refer to the following link to setup an AKS with virtual nodes.
https://docs.microsoft.com/en-us/azure/aks/virtual-nodes-portal
This might take several minutes. Once the cluster is created, you can connect to it using your preferred editor or terminal.
Create Hub and Nodes
Once you’ve connected to the cluster, create a Selenium Hub by running the following command.
You can download all yaml files from https://github.com/nishanperera/Selenium-Grid-on-AKS-With-Virtual-Nodes
kubectl apply -f https://github.com/nishanperera/Selenium-Grid-on-AKS-With-Virtual-Nodes/blob/master/hub-AKS-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: selenium-hub
spec:
selector:
matchLabels:
app: selenium-hub
template:
metadata:
labels:
app: selenium-hub
spec:
containers:
- name: selenium-hub
image: selenium/hub:3.141
resources:
limits:
memory: "128Mi"
cpu: ".5"
ports:
- containerPort: 4444
livenessProbe:
httpGet:
path: /wd/hub/status
port: 4444
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /grid/console
port: 4444
initialDelaySeconds: 30
timeoutSeconds: 5
nodeSelector:
kubernetes.io/role: agent
beta.kubernetes.io/os: linux
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
- key: azure.com/aci
effect: NoSchedule
Run following to see hub pod
kubectl get podsNext, you will need to create the service, so external traffic can be sent the hub.
Kubectl apply -f https://github.com/nishanperera/Selenium-Grid-on-AKS-With-Virtual-Nodes/blob/master/hub-AKS-service.yaml
apiVersion: v1
kind: Service
metadata:
name: selenium-svc
labels:
app: selenium-svc
spec:
selector:
app: selenium-hub
ports:
- port: 4444
targetPort: 4444
type: LoadBalancer
View the service details by running the following command.
kubectl get service selenium-svc
you can view the hub by browsing external-IP with port 4444
http://external-ip:4444/wd/hub/status
you should see following page
Next, we deploy the chrome node to azure ACI. Before you run the yaml file Please replace the Selenium hub IP on yaml file to your hub IP.
Kubectl apply -f https://github.com/nishanperera/Selenium-Grid-on-AKS-With-Virtual-Nodes/blob/master/chrome-node-AKS-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: chrome-node-rc
labels:
app: chrome-node-rc
spec:
replicas: 1
selector:
matchLabels:
app: chrome-node-rc
template:
metadata:
name: chrome-node-rc
labels:
app: chrome-node-rc
spec:
volumes:
- name: dshm
emptyDir:
medium: Memory
containers:
- name: chrome-node-rc
image: selenium/node-chrome-debug:3.141
ports:
- containerPort: 5555
volumeMounts:
- mountPath: /dev/shm
name: dshm
env:
- name: HUB_HOST
value: "10.0.27.181"
- name: HUB_PORT
value: "4444"
resources:
limits:
memory: "1000Mi"
cpu: "500m"
nodeSelector:
kubernetes.io/role: agent
beta.kubernetes.io/os: linux
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
- key: azure.com/aci
effect: NoSchedule
Scaling in action
AKS supports both manual and auto scaling. but for this you will be using auto scaling with HPA (Horizontal Pod Autoscaler)
Run following to setup HPA, notice triggered metric here is cpu-precent. when average of all pods’ cpu-percentage more than 40%, AKS will create a new pod in ACI and schedule it to run.
kubectl autoscale deployment chrome-node-rc --cpu-percent=40 --min=1 --max=5If you are not running any tests, you should only see hub and one chrome node.
Let’s send some traffic to Selenium Grid by running C# tests.
You can check HPA by running following
After few minutes, you will see replicas will increase and decrease.
Known limitations on virtual node
Can’t mount config map to Virtual Kubelet https://github.com/virtual-kubelet/virtual-kubelet/issues/523
References
Horizontal Pod Autoscaler https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
Scaling options for applications in Azure Kubernetes Service (AKS)https://docs.microsoft.com/en-us/azure/aks/concepts-scale
Ministry of Devs 