In this article, we will see how to solve the problem of HPA and KEDA terminating manually scaled up pods in Kubernetes. But before going through the solution, let's briefly understand about HPA and KEDA concepts. HPA or Horizonal Pod Autoscaler is basically used to scales the number of Pods in a deployment, replication controller, replica set, or statefulset based on CPU usage or other custom metrics usage such as memory, request rate, etc and KEDA, also known as Kubernetes Event-Driven Autoscaler is more like a framework that allows Kubernetes workloads such as Deployments or Jobs to scale based on event driven metrics and not just CPU or memory like the default Kubernetes HPA (Horizontal Pod Autoscaler).

KEDA works hand-in-hand with Kubernetes HPA by using ScaledObjects to provide custom, event-driven metrics for autoscaling. While KEDA works well in ideal scenarios to manage resources efficiently in a Kubernetes cluster, it can sometimes cause issues when there is a need to manually scale up the number of pods due to specific requirements. It will immediately terminate the scaled up pods if the event condition has not been fulfilled. This prevents us from increasing the capacity manually whenever required. To get around this problem, we can perform some quick and easy fix without taking down the KEDA. Let me explain the solution by giving an example from a scenario I faced a few days ago.

 

Solved "HPA and KEDA(ScaledObject) terminating manually scaled up pods in Kubernetes"

Also Read: How to Install KubeRay Operator in a Kubernetes Cluster Using Helm

I was having a deployment called cyberithub-example-app running in my Kubernetes cluster and due to some task I had to scale up the capacity. It was running with one replica and I needed to scale it up to 4 replicas. So I tried scaling up replica to 4 from 1 as shown below:-

cyberithub@ubuntu:~$ kubectl scale --replicas=4 deployment cyberithub-example-app
deployment.apps/cyberithub-example-app scaled

It was found to be starting another 3 replicas as you can see below.

cyberithub@ubuntu:~$ kubectl get pods
NAME                                        READY  STATUS   RESTARTS  AGE
cyberithub-example-app-8c291376b1-7f41k     0/1    Running  0         34s
cyberithub-example-app-8c291376b1-3gefk     0/1    Running  0         34s
cyberithub-example-app-8c291376b1-nsf42     0/1    Running  0         34s
cyberithub-example-app-8c291376b1-rt2ak     1/1    Running  0         3m40s

But then after, I suddenly noticed all scaled up replicas started auto terminating as you can see below.

cyberithub@ubuntu:~$ kubectl get pods
NAME                                      READY  STATUS        RESTARTS  AGE
cyberithub-example-app-8c291376b1-7f41k   0/1    Terminating   0         34s
cyberithub-example-app-8c291376b1-3gefk   0/1    Terminating   0         34s
cyberithub-example-app-8c291376b1-nsf42   0/1    Terminating   0         34s
cyberithub-example-app-8c291376b1-rt2ak   1/1    Running       0         3m40s

To investigate this problem, I checked the events and noticed that HPA and KEDA are terminating the scaled up pods.

cyberithub@ubuntu:~$ kubectl get events
LAST SEEN      TYPE      REASON                     OBJECT                                                     MESSAGE
...............................
119s           Normal    ScalingReplicaSet          deployment/cyberithub-example-app                          Scaled up replica set cyberithub-example-app-8c291376b1 to 4 from 1 
74s            Normal    ScalingReplicaSet          deployment/cyberithub-example-app                          Scaled down replica set cyberithub-example-app-8c291376b1 to 1 from 4
75s            Normal    SuccessfulRescale          horizontalpodautoscaler/keda-hpa-cyberithub-example-app    New Size: 1; reason: All metrics below target
15m            Warning   FailedGetResourceMetric    horizontalpodautoscaler/keda-hpa-cyberithub-example-app    failed to get cpu utilization: unable to get metrics for resource cpu: no 
                                                                                                               metrics returned from resource metrics API
...............................

Just to let you know in our case, scaled objects for deployment cyberithub-example-app looks like below where minimum replica configured to be 1 and maximum configured to be 4.

cyberithub@ubuntu:~$ kubectl get so
NAME                      SCALETARGETKIND         SCALETARGETNAME          MIN  MAX  TRIGGERS  AUTHENTICATION  READY  ACTIVE  FALLBACK  PAUSED  AGE
cyberithub-example-app    apps/v1.Deployment      cyberithub-example-app   1    4    cpu                       True   True    Unknown   Unknown 30d

Similarly, our hpa configuration for deployment cyberithub-example-app looks like below where minimum number of pods configured to be running at all times is 1 and maximum number pods it can run up to 4.

cyberithub@ubuntu:~$ kubectl get hpa
NAME                              REFERENCE                            TARGETS         MINPODS  MAXPODS  REPLICAS  AGE
keda-hpa-cyberithub-example-app   Deployment/cyberithub-example-app    24%/80%         1        4        1         30d

So, in order to fix this issue what we can do is that we can set minimum and maximum pod replica to HPA and KEDA to 4 so that it will run with atleast 4 replicas. To edit the configuration of scaled object, run kubectl edit scaledobject cyberithub-example-app command and edit minReplicaCount to 4.

cyberithub@ubuntu:~$ kubectl edit scaledobject cyberithub-example-app
scaledobject.keda.sh/cyberithub-example-app edited

To verify the change, run kubectl get so command again and check MIN value as shown below.

cyberithub@ubuntu:~$ kubectl get so 
NAME                    SCALETARGETKIND     SCALETARGETNAME         MIN    MAX  TRIGGERS AUTHENTICATION  READY ACTIVE FALLBACK PAUSED   AGE 
cyberithub-example-app  apps/v1.Deployment  cyberithub-example-app  4      4    cpu                      True  True   Unknown  Unknown  30d

Similarly, to edit the configuration of hpa keda-hpa-cyberithub-example-app, run kubectl edit hpa cyberithub-example-app command and edit minReplicas to 4. In case if minReplicas needed to increased above 4, let's say to 5 then in that case maxReplicas also needs to increased to 5 as maxReplicas cannot be lesser than minReplicas. If you try to increase minReplicas beyond maxReplicas capacity, it will throw an error about the same. It is also true in case of HPA. So you have to be careful in increasing the number of replicas.

cyberithub@ubuntu:~$ kubectl edit hpa cyberithub-example-app
horizontalpodautoscaler.autoscaling/keda-hpa-cyberithub-example-app edited

To verify the change, run kubectl get hpa command again and check MINPODS and REPLICAS value as shown below.

cyberithub@ubuntu:~$ kubectl get hpa 
NAME                              REFERENCE                          TARGETS   MINPODS  MAXPODS  REPLICAS AGE 
keda-hpa-cyberithub-example-app   Deployment/cyberithub-example-app  24%/80%   4        4        4        30d

Now if you try to scale up the capacity again then this time you will notice that the scaled up pods are not terminating anymore. It is running just fine. This solved my issue without any problem. Hope the same works for you. But please remember to rollback the changes once your work is done so that KEDA can work effectively.