In this article, we will see how to deal with a situation where you will see new pods are getting launched in Kubernetes even though replica is set to 1. Although this issue can happen due to many reasons but most of the time it usually occurs  when you use emptyDir volume in your kubernetes objects such as Pod, Deployment, DaemonSet or Statefulset hitting the maximum assigned sizeLimit. As you might be aware, emptyDir volume are usually employed when there is a need of temporary storage in a container to store temporary files, cache files, sharing data between containers etc. It can be easily provisioned by developers and programmers and can be used for high I/O workloads.

It is usually preferred in a place where size requirement is not much and the termination of volume does not cause any disruption. Since this volume writes to the underlying host, if the disk gets full it might impact other workloads running on the host hence the size is usually kept low. However it is important to note that the usage of emptyDir volume is strictly prohibited where there is need of persistent storage such as in databases and in all applications which requires persistent volume claim(PVC).

In my case, I am using emptyDir volume to store temporary log files which does not needs to be persisted. But unfortunately, suddenly one day I noticed that my pod running with emptyDir volume started showing in ContainerStatusUnknown status and it keeps on launching new pod after certain interval of time.

Solved "New pods keep on launching in kubernetes even though the replica is set to 1"

Also Read: How to republish existing data using Apache Kafka Connect

When I checked the status of pod by running kubectl get pods command, I noticed below output where multiple replicas were launched with all of them showing in ContainerStatusUnknown.

cyberithub@ubuntu:~$ kubectl get pods
NAME                           READY   STATUS                  RESTARTS AGE
example-app-87cb43c871-czst7   0/1     Completed               0        16h
example-app-87cb43c871-czst7   0/1     ContainerStatusUnknown  1        19h
example-app-87cb43c871-czst7   0/1     Completed               0        30h
example-app-87cb43c871-czst7   0/1     ContainerStatusUnknown  1        40h
example-app-87cb43c871-czst7   0/1     ContainerStatusUnknown  1        26h
example-app-87cb43c871-czst7   0/1     ContainerStatusUnknown  1        35h

To troubleshoot this issue, when I checked the description of above pod, I noticed below error:-

cyberithub@ubuntu:~$ kubectl describe pod example-app-87cb43c871-czst7
............................
Status:     Failed
Reason:     Evicted
Message:    Usage of EmptyDir volume "example-log" exceeds the limit "64Mi"
............................

So, it was found that new pods were keep on getting launched due to usage of emptyDir volume example-log exceeding the set limit of 64MB. This was preventing the pod to come back to running state. So to fix this issue, it was required to extend the size of volume example-log to 256MB for now. This size could be different in your case. So in my case, sizeLimit should look like below after the extension.

apiVersion: v1
kind: Pod
metadata:
  name: example-app
spec:
 containers:
 - image: cyberithub.com/test_containers/example-server
   name: example-container
   volumeMounts:
   - mountPath: /log
     name: log-volume
 volumes:
 - name: example-log
   emptyDir:
     sizeLimit: 256Mi

After extending the sizeLimit, I checked the status again. This time I see pod status came back to running state with 1 replica running as defined in replicaset. This confirms that extending the sizeLimit fixed the issue.

cyberithub@ubuntu:~$ kubectl get pods
NAME                           READY   STATUS                  RESTARTS AGE
example-app-64da84b614-xtrb9   1/1     Running                 0        18s

Similarly, in cases of other kubernetes objects such as DaemonSets or Statefulsets, if you see same kind of issue then probably you have to do the same thing there as well. You can easily find the error by checking the description of DaemonSets and Statefulsets. Most of the time, following above step should fix the issue. But in case if your issue is still not fixed, please feel free to write us in comment box so that we can work together in fixing your issue.