Cyberithub

How to Check Logs of Failed CronJobs in Kubernetes Cluster

Advertisements

In this article, we will look into how to check logs of failed cronjobs in Kubernetes Cluster but before that let's first try to understand more about CronJobs. What is CronJob ? Why we need to run CronJobs ? So as you probably know a CronJob is basically a Job scheduled to perform a specific task. If you have ever used crontab in Linux System then you probably know how to schedule a cron job. If you haven't then let me tell you a cronjob object is like a one line of a crontab file written in cron format to run a job periodically on a given schedule.

CronJobs are used for performing regular scheduled actions such as backups, report generation, and so on. Each of those tasks should be configured to recur indefinitely (for example: once a day / week / month). You can define the point in time within that interval when the job should start. Now that we discussed about the cronjob, let's see through an example how to check the logs of a failed CronJob.

Advertisements

How to Check Logs of Failed CronJobs in Kubernetes Cluster

How to Check Logs of Failed CronJobs in Kubernetes Cluster

Also Read: How to Access Kubernetes Cluster Metrics Using Port Forwarding

You might want to argue that cronjob logs can be checked through kubectl logs -n <namespace> cronjob.batch/<cronjob_name> command but as of Kubernetes v1.20, it does not work as you can see below.

Advertisements
[node1 ~]$ kubectl logs -n default cronjob.batch/hello
error: cannot get the logs from *v1beta1.CronJob: selector for *v1beta1.CronJob not implemented

First of all, it is important to note that there is no direct command to check the logs of failed cronjobs. It is because cronjobs are usually responsible for starting, running and deleting the pods so anything specific to a failed cronjob can only be checked through pod logs in case the pod was successfully launched by the cron job or through events in case the pod did not got launched or got destroyed abruptly. So here I am going to explain through an example how we can identify the reason for failed cron jobs. To start with, I have the below example cronjob running in my k8s cluster.

[node1 ~]$ kubectl get cronjobs
NAME  SCHEDULE     SUSPEND ACTIVE LAST SCHEDULE AGE
hello */1 * * * *  False     5    20s           4m31s

The manifest file for above cronjob is taken from the GitHub repo. As you can see, below Job is scheduled to run every 1 minute.

Advertisements
[node1 ~]$ cat cronjob.yaml 
apiVersion: batch/v1beta1
kind: CronJob
metadata: 
   name: hello
spec:
   schedule: "*/1 * * * *"
   jobTemplate:
     spec:
      template:
       spec:
        containers:
        - name: hello
          image: busybox
          args:
          - /bin/sh
          - -c
          - date; echo hello from the K8s cluster
        restartPolicy: OnFailure

To know more about the above hello cronjob, you can describe it by running kubectl describe cronjob hello command as shown below. If you have some other cronjob then you can describe that cronjob by using kubectl describe cronjob <cronjob_name> syntax.

[node1 ~]$ kubectl describe cronjob hello
Name: hello
Namespace: default
Labels: <none>
Annotations: <none>
Schedule: */1 * * * *
Concurrency Policy: Allow
Suspend: False
Successful Job History Limit: 3
Failed Job History Limit: 1
Starting Deadline Seconds: <unset>
Selector: <unset>
Parallelism: <unset>
Completions: <unset>
Pod Template:
Labels: <none>
Containers:
hello:
Image: busybox
Port: <none>
Host Port: <none>
Args:
/bin/sh
-c
date; echo hello from the K8s cluster
Environment: <none>
Mounts: <none>
Volumes: <none>
Last Schedule Time: Sun, 31 Jul 2022 16:19:00 +0000
Active Jobs: hello-1659283920, hello-1659283980, hello-1659284040, hello-1659284100, hello-1659284160, hello-1659284220, hello-1659284280, hello-1659284340
Events:
Type   Reason           Age   From               Message
----   ------           ----  ----               -------
Normal SuccessfulCreate 7m12s cronjob-controller Created job hello-1659283920
Normal SuccessfulCreate 6m11s cronjob-controller Created job hello-1659283980
Normal SuccessfulCreate 5m11s cronjob-controller Created job hello-1659284040
Normal SuccessfulCreate 4m11s cronjob-controller Created job hello-1659284100
Normal SuccessfulCreate 3m11s cronjob-controller Created job hello-1659284160
Normal SuccessfulCreate 2m11s cronjob-controller Created job hello-1659284220
Normal SuccessfulCreate 70s   cronjob-controller Created job hello-1659284280
Normal SuccessfulCreate 10s   cronjob-controller Created job hello-1659284340

To check the status of the pods created by the Scheduled CronJob, you can check the events by using kubectl get events command as shown below. From here you can check the reason of your cronjob failure. For example, in our case one of the pod hello-1659283920-xm7tp did not got scheduled because one of the node in the Cluster is not available. Similarly, you can check and verify the reason in your case.

Advertisements
[node1 ~]$ kubectl get events
LAST SEEN  TYPE    REASON           OBJECT                      MESSAGE
29s        Warning FailedScheduling pod/hello-1659283920-xm7tp  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
114s       Normal  SuccessfulCreate job/hello-1659283920        Created pod: hello-1659283920-xm7tp
53s        Warning FailedScheduling pod/hello-1659283980-54rdq  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
53s        Normal  SuccessfulCreate job/hello-1659283980        Created pod: hello-1659283980-54rdq
114s       Normal  SuccessfulCreate cronjob/hello               Created job hello-1659283920
53s        Normal  SuccessfulCreate cronjob/hello               Created job hello-1659283980

If you have all the pods started and running fine. Then you can check the logs of your pod by using kubectl logs <pod_name> syntax. Here we are checking the logs of hello-1659283920-xm7tp pod by using kubectl logs hello-1659283920-xm7tp command as shown below. If you have multiple containers in a pod then use -c <container_name> option.

[node1 ~]$ kubectl logs hello-1659283920-xm7tp

Leave a Comment