In this article, we will see how to restart apache kafka connectors. But before that, let's understand what are Kafka Connectors. They are basically plugin component which facilitates the interaction between Kafka and external systems such as database, storage, indexes etc. Connectors are integral part of Kafka connect framework playing a vital role in data streaming. It comes in two different types - Source connector and Sink Connector.

Source connector are used where data needs to be transferred from external system to kafka topic whereas sink connectors are used to stream data from kafka topic to external system. The usage of both connectors can be further understood through an example. If you would like to produce data from database to a kafka topic then you would have to configure source connector to stream data from database to kafka topic. Similarly, if you would like to stream data from a kafka topic to an external system such as Cassandra db then you would have to configure sink connector there.

Now that we understand about kafka connectors, lets understand a situation where kafka connectors needs to be restarted. Ideally, there could be multiple use cases where a connector needs to be restarted. For example, if you have noticed that you are not receiving any database updates on kafka topic then there might a problem where source connector may have been in stopped state or not working properly. In those cases, a simple restart of source connector can fix the problem.

Similar can be the case with sink connectors where data from kafka topic may not be getting transferred to database due to connector issue. There also, restarting connector can help fixing this problem. There could be such multiple use cases where a simple restart of connector can solve the problem maximum number of times.

 

How to Restart Apache Kafka Connectors?

Also Read: Solved "HPA and KEDA(ScaledObject) terminating manually scaled up pods in Kubernetes"

Kafka connectors can be easily restarted as and when required using very simple steps. I am going to demonstrate the restart by showing through an example. I have three source kafka connectors currently configured in kubernetes cluster context called cyber. We need to first switch to this context by using kcctl config set-context command. Below is the complete command to switch context.

cyberithub@macos1066 % kcctl config set-context cyber --cluster=https://cyberithub-kafka-connect-cyber.com --username=abfgfdc --password=cyberpass

We can also check all the contexts using kcctl config get-contexts command as shown below.

cyberithub@macos1066 % kcctl config get-contexts
 
 NAME    KAFKA CONNECT URI
 cyber*  https://cyberithub-kafka-connect-cyber.com

After switching to cyber context, we can check all configured connectors using kubectl get connectors command as shown below.

cyberithub@macos1066 % kcctl get connectors

NAME                    TYPE    STATE     TASKS
cyberithub-connector    source  RUNNING   0: RUNNING
hello-connector         source  RUNNING   0: RUNNING
example-connector       source  STOPPED

To stop a connector, use kcctl stop connector <connector_name> command. Here we are stopping one of our connector called cyberithub-connector using below command.

cyberithub@macos1066 % kcctl stop connector cyberithub-connector

Check the connector state by using kcctl get connectors command as shown below. As we can see, cyberithub-connector is now in stopped state.

cyberithub@macos1066 % kcctl get connectors

NAME                   TYPE   STATE    TASKS
cyberithub-connector   source STOPPED
hello-connector        source RUNNING  0: RUNNING
example-connector      source STOPPED

Now that connector is in stopped state,  we can resume it by using kcctl resume connector <connector_name> command.

cyberithub@macos1066 % kcctl resume connector cyberithub-connector

Check the successful starting of connector by using kcctl get connectors command as shown below. Once the source connector is restarted, you would observe that update from external system is getting published to kafka topic. That's all !!

cyberithub@macos1066 % kcctl get connectors

NAME                      TYPE    STATE    TASKS
cyberithub-connector      source  RUNNING  0: RUNNING
hello-connector           source  STOPPED
example-connector         source  STOPPED