In the previous article, we have seen How to Restart Apache Kafka Connectors? In this article, we will see how to modify kafka connectors configuration using kcctl utility. Source or sink connectors configuration can be easily set or modified through kcctl utility without much sweating. All you need to know is the right configuration that needs to be used. Many times after setting up the connectors config, you would realize that some of the config parameters are not optimized or configured properly due to which messages throughput, volume, connector error handling, source and consumer are getting impacted.
To deal with the situation, modifying parameters through kcctl utility would be a good choice. To understand this further, I have configured a setup where kafka connector is connected to a sample MongoDB source. Any changes in MongoDB collection data triggers an event to destination kafka topic called cyber.col.topic through source connector. I have set few of the configuration parameters in kafka connector which makes sure that data delivery throughput will be maximum. I am going to show you how you can easily modify those parameters through kcctl utility as per requirements.
How to modify Kafka Connectors configuration using kcctl
Also Read: Solved "HPA and KEDA(ScaledObject) terminating manually scaled up pods in Kubernetes"
If you would like to check the current set value of all configuration parameters of connector cyberithub-connector then you can check by running kcctl describe connector cyberithub-connector as shown below.
cyberithub@macos1066 % kcctl describe connector cyberithub-connector Name: cyberithub-connector Type: source State: STOPPED Worker ID: example.cyberithub.com:8083 Config: change.stream.full.document: updateLookup collection: testcol connection.uri: mongodb://${file:/usr/share/cyberithub/secrets.properties:username}:${file:/usr/share/cyberithub/secrets.properties:password}@cyberithub.example.server.com:27017/?ssl=true connector.class: com.mongodb.kafka.connect.MongoSourceConnector database: testdb errors.tolerance: none name: cyberithub-connector cyber.topic.comment: cyber.col.topic producer.override.max.request.size: 2700000 tasks.max: 1 topic.prefix: cyber.col Tasks: Topics: cyber.col.topic
If you want to change the maximum size of a request that kafka connector cyberithub-connector can send to kafka topic cyber.col.topic which indeed produced by the producer then you have to modify the producer.override.max.request.size value using below kcctl command.
cyberithub@macos1066 % kcctl patch connector cyberithub-connector --set producer.override.max.request.size=3000000 New connector configuration: Name: cyberithub-connector Type: source State: STOPPED Worker ID: example.cyberithub.com:8083 Config: change.stream.full.document: updateLookup collection: testcol connection.uri: mongodb://${file:/usr/share/cyberithub/secrets.properties:username}:${file:/usr/share/cyberithub/secrets.properties:password}@cyberithub.example.server.com:27017/?ssl=true connector.class: com.mongodb.kafka.connect.MongoSourceConnector database: testdb errors.tolerance: none name: cyberithub-connector cyber.topic.comment: cyber.col.topic producer.override.max.request.size: 3000000 tasks.max: 1 topic.prefix: cyber.col Tasks: Topics: cyber.col.topic
Similarly, if you are looking to change the number of parallel consumer tasks for a specific connector then you can change by running below command. Below step will increase the number of parallel consumer tasks for connector cyberithub-connector to 3 to allow more tasks to run in parallel.
cyberithub@macos1066 % kcctl patch connector cyberithub-connector --set tasks.max=3
- -s, --set=<String=String> Set the following configuration parameter
If any of the connector tasks fail then you can also try to restart it by using kcctl restart task <connector_name>/<task_number> command. For example, if task 1 of connector cyberithub-connector fails then you can try to restart it by using kcctl restart task cyberithub-connector/1 command as shown below.
cyberithub@macos1066 % kcctl restart task cyberithub-connector/1
If you are looking to restart all tasks of a connector instead of restarting a specific task then you can simply using kcctl restart connector <connector_name>. For example, in our case we are looking to restart all tasks(0,1,2) of connector cyberithub-connector by running kcctl restart connector cyberithub-connector command as shown below.
cyberithub@macos1066 % kcctl restart connector cyberithub-connector
If you are looking to change parameter such as schema registry url then you can do that as well by running kcctl patch connector <connector_name> -s=key.converter.schema.registry.url=<schema_registry_url> command. For example, in our case we are updating schema registry url to https://example-schema-registry.cyberithub.com using below command.
cyberithub@macos1066 % kcctl patch connector cyberithub-connector -s=key.converter.schema.registry.url=https://example-schema-registry.cyberithub.com
If you have multiple connectors where you would like to change the schema registry url then you can use below command to update url in all the connectors at once.
cyberithub@macos1066 % kcctl patch connector -e '.*' -s=key.converter.schema.registry.url=https://example-schema-registry.cyberithub.com
- -e, --reg-exp use CONNECTOR NAME(s) as regex pattern(s) to use on all connectors
You can even choose to remove a specific parameter from connector configuration using -r switch. For example, if you would like to remove the parameter key.converter.schema.registry.url from connector configuration then you remove it by using kcctl patch connector cyberithub-connector -r key.converter.schema.registry.url command. After executing below command, if you check connector configuration you will see that parameter key.converter.schema.registry.url disappeared from configuration.
cyberithub@macos1066 % kcctl patch connector cyberithub-connector -r key.converter.schema.registry.url
- -r, --remove=<removeParameters> Remove the following configuration parameter
If you have multiple property to remove from configuration then you can use -r switch multiple times. For example if you would like to remove both parameters value.converter.schema.registry.url and key.converter.schema.registry.url from configuration then you can remove it in a single command as shown below.
cyberithub@macos1066 % kcctl patch connector cyberithub-connector -r value.converter.schema.registry.url -r key.converter.schema.registry.url
You can also set and remove parameter in a single command. For example, you can set number of parallel tasks to run on connector cyberithub-connector and remove parameter key.converter.schema.registry.url in a single line of command by running kcctl patch connector cyberithub-connector -s tasks.max=3 -r key.converter.schema.registry.url as shown below.
cyberithub@macos1066 % kcctl patch connector cyberithub-connector -s tasks.max=3 -r key.converter.schema.registry.url