Balu
Balu Platform Engineer | Devops | Monitoring | CICD | ODD | SQL/NoSQL | ELK | AWS | Jhipster | Microservices | Agile | Scrum

Kafka - Ops

Kafka - Ops

Kafka Operations

  • Create a topic
1
2
3
4
5
docker run \
  --net=host \
  --rm \
  confluentinc/cp-kafka:4.0.0 \
  kafka-topics --create --topic test_topic_name --partitions 3 --replication-factor 2 --if-not-exists --zookeeper  localhost:22181
  • List the Topics
1
2
3
4
5
docker run \
  --net=host \
  --rm \
  confluentinc/cp-kafka:4.0.0 \
  kafka-topics --list --zookeeper  localhost:22181
  • Delete the topic
1
docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-topics --zookeeper localhost:22181 --delete --topic test_topic_name

Modify the Replicas

To increase the number of replicas for a given topic you have to:

  1. Specify the extra replicas in a custom reassignment json file

For example, you could create increase-replication-factor.json and put this content in it:

1
2
3
4
5
6
{"version":1,
  "partitions":[
     {"topic":"signals","partition":0,"replicas":[0,1,2]},
     {"topic":"signals","partition":1,"replicas":[0,1,2]},
     {"topic":"signals","partition":2,"replicas":[0,1,2]}
]}
  1. Use the file with the –execute option of the kafka-reassign-partitions tool or kafka-reassign-partitions.sh depending on the kafka package

For example:

1
docker run -v $PWD:/home/   --net=host   --rm   confluentinc/cp-kafka:4.0.0 kafka-reassign-partitions --zookeeper localhost:22181 --reassignment-json-file /home/increase-replica.json --execute
  1. Verify the replication factor with the kafka-topics tool or kafka-topics.sh - depending on the kafka package
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
docker run \
  --net=host \
  --rm \
  confluentinc/cp-kafka:4.0.0 \
  kafka-topics --zookeeper localhost:22181 --topic <TOPC_NAME> --describe



docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-consumer-groups --bootstrap-server 172.168.2.5:19092,172.168.0.1:29092,172.168.0.1:39092 --group <CONSUMER_GROUP_NAME> --describe
  
  
## expected sample output

Topic:signals   PartitionCount:3    ReplicationFactor:3 Configs:retention.ms=1000000000
Topic: signals  Partition: 0    Leader: 2   Replicas: 0,1,2 Isr: 2,0,1
Topic: signals  Partition: 1    Leader: 2   Replicas: 0,1,2 Isr: 2,0,1
Topic: signals  Partition: 2    Leader: 2   Replicas: 0,1,2 Isr: 2,0,1

reset-offsets

1
2
3
docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-consumer-groups --bootstrap-server 172.168.0.0:19092,172.168.0.1:29092,172.168.0.1:39092 --group <CONSUMER_GROUP_NAME> --reset-offsets --to-earliest --topic <TOPIC_NAME> --execute

docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-consumer-groups --bootstrap-server 172.168.0.0:19092,172.168.0.1:29092,172.168.0.1:39092 --group <CONSUMER_GROUP_NAME> --reset-offsets --to-earliest --topic <TOPIC_NAME> --execute

Describe the Topic/Group

1
docker run --net=host --rm confluentinc/cp-kafka:4.0.0 kafka-consumer-groups --bootstrap-server 172.168.0.0:19092,172.168.0.1:29092,172.168.0.1:39092 --group <CONSUMER_GROUP_NAME> --describe

comments powered by Disqus