Kafka Setup and Configuration

Kafka:

Kafka is a distributed system , it act as a cluster.

A kafka cluster consist of set of brokers , A cluster has a minimum of 3 brokers.

Kafka broker:

The broker is the kafka server , it act as a agent to exchange message between producer and consumer.

producer:

producer is an application that sends messages .It does not send messages directly to the recipients ,it send message only to the kafka server

consumer:

consumer is an application that read message from kafaka server.

kafka topics:

Topic is like a table in database or folder in a file system

Topic is identified by a name

you can have any number of topic .

kafka partition:

kafka topic are divided into number of partitions, which contain record in an unchangable sequence

Offsets:

Offsets is a sequence of ids given to message arrival at partition ,Once the offset is assigned it will never be changed.The first message grts an offset zero.The next message gets an offset 1 and so on..

Consumer group:

A consumer group contains one or more consumers working together to process the messages

Install and setup Apache kafka :

-->Download the tgz file and extract it :

https://dlcdn.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz

-->open the command prompt and go to the location where u downloaded this kafka folder

Zookeper setup:

command:

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Kafka Broker setup:

command:

.\bin\windows\kafka-server-start.bat .\config\server.properties

Creation of Topic:

producer-consumer MESSAGE