4. KAFKA : Sample Program2

4. KAFKA : Sample Program2

Step1 : Download Vmware workstation

https://my.vmware.com/web/vmware/details?downloadGroup=WKST-1259-WIN&productId=524&rPId=20840

                ********@gmail.com  

                PAVAN KUMAR MR

                ********    

 

 

Step2: Download Kafka_VM.ovf

Download Kafka_VM.ova from - https://drive.google.com/file/d/1UVqRyH10vGEh6rOi-0ivpsIS2KLIiYVF/view?ts=5e16bae0

Step3: Open kafka VM in VMWare workstation

Open Terminal in Kaka_VM and execute below commands highlighted in blue..

export http_proxy=www-**************.com:80
export https_proxy=www-*************.com:80
wget www.google.com
//download kafka directly in Workstation
wget http://www-us.apache.org/dist/kafka/2.2.1/kafka_2.12-2.2.1.tgz

//inside kafka_2.13-2.4.0\config
zookeeper.properties    //zookeeper can be downloaded independent also
dataDir=/tmp/zookeeper   //(default store in /tmp folder) - Physically maintain all Znode informations...
clientPort=2181  //on which port zookeeper is running...


server.properties //every broker one server.properties will be there
broker.id=0 //mandatory: unique broker should be there, minimum property
listeners=PLAINTEXT://:9092 //9092 is the port produceer and consumer can communicate
log.dirs=/tmp/kafka-logs
log.retention.hours=168
log.segment.bytes=1073741824   //if segment reaches this size then go that segment to one file  | | | | |

################################################################################

Open Terminal (Start Zookeeper -> Start Kafka -> Create Topic "MyFirstTopic"-> Start Producer on created Topic from  -> Start Consumer on created Topic)
//Start zookeeper & kafka server
cd kafka_2.12-2.2.1
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

bin/kafka-topics.sh --zookeeper localhost:2181 --list 

//Below command will fail bez replication factor is > number of brokers
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic MyFirstTopic --partitions 3 --replication-factor 2   

//Create Topic
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic MyFirstTopic --partitions 3 --replication-factor 1

//Describe Topic
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic MyFirstTopic

//increasing the partition for particular topic
//we cannot decrease the partition
//even when producer/consumer running we can increase the partitions
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic MyFirstTopic --partitions 5

For testing - console based Producer/Consumer
---------------------------------
Terminal-1 (Producer)
--------------------
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic MyFirstTopic

Terminal-2 (Consumer)
---------------------
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic MyFirstTopic

################################################################################