Apache Kafka – Getting Started on Windows 10

Learn to install Apache Kafka on Windows 10 and executing start server and stop server scripts related to Kafka and Zookeeper. We will also verify the Kafka installation by creating a topic, producing few messages to it and then use a consumer to read the messages written in Kafka.

1. Prerequisites

• Java 8 is required to run latest downloads from Kafka site.
• Zookeeper (to store metadata about the Kafka cluster) is also mandatory. Kafka comes with inbuilt Zookeeper to start with. But it is recommended to install zookeeper separately in production environment. Download it from its official site.
• Kafka can be run on any operating system. The linux is recommended OS. With Windows, Kafka have some knows bugs.

This tutorial is for beginners and does not use separate zookeeper instance – to keep things simple and focused towards Kafka only.

2. Download and install Kafka

• Download Kafka from official site. I download the latest version on today which is 2.5.0 and file name is “kafka_2.12-2.5.0.tgz“.
• Copy the downloaded file to some folder and extract it using tar command.

  > tar -xzf kafka_2.12-2.5.0.tgz

• Copy the extracted folder in desired location. I have put it on location “E:\devsetup\bigdata\kafka2.5“.

  Installation is pretty much done !!

3. Startup and Shutdown

To start Kafka, we need to first start Zookeeper and then Kafka. I am writing small batch files which move to Kafka installation directory first and then execute the command in new command prompt window.

3.1. Start Zookeeper

To start zookeeper, we need to run zookeeper-server-start.bat script and pass zookeeper configuration file path.

cd E:\devsetup\bigdata\kafka2.5
start cmd /k bin\windows\zookeeper-server-start.bat config\zookeeper.properties

3.2. Start Kafka

To start Kafka, we need to run kafka-server-start.bat script and pass broker configuration file path.

cd E:\devsetup\bigdata\kafka2.5
start cmd /k bin\windows\kafka-server-start.bat config\server.properties

3.3. Shutdown Kafka

To stop Kafka, we need to run kafka-server-stop.bat script.

cd E:\devsetup\bigdata\kafka2.5
start cmd /k bin\windows\kafka-server-stop.bat

3.4. Shutdown Zookeeper

To stop Zookeeper, we need to run zookeeper-server-stop.bat script.

cd E:\devsetup\bigdata\kafka2.5
start cmd /k bin\windows\zookeeper-server-stop.bat

Do not stop zookeeper and kafka using CTRL+C commands. Always use above .bat files or commands. Otherwise the data corruption may occur.

4. Verify Kafka Installation

First, start Zookeeper and Kafka using above scripts.

• Open a new command prompt, and create new Kafka topic.

  > bin\windows\kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
  
  //Output:
  
  Created topic test.
  

• Now list all the topics to verify the created topic is present in this list. At this step, we have only one topic.

  > bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092
  
  //Output:
  
  test
  

• Now list all the topics to verify the created topic is present in this list. At this step, we have only one topic.

  > bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092
  
  //Output:
  
  test
  

• Produce some messages and submit to test topic. I added two messages i.e. “Hello” and “Kafka”.

  > bin\windows\kafka-console-producer.bat --bootstrap-server localhost:9092 --topic test
  

• Consume the messages and submit to test topic.

  > bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
  
  //Output:
  
  Hello
  Kafka
  

5. Conclusion

In this tutorial, we learned to install Kafka along with Zookeeper. We learned to start and stop both servers. Additionally, we verified the installation by creating a topic, posting some messages and then consuming using console consumer script.

Important thing to note is that we should never stop the servers by kill process or CTRL+C commands. Always use scripts to stop the servers.

Happy Learning !!

Reference: