Setting Up Kafka

Before we can start using Kafka, we need to set it up on your computer. The easiest way to do this is using Docker, which lets us run Kafka without installing it directly on your system.

What is Docker?

Docker is like a lightweight virtual machine that runs apps in containers. Think of containers as isolated boxes that have everything an app needs to run.

Step 1: Install Docker

If you don't have Docker installed:

For Mac: Download and install Docker Desktop for Mac
For Windows: Download and install Docker Desktop for Windows
For Linux: Follow the installation instructions for your distribution

Step 2: Create a Docker Compose File

We'll use Docker Compose to set up Kafka and its dependency, ZooKeeper. Docker Compose lets us define and run multiple containers together.

Create a file named docker-compose.yml in the 01-setup directory with the following content:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.3.0
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

What's in this file?

Let's break down what this file does:

ZooKeeper Container:
- ZooKeeper is a service that Kafka uses to keep track of which servers are up and running
- It helps coordinate the Kafka servers (called brokers)
- We're exposing port 2181, which is the default port for ZooKeeper
Kafka Container:
- This is the actual Kafka server
- It depends on ZooKeeper, so Docker will start ZooKeeper first
- We're exposing port 9092, which is the default port for Kafka
- The environment variables configure Kafka:
  - KAFKA_BROKER_ID: A unique ID for this Kafka server
  - KAFKA_ZOOKEEPER_CONNECT: Tells Kafka how to connect to ZooKeeper
  - KAFKA_ADVERTISED_LISTENERS: The address that producers and consumers will use to connect to Kafka
  - The other settings are for a single-node setup (we're keeping it simple)

Step 3: Start Kafka

Now let's start Kafka using Docker Compose:

Open a terminal
Navigate to the directory containing your docker-compose.yml file
Run the following command:

docker compose up -d

The -d flag runs the containers in the background.

Step 4: Verify Kafka is Running

To check if Kafka is running:

docker compose ps

You should see both ZooKeeper and Kafka containers running.

Step 5: Create a Kafka Topic

A topic is like a category or feed name to which messages are published. Let's create our first topic:

docker compose exec kafka kafka-topics --create --topic first-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

This creates a topic named "first-topic" with 1 partition and 1 replica.

What's a Partition?

A partition is a way to split a topic into multiple parts. This allows Kafka to:

Store more data than can fit on a single server
Process messages in parallel

For now, we're keeping it simple with just 1 partition.

Step 6: List Kafka Topics

To see the topic we just created:

docker compose exec kafka kafka-topics --list --bootstrap-server localhost:9092

You should see "first-topic" in the list.

Step 7: Install Node.js Dependencies

We'll be using Node.js to interact with Kafka. Let's set up a package.json file:

npm init -y
npm install kafkajs dotenv

This installs:

kafkajs: A modern Kafka client for Node.js
dotenv: For loading environment variables from a .env file

Next Steps

Now that we have Kafka running, let's learn about the basic concepts in the next section.

Next: Basic Concepts

Learning Kafka: Set Up

Comments

Learning Kafka

Learning Kafka - Kafka Fundamentals

More from this blog

Learning Kafka - Real World Example

Learning Kafka - Kafka Consumers

Learning Kafka - Kafka Producers

Learning Kafka - Kafka Fundamentals

Setting Up Kafka

What is Docker?

Step 1: Install Docker

Step 2: Create a Docker Compose File

What's in this file?

Step 3: Start Kafka

Step 4: Verify Kafka is Running

Step 5: Create a Kafka Topic

What's a Partition?

Step 6: List Kafka Topics

Step 7: Install Node.js Dependencies

Next Steps

Command Palette

Comments

Learning Kafka

Learning Kafka - Kafka Fundamentals

More from this blog

Setting Up Kafka

What is Docker?

Step 1: Install Docker

Step 2: Create a Docker Compose File

What's in this file?

Step 3: Start Kafka

Step 4: Verify Kafka is Running

Step 5: Create a Kafka Topic

What's a Partition?

Step 6: List Kafka Topics

Step 7: Install Node.js Dependencies

Next Steps