Setting Up Kafka

Before we can start using Kafka, we need to set it up on your computer. The easiest way to do this is using Docker, which lets us run Kafka without installing it directly on your system.

What is Docker?

Docker is like a lightweight virtual machine that runs apps in containers. Think of containers as isolated boxes that have everything an app needs to run.

Step 1: Install Docker

If you don't have Docker installed:

• For Mac: Download and install Docker Desktop for Mac

• For Windows: Download and install Docker Desktop for Windows

• For Linux: Follow the installation instructions for your distribution

Step 2: Create a Docker Compose File

We'll use Docker Compose to set up Kafka and its dependency, ZooKeeper. Docker Compose lets us define and run multiple containers together.

Create a file named docker-compose.yml in the 01-setup directory with the following content:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.3.0
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

What's in this file?

Let's break down what this file does:

1. ZooKeeper Container:

   • ZooKeeper is a service that Kafka uses to keep track of which servers are up and running

   • It helps coordinate the Kafka servers (called brokers)

   • We're exposing port 2181, which is the default port for ZooKeeper

2. Kafka Container:

   • This is the actual Kafka server

   • It depends on ZooKeeper, so Docker will start ZooKeeper first

   • We're exposing port 9092, which is the default port for Kafka

   • The environment variables configure Kafka:

     • KAFKA_BROKER_ID: A unique ID for this Kafka server

     • KAFKA_ZOOKEEPER_CONNECT: Tells Kafka how to connect to ZooKeeper

     • KAFKA_ADVERTISED_LISTENERS: The address that producers and consumers will use to connect to Kafka

     • The other settings are for a single-node setup (we're keeping it simple)

Step 3: Start Kafka

Now let's start Kafka using Docker Compose:

1. Open a terminal

2. Navigate to the directory containing your docker-compose.yml file

3. Run the following command:

docker compose up -d

The -d flag runs the containers in the background.

Step 4: Verify Kafka is Running

To check if Kafka is running:

docker compose ps

You should see both ZooKeeper and Kafka containers running.

Step 5: Create a Kafka Topic

A topic is like a category or feed name to which messages are published. Let's create our first topic:

docker compose exec kafka kafka-topics --create --topic first-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

This creates a topic named "first-topic" with 1 partition and 1 replica.

What's a Partition?

A partition is a way to split a topic into multiple parts. This allows Kafka to:

• Store more data than can fit on a single server

• Process messages in parallel

For now, we're keeping it simple with just 1 partition.

Step 6: List Kafka Topics

To see the topic we just created:

docker compose exec kafka kafka-topics --list --bootstrap-server localhost:9092

You should see "first-topic" in the list.

Step 7: Install Node.js Dependencies

We'll be using Node.js to interact with Kafka. Let's set up a package.json file:

npm init -y
npm install kafkajs dotenv

This installs:

• kafkajs: A modern Kafka client for Node.js

• dotenv: For loading environment variables from a .env file

Next Steps

Now that we have Kafka running, let's learn about the basic concepts in the next section.

Next: Basic Concepts