View on GitHub

Kafka Docker

Run multiple Kafka brokers in Docker

Download this project as a .zip file Download this project as a tar.gz file

Following the excellent tutorials on how to run a multi broker kafka cluster natively: and in vagrant:

Here is the tutorial based on Docker and Compose:

Cluster Setup

  1. Install Docker
  2. Install Compose
  3. Update docker-compose.yml with your docker host IP (KAFKA_ADVERTISED_HOST_NAME)
  4. If you want to customise any Kafka parameters, simply add them as environment variables in docker-compose.yml. For example:
    • to increase the message.max.bytes parameter add KAFKA_MESSAGE_MAX_BYTES: 2000000 to the environment section.
    • to turn off automatic topic creation set KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
  5. Start the cluster
$ docker-compose up

e.g. to start a cluster with two brokers

$ docker-compose scale kafka=2

This will start a single zookeeper instance and two Kafka instances. You can use docker-compose ps to show the running instances. If you want to add more Kafka brokers simply increase the value passed to docker-compose scale kafka=n

Kafka Shell

You can interact with your Kafka cluster via the Kafka shell:



To test your setup, start a shell, create a topic and start a producer:

$ $KAFKA_HOME/bin/ --create --topic topic \
--partitions 4 --zookeeper $ZK --replication-factor 2
$ $KAFKA_HOME/bin/ --describe --topic topic --zookeeper $ZK 
$ $KAFKA_HOME/bin/ --topic=topic \

Start another shell and start a consumer:

$ $KAFKA_HOME/bin/ --topic=topic --zookeeper=$ZK

Running kafka-docker on a Mac:

Install the Docker Toolbox and set KAFKA_ADVERTISED_HOST_NAME to the IP that is returned by the docker-machine ip command.