Getting started with Apache Kafka and Prometheus JMX javaagent on windows DockerToolbox

ok , lets begin.

we will first write the Dockerfile

here , i am using wurstmeister/kafka, which you can get from docker hub

kafka.yml is a prometheus jmx agent config file which the prometheus uses for scrapping ( kafka.yml is in the format which prometheus understands ) get this file from https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

jmx_prometheus_javaagent is a java agent exposing a HTTP server and serving metrics of local JVM .our dockerfile will download from this maven repo https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.11.0/jmx_prometheus_javaagent-0.11.0.jar

get any latest jar from repo1.maven.org ( i used 0.11.0 )

once the Dockerfile is ready, run the docker build command

$ docker build -t dockerpromkafka:latest .

{ don’t forget dot (.) at the end of the command to indicate current folder location of Dockerfile }

lets write docker-compose.yaml file

note: using docker-compose version: ‘2’

1. zookeeper service

zookeeper service

2. prometheus service

prometheus service

here is the prometheus config file called prometheus.yml
note: in targets its kafka-cluster ( same as kafka service name )

prometheus.yml config file

3. kafka-cluster service

kafka service

lets understand this

kafka server starts on port 9092 which is exposed outside as well with 9092.
1099 is the jmx port for getting metrics of kafka application ( remember kafka runs on scala which has jvm )
EXTRA_ARGS is an important environment for this to work
its the same agent jar we downloaded and put in Dockerfile
KAFKA_JMX_OPTS is another important environment variable for JMX ( later we will connect from jconsole )
here 192.168.99.108 is the docker-machine ip ( will try to get dynamically next time )

OK , now we have all the files ready lets build it

$ docker-compose up -d

this should start all the services

$ docker ps -a

to get all container status ( if error try $ docker logs <container id > )

lets get into kafka container

$ docker exec -it <containerid> bash

lets create a topic and start producing data for consumer to consume( when copying the below commands make sure its not dash , its double hyphen )

# kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic my-topic1
# kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic1

to receive messages in our windows host
you need to have kafka installed in https://kafka.apache.org/downloads (unzip to C:\kafka\ ) . now open cmd

> .\bin\windows\kafka-console-consumer.bat — bootstrap-server 192.168.99.108:9092 — topic my-topic1 — from-beginning

now send message from container , you should be able to see in windows cmd

to visualize metrics in prometheus
go to prometheus dashboard http://192.168.99.108:9090
you should be able to get all kafka related query

prometheus dashboard

for the climax: from jconsole or even from jvisualvm

1. enter jconsole in cmd
2. in the local process , you will get the process name as
   sun.tools.jconsole.JConsole 192.168.99.108:1099
   click connect
3. go to MBeans tab

once you expand the → you will get all metrics values
similary try from jvisualvm — task for you guys ….. :)

Thanks y’ll folks , Take it easy…!!