Getting started with Apache Kafka and Prometheus JMX javaagent on windows DockerToolbox
ok , lets begin.
we will first write the Dockerfile
here , i am using wurstmeister/kafka, which you can get from docker hub
kafka.yml is a prometheus jmx agent config file which the prometheus uses for scrapping ( kafka.yml is in the format which prometheus understands ) get this file from https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml
jmx_prometheus_javaagent is a java agent exposing a HTTP server and serving metrics of local JVM .our dockerfile will download from this maven repo https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.11.0/jmx_prometheus_javaagent-0.11.0.jar
get any latest jar from repo1.maven.org ( i used 0.11.0 )
once the Dockerfile is ready, run the docker build command
$ docker build -t dockerpromkafka:latest .
{ don’t forget dot (.) at the end of the command to indicate current folder location of Dockerfile }
lets write docker-compose.yaml file
note: using docker-compose version: ‘2’
- zookeeper service
2. prometheus service
here is the prometheus config file called prometheus.yml
note: in targets its kafka-cluster ( same as kafka service name )
3. kafka-cluster service
lets understand this
kafka server starts on port 9092 which is exposed outside as well with 9092.
1099 is the jmx port for getting metrics of kafka application ( remember kafka runs on scala which has jvm )
EXTRA_ARGS is an important environment for this to work
its the same agent jar we downloaded and put in Dockerfile
KAFKA_JMX_OPTS is another important environment variable for JMX ( later we will connect from jconsole )
here 192.168.99.108 is the docker-machine ip ( will try to get dynamically next time )
OK , now we have all the files ready lets build it
$ docker-compose up -d
this should start all the services
$ docker ps -a
to get all container status ( if error try $ docker logs <container id > )
lets get into kafka container
$ docker exec -it <containerid> bash
lets create a topic and start producing data for consumer to consume( when copying the below commands make sure its not dash , its double hyphen )
# kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic my-topic1# kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic1
to receive messages in our windows host
you need to have kafka installed in https://kafka.apache.org/downloads (unzip to C:\kafka\ ) . now open cmd
> .\bin\windows\kafka-console-consumer.bat — bootstrap-server 192.168.99.108:9092 — topic my-topic1 — from-beginning
now send message from container , you should be able to see in windows cmd
to visualize metrics in prometheus
go to prometheus dashboard http://192.168.99.108:9090
you should be able to get all kafka related query
for the climax: from jconsole or even from jvisualvm
- enter jconsole in cmd
- in the local process , you will get the process name as
sun.tools.jconsole.JConsole 192.168.99.108:1099
click connect - go to MBeans tab
once you expand the → you will get all metrics values
similary try from jvisualvm — task for you guys ….. :)
Thanks y’ll folks , Take it easy…!!