Integrations using MuleSoft Anypoint and Apache Kafka

Mulesoft’s Anypoint Platform in the cloud has its own Advanced Queue as a convenience messaging layer but what if you wanted to use a streaming platform like Apache Kafka? Mulesoft has documented how to use their Kafka Connector but here is an example of how to use it in four easy steps using Kafka on Docker.

1. Setup Kafka

For my developer Kafka setup I have used Confluent’s single node docker image which I find rather convenient to use Kafka. Since I use a Windows Laptop I needed to have Docker Desktop installed first.

Using the Windows Powershell, I change directory into the examples\kafka-single-node folder and start up the Zookeeper and Kafka containers using the Docker compose “Up” command

Using Windows Powershell, exploring the running containers I can see that two are up and running.    

2. Create Kafka Topic

Using Windows Powershell, now I need to create a topic that I will use to publish and consume messages. Since Kafka is running inside a docker container I will have to run my commands inside the container using the docker exec bash command. Once I am inside the container bash it will be like I am working inside a linux server that has Kafka already installed and running.

Now I can proceed to use Linux style commands for kafka to create the topic called “topic1”. The assumption is that I am going to have three consumers for this topic hence I use three partitions.

root@4d7f3a2acad6:/# kafka-topics --bootstrap-server localhost:9092 --create --topic topic1 --replication-factor 1 --partitions 3

I will run a console consumer to see the items being published into the topic

root@4d7f3a2acad6:/# kafka-console-consumer --bootstrap-server localhost:9092 --topic topic1 --from-beginning

3. Create Mule Flow

Using Anypoint Studio, create a New Mule Project

Create Mule Project

We will create a Mule flow that will Publish Messages to the Kafka topic. Drag and Drop the HTTP Listener component from the Mule Palette.

Add HTTP Listener to the Mule Flow

Next we add the Kafka Connector for Mule 4 by searching on the Exchange using the Mule Palette

Add Kafka Connector on Anypoint Studio

Next, Drag and Drop the Kafka > Publish Message component, Create the connector configuration using the configurations from the Kafka Setup

Kafka Producer Configuration

The Mule Flow is now ready to publish messages to Kafka 

Mule Flow with Kafka Publisher

We will now create a Kafka Consumer on the same Mule Flow (Ideally you want this on a separate Mule flow) for demonstration only.

Drag and drop the Apache Kafka > Message Consumer to the Mule Flow, then configure the Consumer Configuration

Kafka Consumer Configuration

The Mule Flow can now consume messages published to “topic1” on Apache Kafka. We will log the message payload to make sure they reach the consumers.

Mule Flow with Consumer that logs messages

Run the Mule flow project on Anypoint Studio. This will create an HTTP Endpoint which can be invoked using Postman

On Starting up the mule flow we can see that the consumer has logged three partitions for the specific topic in the console output

4. Publishing the message to Kafka Topic from the Mule Flow

Using Postman, you can now send a REST payload to the HTTP Listener in the Mule Flow which will send the payload to the Kafka topic. The payload will appear automatically on the Kafka console consumer that we ran earlier as well as the Consumer in the Mule Flow.

Postman call to Mule Flow

Kafka Console Consumer Output

The console output shows us the message that was sent on the topic

Mule Flow Kafka Consumer Console Output

As we can see it is relatively easy to get started on MuleSoft to use Kafka as the messaging layer and take advantage of the power of Kafka and MuleSoft.

• Author
• Recent Posts

Vikram

Director at Fusion Applied

Vikram is a software team leader and developer with over 13 years' experience developing software and managing projects in challenging, fast-paced consulting environments. An experienced OO developer with expertise in J2EE and Oracle Fusion Middleware technologies. His current focus is on helping customers build solutions using the Oracle Fusion Middleware stack including Oracle SOA Suite 11g, BPM, ADF & WebCenter Suite.

Latest posts by Vikram (see all)

• Integrations using MuleSoft Anypoint and Apache Kafka - January 3, 2020
• How to Auto Dismiss an Oracle BPM FYI Task - February 18, 2016
• Oracle OpenWorld 2015: The Adaptive Case Management Feature—Federal Government Healthcare Use Case [CON1568] - October 22, 2015