Kafka Connect SMT’s — Shortening The Learning Curve

Kafka Connect is a powerful ecosystem for building and running streaming data pipelines. Out of the box Kafka Connect provides a number of ways to convert, transform, and process the data for both the Kafka message that gets sent to Kafka from the source datasource and the data that gets sent to the destination datasource.

SMT stands for Single Message Transformation and are built into Kafka Connect. Confluent does a great job at providing examples of each SMT and even provides source code examples. You can also create custom ones which I will cover in a future article.

The documentation lives here.

But what is an SMT really?

To break it down an SMT is a DSL(Domain Specific Language) that describes how to transform a piece of data and its default routing. They are written as a series of key value pairs that build on top of one another and are chainable. The syntax is confusing at first. I converted a classic example to a fake nested Json structure to show the different parts more clearly.

The SMT Language
Fake Example to show Expanded Structure

How to Test It?

This is where the learning curve explodes. First you have to run 4 or more applications just to try this configuration out. Kafka, Zookeeper, Datasource (or Data generator), and Kafka Connect. Once all of this is running, you have to load some data into Kafka using a series of command line tools and if your configuration is wrong you have to reboot your Kafka Connect application (in standalone mode) or delete and reinstall the connector (in distributed mode).

Is there a better way?

This is where I got to thinking, what if there was a way to try these SMT’s without the headache of an entire Kafka stack. The idea of an SMT debugger was born. What if I could go to a website plug in my configuration and example data and see what the transformer would do. I believe for people starting out this will be an easy way to grok the entire SMT before getting into the nuances of the use case they are trying to solve for.

Try it out for yourself!

It currently supports the built-in org.apache.kafka.connect.transforms and a custom one com.github.jcustenborder.kafka.connect.transform.common. I can add more on request.

Software Engineer/Architect

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store