Using Confluent to run your Kafka infrastructure

Using a local running Kafka is obviously the best, fastest and simplest way to start playing with Kafka, and that’s exactly what I did and described in two previous articles, Write a Kafka producer with Java using Spring Boot and Write a Kafka consumer with Java using Spring Boot.

The next step in the journey is clearly centralize and scale Kafka infrastructure to support truly distributed applications and more meaningful use cases.

Nowadays for practically everything in the modern IT domain tools landscape there are several choices, either installing on your own (either on-premises or using IaaS infrastructure provided by some Cloud provider) or instantiating some SaaS; Kafka is no exception and there is plenty of Cloud service providers offering Kafka infrastructure “as a Service”, just to mention a few:

• Confluent (https://confluent.cloud/)
• Aiven (https://aiven.io)
• CloudKarafka (https://www.cloudkarafka.com/)

In this article we will see how to exploit Confluent to instantiate a Kafka cluster and setup the minimal configurations needed for Kafka producers and consumers to interact with Confluent Kafka as a Service.

Setup the environment

Few steps are required to setup a Kafka cluster on Confluent and create the needed configurations to let our Kafka clients (i.e.: the Kafka producer and consumer I developed in my previous articles, that…