Java Spring Boot REST Server with Kafka Consumer

🕓 40 minutes

What you’ll learn

How to set up your application for :

• connecting to Kafka consuming messages from its’ topic,
• providing data to REST API.

In this tutorial, we will create a simple java component with the java spring-boot scaffolder. We want to expose a single REST endpoint for getting client application logs. Logs are published in Kafka topic, so we need a Kafka client configuration as well. Inside ClientAplicationLogs component is simple logic, which stores last 10 messages from Kafka and is available to return them via REST API.

Project source

This example project can be cloned from: http://gitlab.cloud.codenow.com/public-docs/client-authorization-demo/client-application-logs.git

Prerequisites

• Prepare your local development environment for CodeNOW with Java Spring Boot.
  • Follow the tutorial instructions in Java Spring Boot Local Development.
• Run Apache Kafka locally.
  • Use docker compose as described in the section Docker compose and third-party tools of the Java Spring Boot Local Development tutorial.
• Create a new component
  • For details see the section Prerequisites of the Java Spring Boot Local Development tutorial.

Steps

Open your IDE, import the created component and start coding:

• Define the message payload. Here is an example of the Log, which is a simple POJO with basic client data:

  • generate getters and setters with your IDE

    package io.codenow.client.app.logs.service.model;

    import java.util.Date;

    public class Log {

    private Date timestamp;

    private String key;

    private String payload;

    public Log(Date timestamp, String key, String payload) {

    super();

    this.timestamp = timestamp;

    this.key = key;

    this.payload = payload;

    }

• Next prepare the configuration for the Kafka logging client:

  • Go to the Kafka administration console (http://localhost:9000 if using Kafdrop from our Java Spring Boot Local Development manual) and create a new topic client-logging

  • Add maven dependency to your pom.xml

    <dependency>

    <groupId>org.springframework.kafka</groupId>

    <artifactId>spring-kafka</artifactId>

    </dependency>

  • For more details about spring-kafka, see: https://spring.io/projects/spring-kafka

  • Now let’s create simple cache queue storing only last couple of messages. Field ‘limit’ is configured with spring property log.cache.size (see below):

    package io.codenow.client.app.logs.service.kafka;

    import java.util.concurrent.ArrayBlockingQueue;

    import org.springframework.beans.factory.annotation.Value;

    import org.springframework.stereotype.Component;

    import io.codenow.client.app.logs.service.model.Log;

    @Component

    public class FixedSizeLogCache extends ArrayBlockingQueue<Log>{

    private static final long serialVersionUID = 3640159618214478205L;

    private int limit;

    public FixedSizeLogCache(@Value("${log.cache.size}") int limit) {

    super(limit);

    this.limit = limit;

    }

    @Override

    public boolean add(Log item) {

    ensureLimit();

    return super.add(item);

    }

    @Override

    public boolean offer(Log e) {

    ensureLimit();

    return super.offer(e);

    }

    private void ensureLimit() {

    if(super.size() >= limit) {

    super.remove();

    }

    }

    }

• Next step is using spring-boot kafka autoconfiguration and create new kafka listener for logs

  • ‘spring.kafka.topic.name’ is configured via spring properties

    package io.codenow.client.app.logs.service.kafka;

    import java.util.Date;

    import org.springframework.kafka.annotation.KafkaListener;

    import org.springframework.kafka.support.KafkaHeaders;

    import org.springframework.messaging.handler.annotation.Header;

    import org.springframework.messaging.handler.annotation.Payload;

    import org.springframework.stereotype.Component;

    import io.codenow.client.app.logs.service.model.Log;

    @Component

    public class KafkaLogsListener {

    private FixedSizeLogCache cache;

    public KafkaLogsListener(FixedSizeLogCache cache) {

    this.cache = cache;

    }

    @KafkaListener(topics = "${spring.kafka.topic.name}")

    public void logsListener(@Payload String payload, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) String messageKey, @Header(KafkaHeaders.RECEIVED_TIMESTAMP) Date timestamp) {

    cache.add(new Log(timestamp, messageKey, payload));

    }

    }

• Next, create a new controller and put all the parts together

  • For more detail about the spring REST controller, see: https://spring.io/guides/gs/rest-service/

    package io.codenow.client.app.logs.service.controller;

    import org.springframework.web.bind.annotation.GetMapping;

    import org.springframework.web.bind.annotation.RequestMapping;

    import org.springframework.web.bind.annotation.RestController;

    import io.codenow.client.app.logs.service.kafka.FixedSizeLogCache;

    import io.codenow.client.app.logs.service.model.Log;

    import reactor.core.publisher.Flux;

    @RestController

    @RequestMapping("/logs")

    public class ClientApplicationLogsController {

    private FixedSizeLogCache cache;

    public ClientApplicationLogsController(FixedSizeLogCache cache) {

    this.cache = cache;

    }

    @GetMapping

    private Flux<Log> getLogs() {

    return Flux.fromIterable(cache);

    }

    }

• Last but not least, replace configuration in codenow/config/application.yaml with following code:

  • Note that this configuration depends on your local development setup for Kafka and can differ case-by-case. For your local configuration is probably required to uncomment ‘bootstrap-servers: localhost:29092’ and comment the line above.

  • Make sure you follow yaml syntax (especially whitespaces)

    server:

    port: 8080

    spring:

    main:

    banner-mode: off

    zipkin:

    enabled: false

    kafka:

    bootstrap-servers: client-logging-kafka-kafka-brokers.managed-components:9092

    # bootstrap-servers: localhost:29092

    consumer:

    group-id: log-group

    topic:

    name: client-logging

    management:

    endpoints:

    web:

    exposure:

    include: health, prometheus

    log:

    cache:

    size: 10

• Try to build and run the application in your IDE. After startup, you should be able to access your new controller’s swagger: http://localhost:8080/swagger/index.html

What’s next?

If your code works in the local development, you are ready to push your changes to GIT and try to build and deploy your new component version to the CodeNOW environment. For more information see Application Deployment and Monitoring, just make sure to change the application.yaml properties from the local to the production setup.

• Check Get New Apache Kafka for setup in the CodeNOW environment.