Spring Batch Tutorial: Reading Information From a REST API

Spring Batch has a good support for reading data from different data sources such as files (CSV or XML) or databases. However, it doesn’t have a built-in support for reading input data from a REST API. If you want to use a REST API as a data source of your Spring Batch job, you have to implement a custom ItemReader which reads the input data from the REST API.

This blog post describes how you can implement your custom ItemReader. After you have read this blog post, you:

• Understand how you can implement an ItemReader which reads the input data of your batch job by using the RestTemplate class.
• Know how you can configure the ItemReader bean which provides the input data for your batch job.

Let's begin.

Introduction to the Example Application

During this blog post you will implement an ItemReader which reads the input data of your Spring Batch job from a REST API endpoint that processes GET requests send to the path: '/api/student/'. This API endpoint returns the information of all students who are enrolled to an online course. To be more specific, your API endpoint returns the following JSON document:

[
    {
        "emailAddress": "tony.tester@gmail.com",
        "name": "Tony Tester",
        "purchasedPackage": "master"
    },
    {
        "emailAddress": "nick.newbie@gmail.com",
        "name": "Nick Newbie",
        "purchasedPackage": "starter"
    },
    {
        "emailAddress": "ian.intermediate@gmail.com",
        "name": "Ian Intermediate",
        "purchasedPackage": "intermediate"
    }
]

You have to transform the returned JSON document into StudentDTO objects which are processed by your batch job. The StudentDTO class contains the information of a single student, and its source code looks as follows:

public class StudentDTO {

    private String emailAddress;
    private String name;
    private String purchasedPackage;

    public StudentDTO() {}

    public String getEmailAddress() {
        return emailAddress;
    }

    public String getName() {
        return name;
    }

    public String getPurchasedPackage() {
        return purchasedPackage;
    }

    public void setEmailAddress(String emailAddress) {
        this.emailAddress = emailAddress;
    }

    public void setName(String name) {
        this.name = name;
    }

    public void setPurchasedPackage(String purchasedPackage) {
        this.purchasedPackage = purchasedPackage;
    }
}

Next, you will implement a custom ItemReader which reads the input data of your batch job from the described API endpoint.

Implementing Your Custom ItemReader

You can implement your custom ItemReader by following these steps:

First, you have to create a new class (RESTStudentReader) and implement the ItemReader interface. When you implement the ItemReader interface, you must set the type of the returned object to StudentDTO.

After you have created your ItemReader class, its source code looks as follows:

import org.springframework.batch.item.ItemReader;

class RESTStudentReader implements ItemReader<StudentDTO> {

}

Second, you have to add the following private fields to the RESTStudentReader class:

• The final apiUrl field contains the url of the invoked REST API.
• The final RestTemplate field contains a reference to the RestTemplate object which you use when you read the student information.
• The nextStudentIndex field contains the index of the next StudentDTO object.
• The studentData field contains the found StudentDTO objects.

After you have added these fields to the RESTStudentReader class, its source code looks as follows:

import org.springframework.batch.item.ItemReader;
import org.springframework.web.client.RestTemplate;

import java.util.List;

class RESTStudentReader implements ItemReader<StudentDTO> {

    private final String apiUrl;
    private final RestTemplate restTemplate;

    private int nextStudentIndex;
    private List<StudentDTO> studentData;
}

Third, you have to add a constructor to the RESTStudentReader class and implement it by following these steps:

1. Ensure that the constructor takes the url of the invoked REST API and a RestTemplate object as constructor arguments.
2. Implement the constructor by storing its constructor arguments in the fields of the created object. Set the value of the nextStudentIndex field to 0.

After you have implemented the constructor, the source code of the RESTStudentReader class looks as follows:

import org.springframework.batch.item.ItemReader;
import org.springframework.web.client.RestTemplate;

import java.util.List;

class RESTStudentReader implements ItemReader<StudentDTO> {
    
    private final String apiUrl;
    private final RestTemplate restTemplate;

    private int nextStudentIndex;
    private List<StudentDTO> studentData;

    RESTStudentReader(String apiUrl, RestTemplate restTemplate) {
        this.apiUrl = apiUrl;
        this.restTemplate = restTemplate;
        nextStudentIndex = 0;
    }
}

Fourth, you have to add a public read() method to the RESTStudentReader class and specify that the method returns a StudentDTO object. Also, you must ensure that this method can throw an Exception. After you have added this method to the RESTStudentReader class, you have to implement it by following these rules:

• If the student information hasn't been read, read the student information by invoking the REST API.
• If the next student is found, return the found StudentDTO object and increase the value of the nextStudentIndex field (the index of the next student) by 1.
• If the next student isn't found, return null. Ensure that your ItemReader reads the input data from the REST API when its read() method is invoked for the next time (set the value of the nextStudentIndex field to 0, and set the value of the studentData field to null).

After you have implemented the RESTStudentReader class, its source code looks as follows:

import org.springframework.batch.item.ItemReader;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;

import java.util.Arrays;
import java.util.List;

class RESTStudentReader implements ItemReader<StudentDTO> {

    private final String apiUrl;
    private final RestTemplate restTemplate;

    private int nextStudentIndex;
    private List<StudentDTO> studentData;

    RESTStudentReader(String apiUrl, RestTemplate restTemplate) {
        this.apiUrl = apiUrl;
        this.restTemplate = restTemplate;
        nextStudentIndex = 0;
    }

    @Override
    public StudentDTO read() throws Exception {
        if (studentDataIsNotInitialized()) {
            studentData = fetchStudentDataFromAPI();
        }

        StudentDTO nextStudent = null;

        if (nextStudentIndex < studentData.size()) {
            nextStudent = studentData.get(nextStudentIndex);
            nextStudentIndex++;
        }
        else {
            nextStudentIndex = 0;
            studentData = null;
        }

        return nextStudent;
    }

    private boolean studentDataIsNotInitialized() {
        return this.studentData == null;
    }

    private List<StudentDTO> fetchStudentDataFromAPI() {
        ResponseEntity<StudentDTO[]> response = restTemplate.getForEntity(apiUrl,
                StudentDTO[].class
        );
        StudentDTO[] studentData = response.getBody();
        return Arrays.asList(studentData);
    }
}

Before you can use your new ItemReader, you have to configure the RestTemplate bean. Let's move on and find out how you can configure this bean.

Configuring the RestTemplate Bean

You can configure the RestTemplate bean by following these steps:

1. Add a public restTemplate() method to your application context configuration class. Ensure that the restTemplate() method returns a RestTemplate object and annotate it with the @Bean annotation.
2. Implement the restTemplate() method by returning a new RestTemplate object.

If you use Spring Framework, the source code of your application context configuration class looks as follows:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;
 
@Configuration
public class SpringBatchExampleContext {
 
    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

If you use Spring Boot, you can also add the restTemplate() method to your application class which is annotated with the @SpringBootApplication annotation. The source code of this class looks as follows:

import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
@EnableBatchProcessing
@EnableScheduling
public class SpringBatchExampleApplication {

    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }

    public static void main(String[] args) {
        SpringApplication.run(SpringBatchExampleApplication.class, args);
    }
}

After you have configured the RestTemplate bean, you can finally configure your ItemReader bean.

Configuring the ItemReader Bean

You can configure the ItemReader bean by following these steps:

First, you have to create a new configuration class. After you have created this class, its source code looks as follows:

import org.springframework.context.annotation.Configuration;

@Configuration
public class SpringBatchExampleJobConfig {

}

Second, you have to create a new method that configures your ItemReader bean. This method returns an ItemReader<StudentDTO> object, and it takes an Environment object and a RestTemplate object as method parameters.

After you have added this method to your configuration class, its source code looks as follows:

import org.springframework.batch.item.ItemReader;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.web.client.RestTemplate;

@Configuration
public class SpringBatchExampleJobConfig {

    @Bean
    public ItemReader<StudentDTO> itemReader(Environment environment,
                                             RestTemplate restTemplate) {

    }
}

Fourth, you have to implement the itemReader() method by returning a new RESTStudentReader object. When you create a new RESTStudentReader object, you have to pass the following objects as constructor arguments:

• The url of the invoked REST API. You can read this information from a properties file by using the Environment object given as a method parameter.
• The RestTemplate object which is used to query the student information from the invoked REST API.

After you have implemented the itemReader() method, the source code of your configuration class looks as follows:

import org.springframework.batch.item.ItemReader;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.web.client.RestTemplate;

@Configuration
public class SpringBatchExampleJobConfig {

    @Bean
    public ItemReader<StudentDTO> itemReader(Environment environment,
                                             RestTemplate restTemplate) {
        return new RESTStudentReader(environment.getRequiredProperty("rest.api.url"),
                restTemplate
        );
    }
}

You can now write a custom ItemReader which reads the input data of your batch job from a REST API. Let’s summarize what you learned from this blog post.

Summary

This blog post has taught you two things:

• Spring Batch doesn’t have an ItemReader that can read information from a REST API.
• If you want to read the input data of your batch job from a REST API, you can read this information by using the RestTemplate class.

The next part of this tutorial describes how you can read the input data of your batch job from an Excel spreadsheet.

P.S. You can get the example application of this blog post from Github.