Learn to write tests which are easy to read and write: get started with Spock Framework.

Spring Batch Tutorial: Reading Information From a REST API

Spring Batch has a good support for reading input data from different data sources such as files and databases.

However, it doesn’t have a built-in support for reading input data from a REST API. This means that we have to create a custom ItemReader.

This blog post describes how we can create a custom ItemReader that reads the input data of our batch job by using the RestTemplate class.

Let’s get started.

If you are not familiar with Spring Batch, you should read the following blog posts before you continue reading this blog post:

Introduction to Our Example Application

During this tutorial we will implement several Spring Batch jobs that processes the student information of an online course. This time we have to implement an ItemReader that fetches the student information by sending a GET request to an external REST API. This API returns always the following JSON:

[
	{
		"emailAddress": "tony.tester@gmail.com",
		"name": "Tony Tester",
		"purchasedPackage": "master"
	},
	{
		"emailAddress": "nick.newbie@gmail.com",
		"name": "Nick Newbie",
		"purchasedPackage": "starter"
	},
	{
		"emailAddress": "ian.intermediate@gmail.com",
		"name": "Ian Intermediate",
		"purchasedPackage": "intermediate"
	}
]

We have to transform that JSON into StudentDTO objects which are processed by our batch job. The StudentDTO class contains the information of a single student, and its source code looks as follows:

public class StudentDTO {
  
    private String emailAddress;
    private String name;
    private String purchasedPackage;
  
    public StudentDTO() {}
  
    public String getEmailAddress() {
        return emailAddress;
    }
  
    public String getName() {
        return name;
    }
  
    public String getPurchasedPackage() {
        return purchasedPackage;
    }
  
    public void setEmailAddress(String emailAddress) {
        this.emailAddress = emailAddress;
    }
  
    public void setName(String name) {
        this.name = name;
    }
  
    public void setPurchasedPackage(String purchasedPackage) {
        this.purchasedPackage = purchasedPackage;
    }
}

Let’s create a custom ItemReader that reads input data from a REST API.

Creating an ItemReader That Reads Input Data From a REST API

We can create our ItemReader by following these steps:

  1. Create a RESTStudentReader class.
  2. Implement the ItemReader interface and set the type of the returned object to StudentDTO.
  3. Add the following private fields to the RESTStudentReader class:
    • The final apiUrl field contains the url of the external REST API.
    • The final RestTemplate field contains a reference to the RestTemplate object that we use when we fetch the student information.
    • The nextStudentIndex field contains the index of the next StudentDTO object.
    • The studentData field contains the list of found StudentDTO objects.
  4. Add a constructor to the created class and ensure that the constructor has the following constructor arguments:
    • The url of the external REST API.
    • A RestTemplate object.
  5. Implement the constructor by storing its constructor arguments into the fields of the created object. Set the value of the nextStudentIndex field to 0.
  6. Add a public read() method to created class, and specify that the method returns a StudentDTO object and can throw an Exception.
  7. Implement the read() method by following these rules:
    • If the student information has not been fetched, fetch the student information from the external API by using the RestTemplate object.
    • If the next student is found, return the found StudentDTO object and increase the index of the next student by 1.
    • If the next student is not found, return null.

The source code of the RESTStudentReader class looks as follows:

import org.springframework.batch.item.ItemReader;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;

import java.util.Arrays;
import java.util.List;

class RESTStudentReader implements ItemReader<StudentDTO> {


    private final String apiUrl;
    private final RestTemplate restTemplate;

    private int nextStudentIndex;
    private List<StudentDTO> studentData;

    RESTStudentReader(String apiUrl, RestTemplate restTemplate) {
        this.apiUrl = apiUrl;
        this.restTemplate = restTemplate;
        nextStudentIndex = 0;
    }

    @Override
    public StudentDTO read() throws Exception {
        if (studentDataIsNotInitialized()) {
            studentData = fetchStudentDataFromAPI();
        }

        StudentDTO nextStudent = null;

        if (nextStudentIndex < studentData.size()) {
            nextStudent = studentData.get(nextStudentIndex);
            nextStudentIndex++;
        }

        return nextStudent;
    }

    private boolean studentDataIsNotInitialized() {
        return this.studentData == null;
    }

    private List<StudentDTO> fetchStudentDataFromAPI() {
        ResponseEntity<StudentDTO[]> response = restTemplate.getForEntity(
			apiUrl, 
			StudentDTO[].class
		);
        StudentDTO[] studentData = response.getBody();
        return Arrays.asList(studentData);
    }
}


Before we can use our new ItemReader, we have to configure the RestTemplate bean. Let’s find out how we can configure this bean.

Configuring the RestTemplate Bean

We can configure the RestTemplate bean by following these steps:

  1. Add a restTemplate() method to our application context configuration class, ensure that the method returns a RestTemplate object, and annotate the method with the @Bean annotation.
  2. Implement the method by returning a new RestTemplate object.

If we use Spring Framework, the source code of our application context configuration class looks as follows:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

@Configuration
public class SpringBatchExampleContext {

    @Bean
    RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

If we use Spring Boot, we can also add the restTemplate() method into our @SpringBootApplication class. The source code of this class looks as follows:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
public class SpringBatchExampleApplication {

    @Bean
    RestTemplate restTemplate() {
        return new RestTemplate();
    }

    public static void main(String[] args) {
        SpringApplication.run(SpringBatchExampleApplication.class, args);
    }
}
We might have to add some additional dependencies into our build script before we can configure the RestTemplate bean. These dependencies are described in the following:

  • If we are using Spring Framework, we have to add the spring-webmvc dependency into our build script.
  • If we are using Spring Boot, we have to add the spring-boot-starter-web dependency into our build script.

After we have configured the RestTemplate bean, we can finally configure our new ItemReader bean.

Configuring the ItemReader Bean

We can configure our ItemReader bean by following these steps:

  1. Create a configuration class and annotate the created class with the @Configuration annotation.
  2. Add a new method to the created class, ensure that this method returns an ItemReader<StudentDTO> object, and annotate this method with the @Bean annotation.
  3. Ensure that the created method takes an Environment object and a RestTemplate object as method parameters.
  4. Implement the method by returning a new RESTStudentReader object. When we create a new RESTStudentReader object, we have to pass the following objects as constructor arguments:
    • The url of the external REST API. We will fetch this information from a properties file by using the Environment object given as a method parameter.
    • The RestTemplate object that is used to fetch the student information from the external REST API.

The source code of our configuration class looks as follows:

import org.springframework.batch.item.ItemReader;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.web.client.RestTemplate;

@Configuration
public class RESTStudentJobConfig {

    @Bean
    ItemReader<StudentDTO> restStudentReader(Environment environment, 
											 RestTemplate restTemplate) {
        return new RESTStudentReader(
			environment.getRequiredProperty("rest.api.to.database.job.api.url"), 
			restTemplate
		);
    }
}

Let’s summarize what we learned from this blog post.

Summary

This blog post has taught us two things:

  • Spring Batch doesn’t have an ItemReader that can read information from a REST API.
  • If we want to read the input data of our batch job from an external REST API, we have to read this information by using the RestTemplate class.

The next part of this tutorial describes how we can read the input data of our batch job from an Excel spreadsheet.

P.S. You can get the example applications of this blog post from Github: Spring example and Spring Boot example.

About the Author

Petri Kainulainen is passionate about software development and continuous improvement. He is specialized in software development with the Spring Framework and is the author of Spring Data book.

About Petri Kainulainen →

14 comments… add one
  • This was a great tutorial on how to use a REST API as the source of my data for Spring Batch. I am struggling with the best way to unit test it though. Do you have any suggestions or additions for this example?

    Thanks!

    Reply
    • Hi,

      You could move the logic that reads the input data to a separate component (I will call this component RestInputReader) and inject that component to the Spring Batch reader. This gives you the possibility to replace the RestInputReader with a stub when you are writing unit tests for the Spring Batch reader.

      I wouldn’t write unit tests for the RestInputReader because it doesn’t make much sense to replace RestTemplate with a test double because RestTemplate does all the work. I would test it by writing integration tests. If the REST API is an external API, I would replace it with a simple stub.

      If you have any additional questions, don’t hesitate to ask them.

      Reply
  • I have gone through Spring Batch Tutorial: Reading Information From a REST API post. It is very useful. Can please let me know how to run this project?

    Reply
    • Hi,

      Sure. Just clone this repository, select the example you want to run (Spring or Spring Boot), and run the command described in the README.

      Reply
      • Hi Petri,

        How to read XML file using custom itemreader,itemprocessor write into database using itemwriter.

        Pls can u share code for above one

        Waiting for your response
        Advanced thank

        Dandu
        8553562402

        Reply
        • Hi,

          Unfortunately I cannot cover this topic in a single answer. However, you can find my Spring Batch tutorial and a few other great resources from my Spring Batch resource page. Also, this blog post explains how you can read data from XML file and write it to a database.

          Reply
  • Hi Petri,
    Thank you for this great post. I have approximately the same use case than you. But my problem is than my rest service is using a database containing more than 150.000 entries. So I am afraid than my memory is not big enough. How can I handle that problem?

    Reply
    • Hi Florent,

      I agree that it’s not a good to load the content of your database into memory. I would probably implement an API that supports pagination and write an ItemReader that reads the data one page at the time.

      If you don’t know how you can do it, don’t hesitate to ask additional questions.

      Reply
  • Thanks Petri, it helped me a lot your example.

    Reply
    • You are welcome. I am happy hear that this blog post was useful to you.

      Reply
  • Petri
    The ItemReader you have is stateful. Do you think it is thread-safe?

    ~Sherin

    Reply
    • By the way it is a great example

      ~Sherin K Syriac

      Reply
      • Thank you for your kind words. I really appreciate them. About your question:

        Yes, my ItemReader is stateful, and Spring Batch assumes that this is the case. The Javadoc of the ItemReader interface states that:

        Implementations are expected to be stateful and will be called multiple times for each batch, with each call to read() returning a different value and finally returning null when all input data is exhausted.

        Implementations need not be thread-safe and clients of a ItemReader need to be aware that this is the case.

        Reply

Leave a Comment