Most enterprise applications rely heavily on batch jobs. They run during the night and do all the time consuming tasks that cannot be done during business hours. These tasks are often critical to the business and errors can cause serious damage (i.e. cost a lot of money).
That’s why it is important that we write robust batch jobs that provide the correct output, are fault tolerant, and are as fast as possible. Spring Batch can help us to achieve these goals.
This blog post is the first part of my Spring Batch tutorial, and it provides a quick introduction to Spring Batch. After we have read this blog post, we:
- Can specify the term batch job.
- Understand why we should use Spring Batch instead of writing our own batch jobs.
- Can identify the basic building blocks of a Spring Batch job.
Let’s start by defining the term batch job.
What Is a Batch Job?
A batch job is often defined as follows:
A batch job is a computer program or set of programs processed in batch mode. This means that a sequence of commands to be executed by the operating system is listed in a file (often called a batch file, command file, or shell script) and submitted for execution as a single unit.
However, this definition is not very pragmatic, and it doesn’t help us to understand what kind of batch jobs are required by a typical enterprise application. That’s why I will provide my own definition:
A batch job reads input data, processes the input data, and writes the processed data to the configured output.
The following figure illustrates a simple batch job that fulfills my definition:
As we can see, this batch job has only one step. This is perfectly fine if our batch job has only one logical task. For example, if we are implementing an import job that reads information from an input file and writes it to the database, our job has only one logical task.
However, some batch jobs have more than one logical task. For example, we might have to implement a batch job that imports information from an input file and creates an export file that’s exported to other applications. In other words, our batch job has two logical tasks. This means that it has two steps as well.
It seems that I have to rewrite my definition. The final version is:
A batch job consists of one or more steps. Each step is responsible of completing one logical task. Every step reads input data, processes the input data, and writes the processed data to the configured output. If the batch job has more than one step, the output of a step is often used as an input of the next step.
The following figure illustrates a batch job that has two steps:
We have now defined the term batch job. Let’s find out why we should implement our batch jobs by using Spring Batch.
How Can Spring Batch Help Us?
I have written a lot of batch jobs during my career and seen many batch jobs written by other developers. I have noticed that most non-trivial batch jobs (including mine), which don’t use any framework or library, suffer from these problems:
- The code that implements the batch job is a mess. Because it has only one huge step, no one cannot really understand how the batch job works.
- The batch job is slow because it does everything inside a HUGE transaction.
- The batch job doesn’t have a real error handling. If an error occurs during a batch job, the job simply fails. However, if we are lucky, the batch job might write an error message to a log file.
- The batch job doesn’t clean up the output data that’s written to the configured output if it fails. This is a problem because we cannot trust the data that’s produced by the batch job. In other words, we have to ensure (manually) that the output data of the batch job is correct. This is a waste of time.
- The batch job doesn’t report its final state. In other words, there is no easy way to figure out if the batch job was finished successfully.
We can (of course) fix every one of these problems. If we decide to follow this approach, we face two new problems:
- We have to essentially create an in-house batch job framework, and it is extremely hard to get everything right at the first time.
- Creating an in-house batch job framework is a big task and it takes time that we don’t often have. This means that we cannot fix the problems found from the first version of our batch job framework because we don’t have time to do it. That’s why all in-house frameworks have their own oddities.
Luckily, we don’t have to implement our own batch job framework because Spring Batch solves all of these problems. It provides the following features that helps us to solve these problems:
- It helps us to structure our code in a clean way by providing the infrastructure that’s used to implement, configure, and run batch jobs.
- It uses so called chunk oriented processing where items are processed one by one and the transaction is committed when the chunk size is met. In other words, it provides us an easy way to manage the size of our transactions.
- It provides proper error handling. For example, we can skip items if an exception is thrown and configure retry logic that’s used to determine whether our batch job should retry the failed operation. We can also configure the logic that’s used to decide if our transaction should be rolled back.
- It writes comprehensive log to the used database. This log contains the metadata of each job and step execution, and it’s extremely useful if we have to troubleshoot a failed batch job. Because the log is written to a database, we can access it by using a database client.
We should now understand that Spring Batch solves the problems caused by handwritten batch jobs. Let’s move on and take a quick look at the key components of a Spring Batch job.
The Key Components of a Spring Batch Job
A Spring Batch job consists of the following components:
Jobrepresents a single Spring Batch job. Each job can have one or more steps.
Steprepresents an independent logical task (i.e. import information from an input file). Each step belongs to one job.
ItemReaderreads the input data and provides the found items one by one. An
ItemReaderbelongs to one step and each step must have one
ItemProcessortransforms items into a form that’s understood by the
ItemWriterone item at a time. An
ItemProcessorbelongs to one step and each step can have one
ItemWriterwrites an information of an item to the output one item at a time. An
ItemWriterbelongs to one step and each step must have one
The following figure illustrates the relationships of these components:
We can now define the term batch job, we understand why we should use Spring Batch, and we can identify the key components of a Spring Batch job. Let’s summarize what we learned from this blog post.
This blog post has taught us five things:
- A batch job consists of one or more steps. Each step is responsible of completing one logical task. Every step reads input data, processes the input data, and writes the processed data to the configured output. If the batch job has more than one step, the output of a step is often used as an input of the next step.
- We should use Spring Batch because it solves the problems caused by handwritten batch jobs.
- A Spring batch
Jobcan have one or more steps.
Stepmust have one
Stepcan have one
The next part of this tutorial describes how we can get the required dependencies with Maven.