I think that software development is more than just a job. I see myself as an artisan who is trying to get better every day. The "easiest way" to do this is to find a few good tools and find the answers to these questions:
- When should I use the tool X?
- How should I use the tool X?
Automated testing is a very important part of software development, but I haven’t seen a lot of blog posts that let you to take a peek into the toolbox of another developers.
This blog post lets you to take a peek into my toolbox. I will reveal 12 tools that I use for writing unit and integration tests. I will also provide links to other interesting webpages that help you to understand how you can use them.
Let's get started.
Taking a Peek Into My Toolbox
Before we can use the tools that are described in this blog post, we have to create a build that can run both unit and integration tests. I have written two blog posts that describe how we can do this:
- Integration Testing with Maven describes how we can create a Maven build that has different directories for unit and integration tests.
- Getting Started With Gradle: Integration Testing With the TestSets Plugin describes how we can create a Gradle build that has different directories for unit and integration tests.
We are now ready to take closer look at my toolbox. I have divided my tools into different categories that makes this blog post easier to read.
I give you 12 tools that I use for writing unit and integration tests:
Running Tests
JUnit is a framework that I use for writing both unit and integration tests. I like JUnit because it is the most popular testing framework for Java programming language. In other words, it has a lot extensions and it is easy to find solutions to your problems.
NestedRunner is a JUnit runner that allows us to run test methods placed in nested inner classes. I like NestedRunner because of the following reasons:
- We can replace long method names with BDD style class hierarchy.
- We can remove duplicate code by moving that code to setup methods and putting these methods to the correct inner classes.
- We can link the constants with the test cases that use them by declaring the constants in the correct inner class.
junit-dataprovider is a JUnit runner that allows us to write parametrized tests by using a TestNG like data provider. This is a huge improvement over the default way of writing parametrized tests that pretty much sucks.
Additional Reading:
- Unit Testing with JUnit describes how we can write unit tests with JUnit 4.X.
- Writing Clean Tests: Small Is Beautiful describes how we can solve the problems found from "clean" unit tests.
- junit-dataprovider: Getting Started describes how we can start using junit-dataprovider.
- junit-dataprovider: Features describes the features of the junit-dataprovider.
- Alternative to JUnit Parametrized Classes: junit-dataprovider describes how we can replace the standard parametrized tests with tests that use the junit-dataprovider.
Mocking, Stubbing, and Faking Stuff
Mockito is the most popular mocking framework for writing unit tests. I like it because it has a simple API, a lot of useful features, and excellent documentation.
Greenmail is an in-memory email server that supports SMTP, POP3, and IMAP with SSL socket support. I like it because it is really easy to use. When I was looking for a "fake" email server, I tried several alternatives and Greenmail was the only one that worked as I wanted.
MockFtpServer is a library that provides two different FTP server implementations (fake/stub) that are useful for testing different scenarios. If we need test code that interacts with a FTP server, MockFtpServer is our weapon of choice.
Additional Reading:
- Mockito Reference Manual is an excellent example of a reference documentation that actually helps you to get the job done.
- Integration Testing IMAP, SMTP and POP3 with Greenmail describes how you can write integration tests for code that sends email messages.
- FakeFtpServer - Getting Started describes how you can use the FakeFtpServer.
- StubFtpServer - Getting Started describes how you can use the StubFtpServer.
- FakeFtpServer or StubFtpServer? helps you to decide which FTP server implementation you should use in your tests.
Writing Assertions
Hamcrest provides matchers that we can use to write assertions for our unit and integration tests. I use it when I need to write assertions for unit or integration tests that use the Spring MVC Test framework.
AssertJ provides a fluent API for writing assertions that have helpful error messages. It improves the readability of our test code and helps us to transform our test cases into executable specifications that speak the correct domain-specific language.
Additional Reading:
- The Hamcrest Tutorial describes how you can get started with Hamcrest.
- Using Hamcrest for testing describes how you can use regular Hamcrest matchers in your unit tests and extend its capabilities by creating custom matchers.
- AssertJ Core Overview provides an overview to AssertJ Core.
- Turning Assertions Into a Domain-Specific Language describes how we can create custom AssertJ assertions.
- Writing Clean Tests: Replace Assertions With a Domain-Specific Language describes why we should replace regular JUnit assertions with custom assertions that use the correct domain-specific language.
Testing Data Access Code
H2 is a very fast in-memory database that is useful for writing integration tests that are run in the local development environment.
DbUnit is a JUnit extension that can be used to initialize the database into a known state before each integration test and ensure that the database contains the correct data. DbUnit has its warts, but it is a very useful tool because it helps us to separate the test data creation from the tested code.
Additional Reading:
- DbUnit Core Components describes the core components of DbUnit. If we want to write tests that use DbUnit, we should know what these components are.
- DbUnit Best Practices provides five rules that help us to write better tests for our data access code.
- Writing Tests for Data Access Code helps us to write deterministic data access tests which test the right thing, are easy to read, and easy to maintain.
Testing Spring Applications
Spring Test is the Swiss army knife of writing automated tests to Spring applications. It provides a first class support for writing unit and integration tests to Spring powered applications.
Spring Test DbUnit integrates DbUnit with the Spring Test Framework. If we need to write data access tests for a Spring application that uses relational database, Spring Test DbUnit helps us to do it.
Additional Reading:
- The Spring Framework Reference Manual: 11. Testing describes every feature of the Spring Test framework.
- Spring MVC Test Tutorial describes how we can write unit and integration tests for Spring MVC controllers.
- Spring Data JPA Tutorial: Integration Testing describes we you can write data access tests for our Spring powered repositories.
- Spring From the Trenches: Using Null Values in DbUnit Datasets describes how we can use null values in our DbUnit datasets when we are writing data access tests for a Spring application.
- Spring From the Trenches: Resetting Auto Increment Columns Before Each Test Method describes how we can reset the auto increment columns before each test method when we are writing data access tests for a Spring application.
What Testing Tools Do You Use?
I have now revealed the tools that I use for writing unit and integration tests. I challenge you to do the same. You can participate in this challenge by following these simple steps:
- If you have a blog, you can write a blog post and leave a comment to this blog post.
- If you don’t have a blog, you can leave a comment to this blog post.
Why should you do this?
Well, if helping other people isn’t enough for you, I promise to help you to get more readers by linking to your blog post, tweeting it, and sharing it with the Java Testing Society.
I guess you tried Spock? It replaces half of the tools you suggest: JUnit, NestedRunner, junit-dataprovider, Mockito, hamcrest, AssertJ. In order to run Spring integration tests you need an extra spock-spring plugin.
I haven't tried the Spock Framework yet. It is on my to-do list and I hope that I can try it during the summer.
indeed, you should try spock and you will remove half of the tools you mentioned. plus, your tests will become more readable and easier to maintain.
Yes. Spock all the way! +Geb for Browser automation tests
First, thank you for the post. As usual, what you write is useful and interesting.
For the section "Test Access Code" you have the well known DbUnit (http://dbunit.sourceforge.net/). It's very handy because you can easily put your database in a desired state. Moreover, it comes with several useful methods and features when you need to make tests with a database.
More generally, you have Unitils (http://unitils.org/summary.html) that provides a huge amount of modules and features to make your life easier when you write unit & integration tests!
Thank you for your kind words. I really appreciate them. Also, thank you for sharing your testing tools.
verify your exceptions in a better way:
https://code.google.com/p/catch-exception/
fest, fluent assertions
https://code.google.com/p/fest/ (will be replaced by assertj probably)
when you need a hammer
https://code.google.com/p/catch-exception/
testing equals is boring
https://github.com/jqno/equalsverifier
junit sucks at parameterising
https://github.com/Pragmatists/junitparams
these are the ones which came to mind right now, but there is more.
when you need a hammer should link to: https://code.google.com/p/powermock/
was a mistake :)
Ah. That explains it. Anyway, thank you for sharing your testing tools!
I'm using most of the tools mentioned in the article.
JUnit/AssertJ/Mockito for unit tests,
Spring test/Spring DBUnit test/DBUnit for repository integration tests
Spring test/RESTAssured for REST API integration tests
Thank you for sharing your testing tools. I left REST-assured out from my list because I haven't used it personally. However, I have heard that it is a very useful library if you need to write "real" integration tests for your REST API.
NestedRunner was new to me and I find it very interesting as you can build a structured setup hierarchy for your tests. Definitely something I'm going to try next.
There are a lot of good tools mentioned already in the comment section but I have to mention a couple more: JsonUunit with fluent assertions for comparing JSON in unit tests. It provides neat things like treating nulls as absent and ignoring extra fields.
https://github.com/lukas-krecan/JsonUnit
In addition to comparing JSON it can be useful to compare XML with xmlunit.
https://github.com/xmlunit/xmlunit
One more that I did not see mentioned is tempus-fugit for testing concurrent code.
http://tempusfugitlibrary.org/
Hi Arto,
Thank you for mentioning these testing libraries. I had heard about xmlunit, but JsonUnit and tempus-fugit are new to me.
Hi,
thanks for the comprehensive list!
For people working with MongoDB I can recommend Fongo - https://github.com/fakemongo/fongo - which fakes a MongoDB (hence the name) and intercepts calls to the standard Java driver, such that it is really easy to use with frameworks like Spring Data. It provides some useful test annotations for backing your test with JSON files simulating Mongo collection contents.
Hi Martin,
Thank you for your kind words. I really appreciate them.
Also, thank you for recommending Fongo. I will use it when I write integration tests for the example application of my blog post titled: Creating a REST API With Spring Boot and MongoDB.
javOSize, although it is not specifically designed QA tool, it is really helpful for troubleshooting and performance testing. Some Accenture guys are already using it, as I have just seen in their webpage, and it is totally free.
As the tool seems really new I am posting the link so you can find it easily: http://www.javosize.com
Hi Pedro,
Thank you for sharing this! It looks really interesting (and a bit similar than NewRelic). I took quick look at the getting started documentation and it explains some of the most basic use cases pretty well. I will definitely evaluate this tool after my summer holiday.
I've gotten some decent mileage out of awaitility (https://github.com/jayway/awaitility) when I need to test something involving asynchronous processing. It ends up being a more succinct and efficient way to poll for a result when testing an asynchronous process without littering your test methods with while loops and Thread.sleep calls.
Wow. I took a quick look at the user guide and it seems that Awaitility makes tests a lot easier to read. Thank you for sharing this (I hadn't heard about it before your comment).
I don't get why we don't get more focus on property based testing like in Haskell. In Java, Functional Java provides the best and most faithful implementation, http://functionaljava.org/. The original paper is available at http://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf.
Hi Mark,
I took a very quick look at the paper and noticed that it argues that a testing tool should be able to generate test cases automatically. There is a library called RandomizedTesting that helps you to do this (if you use JUnit). Since I didn't read the entire paper yet, I won't comment property based testing, but I can share some of the challenges which I face on my work.
My biggest challenge is to convince people to write tests. It seems that developers can be divided into four groups:
Because I am passionate about automated testing, I want to help developers to either start writing them or learn new trick that help them to write better tests. Surprisingly, I spend most of my time trying to convince people to write automated tests or write tests that are easy to read and maintain. That is why I feel that it doesn't make sense to convince people to use "advanced" testing techniques when they aren't necessarily using the obvious ones yet.
Hi.
I switched to use TestNG (supported by Spring test) instead of JUnit.
DbSetup - http://dbsetup.ninja-squad.com/index.html
looks promised as replacement of DbUnit as it is Java configurable. You have not to deal with xml/yaml db descriptions.
To write build in-container tests in Java EE world I used http://arquillian.org/ tool
Hi Arho,
Saw your posts on an interesting thread about whether one should test private methods, here: https://henrikwarne.com/2014/02/09/unit-testing-private-methods/
I am a newbie at TDD and trying to teach myself... this is pretty hard. I have worked through Growing Object-Oriented Software Guided by Tests, and also Practical Unit Testing with JUnit and Mockito. I am getting... um, somewhere. But when I start to try to apply some of the thinking to my own projects I quickly get confused ... and find that I'm exposing too much of the SUT classes... and probably making my tests far too fine-grained.
Your line in the above thread gives the impression that you have reached the "Holy Grail" of making the right methods public, and successfully then refactoring so that private methods emerge (effortlessly and elegantly).
The question is, how do I get from here to there? Please could you recommend maybe another book or two, or a site or two?
PS I see you have one book involving the Spring Framework. This is (currently) one framework too far for me. Mockito and AssertJ are my ideal combination.
Thanks,
Mike
(you could just email me if you want...)
I need to test Vaadin Framework application ,which kind of tool is good other then "TestBench"
Hi,
Unfortunately I cannot help you since I have no experience from Vaadin.
Good list of tools here. My team works on Spring applications and mostly work with JUnit, Mockito, and Powermock. We leverage a tool from Parasoft called the Jtest Unit Test Assistant to create our unit test and it does it good job of actually creating readable unit tests. http://www.parasoft.com/products/jtest#unittesting
Found it helpful. Thanks for sharing!