Getting Started With Gradle: Integration Testing

Because the standard project layout of a Java project defines only one test directory (src/test), we have no standard way to add integration tests to our Gradle build.

If we want to use the standard project layout, we can add integration tests to our Gradle build by using one of the following options:

  • We can add our integration tests to the same directory than our unit tests. This is an awful idea because integration tests are typically a lot slower than unit tests. If we decide to use this approach, the length of our feedback loop is a lot longer than it should be.
  • We can create a new project and add our integration tests to that project. This makes no sense because it forces us to transform our project into a multi-project build. Also, if our project is already a multi-project build, we are screwed. We can of course add all integration tests to the same project or create new integration test project for each tested project, but it would be less painful to shoot ourselves in the foot.

It is clear that we need a better way. This blog post describes how we create a Gradle build that fulfils the following requirements:

  • Integration and unit tests must have different source directories. The src/integration-test/java directory must contain the source code of our integration tests and the src/test/java directory must contain the source code of our unit tests.
  • Integration and unit tests must have separate resource directories. The src/integration-test/resources directory must contain the resources of our integration tests. The src/test/resources directory must contain the resources of our unit tests.
  • We must be able to configure compile time and runtime dependencies for our integration tests.
  • We must be able to run either our unit tests or integration tests.
  • We must be able to run all tests.
  • If an integration test fails, our build must fail as well.
  • Integration and unit tests must have separate HTML reports.

Let’s start by configuring the source and resource directories of our integration tests.

This example application of this blog post is tested with Gradle 4.6.

Additional Reading:

If you are not familiar with Gradle, you should read the following blog posts before you continue reading this blog post:

Configuring the Source and Resource Directories of Our Integration Tests

We can add new source and resource directories to our Gradle build by using the sourceSets build script block. Armed with this information, we can configure the source and resource directories of our integration tests by following these steps:

  1. Create a new source set called integrationTest.
  2. Ensure that the output of the main and test source sets is added to the compile time classpath.
  3. Ensure that the output of the main and test source sets is added to the runtime classpath.
  4. Set the source directory of our integration tests to src/integration-test/java.
  5. Set the resource directory of our integration tests to src/integration-test/resources.

When we are done, our build.gradle file should have the following sourceSets build script block right after the repositories build script block:

sourceSets {
    integrationTest {
        java {
            compileClasspath += main.output + test.output
            runtimeClasspath += main.output + test.output
            srcDir file('src/integration-test/java')
        }
        resources.srcDir file('src/integration-test/resources')
    }
}
Sometimes you might not want that integration tests depend from unit tests. For example, you might want to use two different implementations of the same API. If this is the case, you should take a look at Jukka's comment. Also, I recommend that you take a look at Jonathan's comment that describes the difference between files() and file() methods.

Additional Reading:

When we run the command: gradle properties at the command prompt, we will see a long list of the project’s properties. The properties that are relevant for this blog posts are shown in the following:

> gradle properties
:properties

------------------------------------------------------------
Root project
------------------------------------------------------------
configurations: [configuration ':archives', configuration ':compile', configuration ':default', configuration ':integrationTestCompile', configuration ':integrationTestRuntime', configuration ':runtime', configuration ':testCompile', configuration ':testRuntime']

sourceSets: [source set 'integration test', source set 'main', source set 'test']
sources: [Java source 'main:java', JVM resources 'main:resources', Java source 'test:java', JVM resources 'test:resources', Java source 'integrationTest:java', JVM resources 'integrationTest:resources']

BUILD SUCCESSFUL

Total time: 3.34 secs

As we can see, we added a new source and resource directories to our Gradle build. The interesting this is that when we created a new source set, the Java plugin added two new dependency configurations to our build:

  • The integrationTestCompile configuration is used to declare the dependencies that are required when our integration tests are compiled.
  • The integrationTestRuntime configuration is used to declare the dependencies that are required to run our integration tests. This configuration contains all dependencies that are added to the integrationTestCompile configuration.

Let’s move and find out what kind of configuration changes we have to make before these dependency configurations are useful to us.

Configuring the Dependency Configurations of Our Integration Tests

When we configured the source and resource directories of our integration tests, we created a source set that created two new dependency configurations: integrationTestCompile and integrationTestRuntime. The problem is that these configurations do not contain the dependencies of our unit tests.

We could solve this problem by adding the required dependencies to these configurations, but we won’t do that because adding duplicate configuration is an awful idea. Instead we will configure these dependency configurations by following these steps:

  1. Ensure that the integrationTestCompile configuration contains the dependencies that are required to compile our unit tests.
  2. Ensure that the integrationTestRuntime configuration contains the dependencies that are required to run our unit tests.

We can make these changes by using the configurations build script block. In other words, we must add the following code to our build.gradle file between the sourceSets and the dependencies build script blocks:

configurations {
    integrationTestCompile.extendsFrom testCompile
    integrationTestRuntime.extendsFrom testRuntime
}

We can now add dependencies to these configurations. For example, if we want to use AssertJ 3.0 in our integration tests, we have to add the assertj-core dependency to the integrationTestCompile configuration. After we have done this, the dependencies build script block found from our build.gradle file looks as follows:

dependencies {
    compile 'log4j:log4j:1.2.17'
    testCompile 'junit:junit:4.11'
    integrationTestCompile 'org.assertj:assertj-core:3.0.0'
}

Our next step is to create the task that runs our integration tests. Let’s find out how we can do that.

Creating the Task That Runs Our Integration Tests

We can create the task that runs our integration tests by following these steps:

  1. Create a new task called integrationTest and set its type to Test.
  2. Configure the location of the compiled test classes.
  3. Configure the classpath that is used when our integration tests are run.

We can create and configure the integrationTest task by adding the following code to our build.gradle file:

task integrationTest(type: Test) {
    testClassesDirs = sourceSets.integrationTest.output.classesDirs
    classpath = sourceSets.integrationTest.runtimeClasspath
}
David pointed out that Gradle skips tasks whose input and output are up to date. If you want to ensure that your integration tests are run every time when you run the integrationTest task, you have to tell Gradle that the outputs of the integrationTest task should always be considered out of date.

A Gradle task has a property called outputs, and the type of this property is TaskOutputs. If you want that the outputs of the integrationTest task are always considered out of date, you have to ensure that the upToDateWhen() method of the TaskOutputs interface always returns false. After you have done this, the declaration of the integrationTest task looks as follows:

task integrationTest(type: Test) {
    testClassesDirs = sourceSets.integrationTest.output.classesDirs
    classpath = sourceSets.integrationTest.runtimeClasspath
    outputs.upToDateWhen { false }
}

Additional Reading:

We have created the task that runs our integration tests, but the problem is this task is not invoked during our build. Because want to include it in our build, we have to follow these steps:

  1. Ensure that our integration tests are run before the check task and that the check task fails the build if there are failing integration tests.
  2. Ensure that our unit tests are run before our integration tests. This guarantees that our unit tests are run even if our integration tests fails.

We can do these configuration changes by adding the following lines to our build.gradle file:

check.dependsOn integrationTest
integrationTest.mustRunAfter test

We are done! Let’s move on and find out how we can run our tests.

Running Our Tests

We have now created a new task that runs our integration tests and integrated that task with our Gradle build. We are finally ready to run our unit and integration tests. The requirements of our Gradle build states that:

  • We must be able to run our only unit tests.
  • We must be able to run only integration tests.
  • We must be able to run all tests.

Let's go through these requirements one by one.

First, if we want to run only unit tests, we can use one of these two options:

  • We can run our unit tests by running the command: gradle clean test at the command prompt.
  • We can run our build and exclude integration tests by running the command: gradle clean build -x integrationTest at the command prompt.

Second, if we want to run only integration tests, we can choose one of the following options:

  • We can run our integration tests by running the command: gradle clean integrationTest at the command prompt.
  • We can run our build and exclude unit tests by running the command: gradle clean build -x test at the command prompt.

Third, if we want to run all tests, we can use one of these two options:

  • We run our unit and integration tests by running the command: gradle clean test integrationTest at the command prompt.
  • We can run our build by running the command: gradle clean build at the command prompt.

When we run our tests, Gradle creates the HTML reports of our unit and integration tests to the following directories:

  • The build/reports/tests/integrationTest directory contains the HTML report that contains the test results of our integration tests.
  • The build/reports/tests/test directory contains the HTML report that contains the test results of our unit tests.

Let’s summarize what we learned from this blog post.

Summary

This blog post has taught us the following things:

  • If we add a new source set to our build, the Java plugin creates the compile time and runtime dependency configurations for it.
  • We can include the dependencies of an another dependency configuration by using the extendsFrom property of the Configuration.
  • We can create a task that run our integration tests by creating a new Test task, and configuring the location of the integration test classes and the used classpath.
  • We can add dependencies to a task and configure the order in which our tasks are invoked.
  • We can exclude tasks by using the -x command-line option.

P.S. You can get the example application of this blog post from Github.

P.P.S. You should take a look at the Gradle TestSets plugin. It allows you to specify additional test sets and add these test sets to your build.

If you want learn how to use Gradle for running unit, integration, and end-to-end tests, which ensure that your Spring web application is working as expected, take a look at my upcoming Test With Spring Course.
83 comments… add one
  • David Karr May 6, 2015 @ 18:36

    Glad to see articles like this, but I have a few comments.

    First, concerning what test failures should fail the build:

    It is very important that unit tests are run every time the build is run, and the failure of any unit test should fail the "build" task. However, your configuration here will always run all the integration tests every time the build is run, and the failure of any integration test will also fail the "build" task. It's important for integration tests to be run as often as possible, but their failure shouldn't be tied to the regular build. Integration tests by their nature are fragile. These tests could fail for reasons outside of the developer's control, and they are slow.

    In short, integration tests should not be tied to the "build" task.

    One point about "running just the integration tests". Note that Gradle only runs tasks whose inputs or output are out of date. If you run the task that runs your integration tests, it should also build the software and run the unit tests. If none of those things are out of date, it will just skip them. If they ARE out of date, attempting to only run the integration tests is just asking for trouble.

    As I believe that integration tests are somewhat fragile, and can depend on things outside of the direct application dependencies, I believe that integration tests should always run, if you run the task itself. Out of the box, if no inputs or outputs have changed, integration tests will be skipped, even if you run the task directly. To make integration tests always run, I add a block like this:

    project.integTest {
    	// This forces integration tests to always run if the task is run.
    	outputs.upToDateWhen { false }
    }
    


    And to restate my point about not tying integration tests to the "build" task, I also add the following:

    // Makes integration tests not run by default.
    check.dependsOn -= integTest
    


    Finally, I believe that all the details you described for setting up integration tests are correct (except for that one detail that I described earlier). It's useful to note that someone has already packaged all of these details into a Gradle plugin (although I'm not certain the reporting separation was done). The "testsets" plugin, at "org.unbroken-dome.test-sets" does this.

    With this in place, you only have to include the following block:

    testSets {
    	integTest
    }
    


    This creates appropriate configurations and tasks with "integTest" in each name, setting up appropriate relationships.

    • Petri May 6, 2015 @ 21:45

      Hi David,

      Thank you from your comment! I don't have time to answer to it right now, but I wanted to let you know that I will read it again later this week and write a proper answer to it. It seems that I will have to make some changes to my build script.

      By the way, I am really grateful that you took the time to point out my mistakes!

    • Petri May 7, 2015 @ 21:03

      Hi David,

      It is very important that unit tests are run every time the build is run, and the failure of any unit test should fail the “build” task. However, your configuration here will always run all the integration tests every time the build is run, and the failure of any integration test will also fail the “build” task. It’s important for integration tests to be run as often as possible, but their failure shouldn’t be tied to the regular build.

      Integration tests by their nature are fragile. These tests could fail for reasons outside of the developer’s control, and they are slow.

      In short, integration tests should not be tied to the “build” task."

      That is true. My configuration runs the integration tests when the build task is run. The reason for this is that when I am writing code, i rarely run the full build (i.e. invoke the build task).

      If I want to run my example application, I run the command gradle clean run at the command prompt. When I look the output of this command, I notice that the build task is not invoked:

      
      > gradle clean run
      :clean
      :compileJava
      :processResources
      :classes
      :run
      INFO  - HelloWorld                 - Received message: Hello World!
      
      BUILD SUCCESSFUL
      
      Total time: 3.528 secs
      
      

      If I want to build the binary distribution, I can do that by running the command gradle clean distZip at the command prompt. Once again, the build task is not invoked:

      
      > gradle clean distZip
      :clean
      :copyLicense
      :compileJava
      :processResources
      :classes
      :jar
      :startScripts
      :distZip
      
      BUILD SUCCESSFUL
      
      Total time: 3.875 secs
      
      

      If I am writing a web application, I can start it by running command: gradle clean appStart at the command prompt. When I do this, I see that the build task is not invoked:

      
      Petris-MacBook-Pro:web-application loke$ gradle clean appStart
      :clean
      :compileJava
      :processResources
      :classes
      :prepareInplaceWebAppClasses
      :prepareInplaceWebAppFolder
      :prepareInplaceWebApp
      :appStart
      
      Run 'gradle appStop' to stop the server.
      
      

      In other words, it seems that my tests are run only when I invoke the test tasks (either test or integrationTest) or I run the full build. I agree that integration tests are slow and fragile, but because I run the full build only if I want to run it, I don't see why it is harmful to ensure that the integrationTest task fails the full build. Am I missing something?

      One point about “running just the integration tests”. Note that Gradle only runs tasks whose inputs or output are out of date. If you run the task that runs your integration tests, it should also build the software and run the unit tests. If none of those things are out of date, it will just skip them. If they ARE out of date, attempting to only run the integration tests is just asking for trouble.

      If they are outdated, doesn't Gradle run the required tasks before it runs integration tests? I tested this with the example application by following these steps:

      1. Run command: gradle clean integrationTest
      2. Run command: gradle integrationTest
      3. Modify the MessageService class to return 'Hello World!1'.
      4. Run command: gradle integrationTest

      When I run the integration tests for the last time, I see the following output:

      
      Petris-MacBook-Pro:integration-tests loke$ gradle integrationTest
      :compileJava
      :processResources UP-TO-DATE
      :classes
      :compileTestJava
      :processTestResources UP-TO-DATE
      :testClasses
      :compileIntegrationTestJava
      :processIntegrationTestResources UP-TO-DATE
      :integrationTestClasses
      :integrationTest
      
      net.petrikainulainen.gradle.MessageServiceIT > getMessage_ShouldReturnMessage FAILED
          org.junit.ComparisonFailure at MessageServiceIT.java:22
      
      1 test completed, 1 failed
      :integrationTest FAILED
      
      

      It looks like the MessageService class is compiled and the integration test fails. Is there something that I don't see?

      It seems that the unit tests aren't invoked if I run the integrationTest task, but I am not sure if that is a problem (I have to update the blog post though).

      As I believe that integration tests are somewhat fragile, and can depend on things outside of the direct application dependencies, I believe that integration tests should always run, if you run the task itself. Out of the box, if no inputs or outputs have changed, integration tests will be skipped, even if you run the task directly.

      I agree. This is annoying. Thank you for pointing this out. I will update the blog post and the example application.

      It’s useful to note that someone has already packaged all of these details into a Gradle plugin.

      I agree. I will add a note about this to the blog post.

  • Geoffroy Warin May 7, 2015 @ 21:26

    Hi Petri,

    Thank you for your article and thanks David for pointing out the testsets plugin.

    Just a little precision, when you write the sourceSets declaration, it seems that the srcDirs declarations for the sources and the resources are optional.
    By convention, it seems that gradle use the name of the source set to create add the appropriate directories to the classpath.

    For instance, if you name your sourceSet "toto", it would add "src/main/toto/java" and "src/main/toto/resources" to the classpath. It will even consider "src/main/toto/groovy" if you use the groovy plugin, which is handy.

    • Petri May 7, 2015 @ 22:15

      Hi,

      Just a little precision, when you write the sourceSets declaration, it seems that the srcDirs declarations for the sources and the resources are optional. By convention, it seems that gradle use the name of the source set to create add the appropriate directories to the classpath.

      How did you declare your source set? I tried this with the example application of this blog post, and I couldn't get it to work.

      First, I declared my source set as follows:

      
      sourceSets {
          integrationTest {}
      }
      
      

      When I ran my integration tests, nothing was compiled:

      
      > gradle clean integrationTest
      :clean UP-TO-DATE
      :compileIntegrationTestJava UP-TO-DATE
      :processIntegrationTestResources UP-TO-DATE
      :integrationTestClasses UP-TO-DATE
      :integrationTest UP-TO-DATE
      
      BUILD SUCCESSFUL
      
      Total time: 2.839 secs
      
      

      Second, I tried this declaration:

      
      sourceSets {
          integrationTest {
              java {
                  compileClasspath += main.output + test.output
                  runtimeClasspath += main.output + test.output
              }
          }
      }
      
      

      When I ran my integration tests, Gradle compiled my application and the unit test classes:

      
      > gradle clean integrationTest
      :clean UP-TO-DATE
      :compileJava
      :processResources
      :classes
      :compileTestJava
      :processTestResources UP-TO-DATE
      :testClasses
      :compileIntegrationTestJava UP-TO-DATE
      :processIntegrationTestResources UP-TO-DATE
      :integrationTestClasses UP-TO-DATE
      :integrationTest UP-TO-DATE
      
      BUILD SUCCESSFUL
      
      Total time: 4.311 secs
      
      

      I would love to get rid of the directory declarations because they seem a bit redundant. If you know how to fix my build script, let me know, and I will update this blog post and the example application.

    • Petri May 9, 2015 @ 10:31

      I just realized that you were probably talking about the Gradle TestSets plugin. I will write a new blog post that describes how you can simplify your build script with that plugin.

      • Geoffroy Warin May 10, 2015 @ 20:47

        Nope, here is the full configuration:

        
        configurations {
            integrationTestCompile.extendsFrom testCompile
            integrationTestRuntime.extendsFrom testRuntime
        }
        
        sourceSets {
            integrationTest {
                compileClasspath += main.output + test.output
                runtimeClasspath += main.output + test.output
            }
        }
        
        task integrationTest(type: Test) {
            testClassesDir = sourceSets.integrationTest.output.classesDir
            classpath = sourceSets.integrationTest.runtimeClasspath
            reports.html.destination = file("${reporting.baseDir}/${name}")
        }
        
        
        • Petri May 12, 2015 @ 20:55

          For some reason I still get the same output than before. Which Gradle version are you using? I am using Gradle 2.4.

  • Siarhei Skavarodkin Jun 6, 2015 @ 17:04

    Hi Petri,

    Nice article. Thank you a lot. Doing the stuff as described I encountered the issue as follows:
    1. when I run "gradle clean build" from console everything is fine
    2. however if I run the same in Intellij Idea 14.1.3 debug mode, then I receive:

    11:00:45: Executing external tasks 'clean build'...
    :clean
    Note: Some input files use unchecked or unsafe operations.
    Note: Recompile with -Xlint:unchecked for details.
    :compileJava
    :processResources
    :classes
    :jar
    :startScripts
    :distTar
    :distZip
    :assemble
    :compileTestJava
    :processTestResources UP-TO-DATE
    :testClasses
    Connected to the target VM, address: '127.0.0.1:2585', transport: 'socket'
    Disconnected from the target VM, address: '127.0.0.1:2585', transport: 'socket'
    :test
    :compileIntegrationTestJava
    :processIntegrationTestResources UP-TO-DATE
    :integrationTestClasses
    :integrationTest
    FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
    ERROR: transport error 202: connect failed: Connection refused
    ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
    JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [debugInit.c:750]
    Could not write standard input into: Gradle Test Executor 2.
    java.io.IOException: Идет закрытие канала
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:315)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOutputHandleRunner.java:50)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
    at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    :integrationTest FAILED

    FAILURE: Build failed with an exception.

    * What went wrong:
    Execution failed for task ':integrationTest'.
    > Process 'Gradle Test Executor 2' finished with non-zero exit value 1

    * Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

    BUILD FAILED

    Total time: 3.881 secs
    Process 'Gradle Test Executor 2' finished with non-zero exit value 1
    11:00:50: External tasks execution finished 'clean build'.

    I'm just wondering if you (or anyone) encountered such issue and if there is a way to debug Integration Tests from IDEA ?

    Best regards, Siarhei

    • Petri Jun 6, 2015 @ 18:26

      Hi Siarhei,

      Thank you for your kind words. I really appreciate them. Also, I recommend that you read my blog post titled: Getting Started With Gradle: Integration Testing With the TestSets Plugin. It describes how you can add integration tests to your Gradle build by using the Gradle TestSets plugin and remove unnecessary clutter from your build.gradle file.

      I was able to reproduce your problem and tried to find a solution to it. I found a few StackOverflow questions that looked promising:

      In other words, it seems that this problem might be related to your network configuration. I would check the following things:

      • Is the port used by another process or blocked by your firewall?
      • Can you ping the address 127.0.0.1?

      I haven't been able to solve this problem yet, but I will update my answer if I happen to find a solution to it.

      • Siarhei Skavarodkin Jun 8, 2015 @ 17:07

        Hi Petri,

        Thank you for the update. I will take a look later on. If I manage to find a solution I will post it there as well.

        Best Regards, Siarhei

  • bhanu Nov 10, 2015 @ 10:43

    C:\Applications\GradleTests>gradle clean build
    :clean
    :compileJava
    :processResources
    :classes
    :jar
    :assemble
    :compileTestJava
    :processTestResources
    :testClasses
    :test
    :compileIntegrationTestJava

    FAILURE: Build failed with an exception.

    * What went wrong:
    Could not resolve all dependencies for configuration ':integrationTestCompile'.
    > Could not resolve org.assertj:assertj-core:2.1.0..
    Required by:
    :GradleTests:1.0
    > Could not GET 'https://repo1.maven.org/maven2/org/assertj/assertj-core/2.1.0./assertj-core-2.1.0..pom'.
    > Connection to https://repo1.maven.org refused

    * Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

    BUILD FAILED

    • xhi2018 Nov 11, 2015 @ 22:38

      Hi bhanu,
      may you mistyped the dependencies configuration like this in your build.gradle script..?

      dependencies {
      integrationTestCompile 'org.assertj:assertj-core:2.1.0.'
      }

      see the tailing dot '.' in the version nummer. Can you post your dependencies configuration.

      • Petri Nov 11, 2015 @ 23:34

        It might also be a some kind of network problem. This line:

        
        > Connection to https://repo1.maven.org refused
        
        

        suggests that Gradle cannot connect to the Maven repository because the connection is refused.

  • Trevor Nov 10, 2015 @ 23:33

    I am using gradle version 2.8. I am getting a message that says could not find property ‘integrationTests’ on root project ‘foo’.

    any thoughts?

    • Petri Nov 11, 2015 @ 23:36

      Hi Trevor,

      I haven't tested my example with Gradle 2.8. I will take a look at this tomorrow and let you know if I face the same problem.

      • Petri Nov 12, 2015 @ 20:58

        I tested my example with Gradle 2.8 and I didn't notice any problems. Can you post your build.gradle file?

        • Trevor Nov 12, 2015 @ 22:19

          Thanks Petri. For looking into this. I am on a closed network. I will try to type it out for you. At a high level this is my build.gradle file though. Apologize in advance for typos.

          
          apply plugin: 'spring-boot'
          apply plugin: 'idea'
          
          configurations{jaxb}
          
          repositories{....}
          
          dependencies {.....}
          
          def jaxbtargetDir = file("....")
          
          task jaxb(dependsOn: folderClean){....ant.xjc code here}
          
          idea{
            module{
             sourceDirs += add generated folder here
             generatedSourceDirs += add generated folder here
           }
          }
          
          sourceSets{
            main{
              java {
              srcDir 'src/main/generated'   
              }
            }
          
           integrationTest {
                  java {
                      compileClasspath += main.output + test.output
                      runtimeClasspath += main.output + test.output
                      srcDir file('src/integration-test/java')
                  }
                  resources.srcDir file('src/integration-test/resources')
              }
          }
          
          configurations{
            integrationTestCompile.extendsFrom testCompile
            integrationTestRuntime.extendsFrom testRunTime
          }
          
          check.dependsOn integrationTests
          integrationTests.mustRunAfter test
          
          task integrationTests(type: Test){
            testClassesDir = sourceSets.integrationTest.output.classDir
            classpath = sourceSets.integrationTest.runtimeClasspath
          }
          
          tasks.withType(Test){
          ....
          }
          
          compileJava.dependsOn jaxb
          
          task wrapper(type:Wrapper){
            gradleVersion = '2.3'
          }
          
          
          • Petri Nov 12, 2015 @ 23:10

            Hi Trevor,

            It's quite late here, and I need to get some sleep. I will take a look at this tomorrow.

          • Petri Nov 14, 2015 @ 0:42

            Hi Trevor,

            I have to admit that I am not entirely sure what your problem is.

            Have you tried renaming your integration test task to integrationTest? This probably doesn't make any difference but that was the name I used in my example.

            Which command did you use when you tried to run your integration tests?

          • Trevor Nov 16, 2015 @ 15:17

            No worries Petri. Thank you for your assistance. I will try to look further into the problem and report back with my findings.

          • Juho Dec 1, 2015 @ 16:32

            Hi,
            If you have a multi-project build with subproject 'bar' and this is the contents of $rootDir/bar/build.gradle, then running ./gradlew integrationTests should generate
            Task 'integrationTests' not found in root project 'foo'. since integration test task is not configured for root project.

            Try running ./gradlew :bar:integrationTests instead.

          • Jeshan Babooa Dec 1, 2015 @ 17:06

            The project compiles successfully on my machine.
            If you omit the 's' at the end of "integrationTests", then you get the error above.

            Trevor: May I suggest that you copy the build.gradle first and then adapt it to your needs?

          • Jeshan Babooa Dec 2, 2015 @ 19:37

            What Juho said is probably it. (I didn't notice his comment before posting mine!)

          • Petri Dec 2, 2015 @ 23:22

            Thanks guys! I will try to contact Trevor and hopefully we will see if these suggestions solves his problem.

          • Trevor Mar 2, 2016 @ 16:41

            @Juho - That was the problem exactly. Thanks so much everyone for your help.

  • Jim C Dec 21, 2015 @ 22:22

    This works great for me but with one small exception. It ignores "--continue" if I want to force the build to ignore test failures. Any ideas for a fix?

    • Petri Dec 23, 2015 @ 23:27

      I have confess that I don't how to fix this problem. I will share your question with my Twitter followers and hope that someone knows the answer to your question.

    • n3bu Jan 5, 2016 @ 13:27

      hello jim,

      the following works for me (inside of a task):
      ignoreFailures = true

      @petri: I like your article too, rock on and keep it up! :)

      • Petri Jan 5, 2016 @ 20:39

        Thank you for answering to Jim's question. I really appreciate it :) Also, it was nice to hear that you liked this blog post.

  • Amuthan Mar 1, 2016 @ 21:29

    Hi...

    After introducing integration sourceset everything working fine in gradle world, but when I try to import the project into eclipse using eclipse plugin (gradlew Eclipse) eclipse class path is not updated properly I guess. so my question is how to make eclipse to know about your new sourceset?

    • Petri Mar 1, 2016 @ 22:07

      Hi Amuthan,

      Unfortunately I don't know the answer to your question because I haven't been using Eclipse for six years (could be more, I cannot remember for sure).

    • David May 10, 2016 @ 21:02

      Amuthan - I am hitting the same problem and wondering if you ever got around it?

      • Ikharus Jul 5, 2016 @ 18:46

        Copy of my answer on StackOverflow:

        Adding this:

        eclipse {
        classpath {
        plusConfigurations.add configurations.integrationTestCompile
        plusConfigurations.add configurations.integrationTestRuntime
        }
        }
        to the gradle file solved the problem. I hope it does the same for you.

        See http://stackoverflow.com/a/38207361/4336562

        • Álvaro Mar 25, 2020 @ 11:44

          Thank you VERY MUCH for posting this

  • Nico Apr 19, 2016 @ 14:45

    Great Post! Easy to understand and simple to execute. Exactly what I was looking for :)

    • Petri Apr 19, 2016 @ 20:32

      Thank you for your kind words. I really appreciate them. Also, you might want to take a look at the Gradle TestSets plugin. It helps you to get rid of unnecessary configuration.

  • anurag kulkarni Jul 21, 2016 @ 2:29

    Hi,

    Excellent post. However, I have run into a problem.
    I would like to know, how to add test config files to integration tasks from the spring cloud config? Do we have to set a separate property or can we add them into the jvmArgs for the tests to run?

    Cheers,
    Anurag

    • Petri Jul 21, 2016 @ 12:44

      Hi Anurag,

      Unfortunately I haven't used Spring Cloud => I don't know how you can solve your problem.

  • Jukka Nov 2, 2016 @ 6:58

    Hi Petri,

    thanks for good guide. Seems to be the only (correct) one on the internet.

    I noticed that integration tests shouldn't generally depend on unit test. Usually they might, but not always.

    For example if I unit test JAX-RS client I depend on "jersey-test-framework-provider-inmemory" artifact. And integration-test there should be real container (eg. "jersey-test-framework-provider-jetty") but that in-memory implementation can't exists (or it messes up things).

    So maybe you have to specify more of your requirement: "* We must be able to configure compile time and runtime dependencies for our integration tests." to depend only main (sources and dependencies) by default.

    • Jukka Nov 2, 2016 @ 7:15

      Hi Petri again,

      in my case I need that only integrationTestRuntime doesn't extend testRuntime. So think this "runtimeClasspath += main.output" fixes it. (Not tested yet.)

      • Petri Nov 2, 2016 @ 12:10

        Hi Jukka,

        Good point. I will add a note about this to the blog post and include a link to your comment => hopefully your comment will help other people who have the same requirements as you do.

  • Richard Nov 2, 2016 @ 11:13

    Hi,
    Great article, does exactly what I wanted - I'm developing Java client libraries around a web service and wanted to separate real calls to web service from plain vanilla units. Thanks!

    • Petri Nov 2, 2016 @ 12:05

      Hi Richard,

      Thank you for your kind words. I really appreciate them. Also, it was nice to hear that this blog post was useful to you.

  • Jonathan Feb 19, 2017 @ 1:28

    While not strictly needed, the srcDirs = files("integration-test") is a bit more precise than srcDir = file("integration-test"). It replaces / sets the value for the srcDirs folder rather than add one.

    The srcDir option adds to the existing list based on the convention: sourceSets.integTest.java.srcDirs [/Users/yoni/dev/training/classes/gradle/play/src/integTest/java, /Users/yoni/dev/training/classes/gradle/play/integration-test]

    • Jonathan Feb 19, 2017 @ 1:48

      Actually a correction is in order:

      srcDirs = files(“integration-test”) replaces the value of srcDirs whereas the src file("integration-test") adds a source directory.

      • Petri Mar 4, 2017 @ 11:35

        Hi,

        Thank you for pointing this out! I have to admit that I don't use the technique described in this post anymore because I found the Gradle TestSets plugin, but your comment made me understand the difference between these two options.

  • Oleg Sep 7, 2017 @ 0:32

    Hi Petri
    Thanks for the great post and very good blog. I'm your reader for many years. And I have some questions.
    In my project (Spring Boot Web app backend), I have three packages of tests. "tests" for unit tests, "tests-integration" and "tests-functional" - here I start the full app context and test some business tests through my app REST API. For example, client register himself, pay a monthly fee for subscription and get access to some goods (e-books for example).
    1) Is it possible to run integration and functional tests in parallel? I use an embedded database. That's why in theory they could go in parallel.
    2) What is the best approach to change clock time during tests? In my case I would like to test, if a user is come back after one month and didn't pay next monthly fee, then his access to goods is expired. Currently, I have endpoint in my production code, where I can change app clock time (of course this endpoint work only with active profile test)
    3) What is the best approach to get Schedulers work during tests? Same case, I have a schedule which is running at the end of the month. Scheduler checks extend client subscription or no. Here I have some workaround as well. I have an endpoint in prod. code by which I can execute scheduler.
    4) What do you think about Contract testing (Consumer Driven Contracts) in microservices? Tools like Pact and Spring Cloud Contracts
    5) Kotlin Spek is next Groovy Spock : ) ? Does Spock have future?
    Thnx so much for everything you do!

    • Petri Sep 7, 2017 @ 21:54

      Hi Oleg,

      Thank you for your kind words. I really appreciate them. I will answer to your questions in the following:

      Is it possible to run integration and functional tests in parallel? I use an embedded database. That’s why in theory they could go in parallel.

      I have to admit that I don't know if this is possible. It seems that it is possible to run the test task in parallel but I don't if you can actually run two different tasks in parallel.

      What is the best approach to change clock time during tests? In my case I would like to test, if a user is come back after one month and didn’t pay next monthly fee, then his access to goods is expired. Currently, I have endpoint in my production code, where I can change app clock time (of course this endpoint work only with active profile test)

      Typically I create a DateTimeService interface and write two implementations for this interface: the first implementation returns current time and the second one returns constant time (always the same time). Then I just register two beans and use the latter bean when I run my tests. This works well for me, but if you need to change the time between your tests, your solution is good as well.

      What is the best approach to get Schedulers work during tests? Same case, I have a schedule which is running at the end of the month. Scheduler checks extend client subscription or no. Here I have some workaround as well. I have an endpoint in prod. code by which I can execute scheduler.

      I never run my schedulers when I run my tests because they can mess up my test data and cause false positives. However, if you absolutely have to run a scheduled task, I would probably just invoke the method which "starts" the tasks. Obviously, this doesn't work if you are using Spring Batch. Also, if I am writing tests for the scheduled task, I simply invoke the method which runs the task. Again, if you use Spring Batch, it should have a testing lib which makes it easier to write tests for Spring Batch jobs.

      What do you think about Contract testing (Consumer Driven Contracts) in microservices? Tools like Pact and Spring Cloud Contracts

      I have read maybe two blog posts about this topic => I don't really have an opinion about it yet.

      Kotlin Spek is next Groovy Spock : ) ? Does Spock have future?

      I think that Spock has a very bright future as long as people are writing their applications with Groovy. Most Java developers I know are still using JUnit 4, and that is why I assume they will start using JUnit 5 instead of Kotlin Spek or Spock (I just mean that Spock was never that popular among Java devs). That being said, Kotlin Spek might might replace Spock if you need to write tests for Java apps and you don't want to use Java for testing (this is just a hunch).

  • Stefan Nov 21, 2017 @ 15:37

    Hi Petri,

    thanks for this great article. I just try to get different test source sets used by the junit5-gradle-plugin, but unforunately tests in a new source set integration-tests won't get executed.
    Do you plan to provide an update of the setup for Gradle using JUnit 5?
    Thanks in advance
    Stefan

    • Petri Nov 22, 2017 @ 21:57

      Hi Stefan,

      I answered to your question here.

  • Lucas Saldanha Apr 5, 2018 @ 6:42

    Hi Petri,

    When creating the integrationTest task in Gradle > 4.0, the `testClassesDir` and `x.output.classesDir` are deprecated.

    For newer Gradle versions the correct is:

    
    task integrationTest(type: Test) {
        testClassesDir = sourceSets.integrationTest.output.classesDir
        classpath = sourceSets.integrationTest.runtimeClasspath
        outputs.upToDateWhen { false }
    }
    
    
    • Lucas Saldanha Apr 5, 2018 @ 6:43

      Sorry, I posted the wrong snippet. The correct one is:

      
      task integrationTest(type: Test) {
        testClassesDirs = sourceSets.integrationTest.output.classesDirs
        classpath = sourceSets.integrationTest.runtimeClasspath
        outputs.upToDateWhen { false }
      }
      
      
      • Petri Apr 5, 2018 @ 21:42

        Hi Lucas,

        Good catch and thank you for reporting this. I will this problem during the next weekend and update this blog post.

  • Lucas Saldanha Apr 5, 2018 @ 7:02

    Hi Petri,

    Another deprecation warning. When setting the output folder for the test reports, instead of:

    
    reports.html.destination = file("${reporting.baseDir}/${name}")
    
    

    One should use:

    
    reports.html.setDestination file("${reporting.baseDir}/${name}")
    
    

    Cheers!

    • Petri Apr 5, 2018 @ 21:43

      Hi Lucas,

      Again, thank you for reporting this. I will fix this problem during the next weekend and update this blog post.

  • Minjun Yu Apr 14, 2018 @ 23:46

    Hi Petri,

    First of all, thank you for this helpful tutorial.
    I have found that with gradle 4.51, task `reports.html.destination = file("${reporting.baseDir}/${name}")` is not necessary. The reports for IT and unit-test are already stored in separated directories.

    It would be great if you can confirm on this.
    Thank you again!!

    Minjun

    • Petri Apr 16, 2018 @ 17:48

      Hi,

      Thank you for reporting this problem. I have updated the example application (and the blog post). Also, I ensured that the example is working with Gradle 4.6.

  • Eugene Jul 23, 2018 @ 7:03

    Hi Petri!
    Thanx a lot for your article. IT task is very important and unfortunately there is not so much information about how to setup ITs in Sring Boot app in Gradle. I made everything as you told in article but faced some issues. As i told i created Spring Boot 2 app and need to run IT on it. actually i added a very simple test there. Under debug in Intelly Idea (ver. 2018.1.4) i see the following message:
    > Task :tci:cleanIntegrationTest
    > Task :tci:compileJava
    Note: Some input files use unchecked or unsafe operations.
    Note: Recompile with -Xlint:unchecked for details.
    > Task :tci:processResources UP-TO-DATE
    > Task :tci:classes
    > Task :tci:compileTestJava UP-TO-DATE
    > Task :tci:processTestResources UP-TO-DATE
    > Task :tci:testClasses UP-TO-DATE
    > Task :tci:compileIntegrationTestJava UP-TO-DATE
    > Task :tci:processIntegrationTestResources UP-TO-DATE
    > Task :tci:integrationTestClasses UP-TO-DATE
    > Task :tci:integrationTest FAILED
    FAILURE: Build failed with an exception.
    * What went wrong:
    Execution failed for task ':tci:integrationTest'.
    > No tests found for given includes: [ru.cft.tci.processing.SimpleTest.test](filter.includeTestsMatching)

    From command line (./gradlew clean build) i see following:
    > Task :tci:integrationTest FAILED

    ru.cft.tci.TciApplicationITests > initializationError FAILED
    java.lang.Exception

    1 test completed, 1 failed

    FAILURE: Build failed with an exception.

    * What went wrong:
    Execution failed for task ':tci:integrationTest'.

    Do you have any ideas why this may happen? It seems smth. was not properly configured but i can't find out what exactly is wrong(
    Thanx in advance for any assist.

  • Eugene Jul 23, 2018 @ 7:05

    Just forgot to mention that my version of gradle is 4.8:
    $ ./gradlew --version

    ------------------------------------------------------------
    Gradle 4.8
    ------------------------------------------------------------

    Build time: 2018-06-04 10:39:58 UTC
    Revision: 9e1261240e412cbf61a5e3a5ab734f232b2f887d

    Groovy: 2.4.12
    Ant: Apache Ant(TM) version 1.9.11 compiled on March 23 2018
    JVM: 1.8.0_172 (Oracle Corporation 25.172-b11)
    OS: Windows 10 10.0 amd64

    • Petri Jul 23, 2018 @ 12:37

      Hi,

      Unfortunately it's a bit hard to say what is going on because I cannot debug your application and see your build script. It seems that Gradle cannot find your tests, but I don't know what the root cause of this issue is. That being said, have you tried using the Gradle TestSets plugin? I recommend that you take a look at it because it helps to clean up your build script.

  • Eugene Jul 29, 2018 @ 11:06

    Hi Petri! Thanx a lot for your reply. Unfortunately i can't share the project(( it's not my property. I'll try to work w/ TestSets plugin, maybe it would b helpfull somehow

    • Eugene Jul 29, 2018 @ 11:39

      It seems i can see where the issue is: it seems that 4.8th Gradle is not working properly w/ JUnit5. At least w/ JUnit4 i could load the context in my project in integration tests

      • Eugene Jul 29, 2018 @ 11:44

        Petri, what do U think about this? How to you think - was the JUnit5+Gradle issue fixed in TestSets plugin?

        • Petri Jul 31, 2018 @ 11:09

          Hi,

          Actually, I started to wonder if this problem is caused by either the Gradle integration of IntelliJ Idea or some other setting that is used in your project. The reason why I suspect this is that I updated my Gradle version to 4.8.1 and ran the integration tests of my example application (this blog post and the TestSets plugin) at command prompt. When I did this, I noticed that I was able to run my integration tests and all tests passed. I will try to import these projects to IntelliJ Idea later today. I hope that it will shed some light to this issue.

          • Eugene Jul 31, 2018 @ 13:26

            Ok, thank you! Waiting 4 your feedback

          • Petri Aug 5, 2018 @ 17:49

            Hi,

            I did some experiments and I was able to import both Gradle builds (this one and the one which uses the TestSets plugin). Also, I was able to run integration tests (I ran the integrationTest task) without running into any problems.

  • Sylvain Aug 8, 2018 @ 6:10

    Thank you very much for your blog!

  • Guillaume Oct 10, 2018 @ 16:15

    Hello !
    I think that the Gradle documentation has evolved since your post was written.
    It could be worth mentioning this paragraph :
    *Configuring integration tests* (https://docs.gradle.org/current/userguide/java_testing.html#sec:configuring_java_integration_tests).

    Thanks

    • Petri Nov 14, 2018 @ 20:52

      Hi,

      Good point. I will update this blog post and include that link in the updated blog post.

  • Durim Jan 9, 2019 @ 19:39

    Thank you for this great article

    This config didn't work for me, I have Gradle 5.0, so this config worked for me, shared as Gist:
    https://gist.github.com/durimkryeziu/e987631e1ed304582fbfa75d970d0ee6

    • Petri Jan 10, 2019 @ 20:37

      I have to admit that I haven't tested this with Gradle 5 (yet). That's why I am happy that you decided to share your solution. Thank you for sharing.

  • Garima Feb 26, 2019 @ 12:44

    Hi Petri
    I am using the same command as above for separating integration test and unit test
    while running on android studio fro ide unit test works fine but for integrationTest it is showing test not found:empty test suite
    Can you guide me in this matter

  • Alex Jul 18, 2019 @ 0:09

    SUCCESS: Executed 0 tests

    I can get a "functional" task, but I can't get it to actually run my tests. It breaks just fine when the test code won't compile though.

    • Anonymous Jul 18, 2019 @ 16:10

      I was missing useJUnitPlatform() being applied to the integration tests.

      • Petri Jul 20, 2019 @ 21:16

        It's good to hear that you were able to solve your problem. Also, I am sorry that I didn't answer to you sooner, but I am on summer holiday and I decided to take a little break from my normal responsibilities (including this blog).

  • Ilya Oct 29, 2019 @ 10:51

    Hi,

    There are simple and powerful plugin used by our company. It allow with single line to add all described in this article staff to your project.
    https://github.com/Softeq/spring-querydsl-project-template
    https://softeq.github.io/itest-gradle-plugin/

    - Ilya

  • William Hruska Oct 27, 2020 @ 10:17

    Nice.

  • KISS Jan 17, 2022 @ 18:56

    This is a bit like reinventing the wheel again and moving out of the standard. It's preferable to have the tests together at the standard folder 'test' and just make a couple a sourceSets with filters if you want to separate the execution of unit tests (*Test.java) from integration tests (*IT.java).

  • KISS2 Jun 3, 2022 @ 0:07

    Exactly same opinion as KISS, not only it's like reinventing the wheel, but this post needlessly makes thing complicated and will confuse people. Filtering is the standard.

    • What Sep 5, 2022 @ 12:00

      Confusing people? I dont think thats correct, what your stating is based on? it cant confuse people since there is no standard against which this is done. Every project and every company works differently. It depends what they have in place, this is a "good to know". If my company follow this pattern in all their projects - it would be fine.

Leave a Reply