Generating Selenium Test Reports With TestProject OpenSDK

After you have written tests which use the TestProject OpenSDK and JUnit 5, you most likely want to run them. When you run your tests, the TestProject platform generates test reports which are published on the reporting dashboard.

This blog post provides an introduction to the test reports which are generated by the TestProject platform and describes how you can customize the information that's displayed on the generated test reports.

After you have read this blog post, you:

• Understand what kind of test reports are generated by the TestProject platform when you run tests which use the TestProject OpenSDK and JUnit 5.
• Know why you should customize the test reports generated by the TestProject platform.
• Can customize the information that's displayed on the generated test reports.

Let's begin.

Introduction to TestProject OpenSDK Test Reports

When you run your tests, the TestProject agent sends test execution data to the TestProject platform. The TestProject platform generates test reports by using the test execution data and publishes the generated test reports on the reporting dashboard.

However, some test execution data is generated by the TestProject OpenSDK. This data includes the name of the project, the name of the job, and the name of the invoked test. When the TestProject OpenSDK gathers this information, it follows these rules:

• The name of the project is equal to the package name of your test class.
• The name of the job is equal to the name of your test class.
• The name of the test is equal to the name of your test method.

A test report that's generated by the TestProject platform has these sections:

• The execution history section allows you see the job execution history of the selected project.
• The browser statistics section identifies the web browsers which were used to run the tests of the invoked job.
• The general statistics section contains the basic statistics of the invoked job. You can also export the test report in PDF format by clicking the links found from this section.
• The invoked test methods section displays the test methods which were run during the invoked job.
• The steps of the selected tests section displays the steps of the selected test method. This information is gathered by the TestProject agent which sends it to the TestProject platform.

The following figure illustrates the layout of the test report generated by the TestProject platform when you run the example tests of my previous blog post:

Next, you will learn why you should customize your test reports.

Why Should You Customize Your Test Reports?

The default test reports generated by the TestProject platform are very useful, but if you want to, you can also customize them according to your requirements. For example, the default reports use technical language instead of natural language, and since one of the goals of testing is to share information with different shareholders (who are not necessarily developers or testers), you can make your test reports easier to read. You can achieve this goal by following these rules:

• Use the name of your project or the name of the tested component as the name of your project that's shown on the test report. The choice between these two options depends on the way you want to share your test results. For example, if you have a small project which doesn't have too many test classes (aka jobs), you should use the name of your project as the name of the project shown on the test report. On the other hand, if you have a big project which has multiple large components, it might make sense to use the name of tested component as the name of the project which is shown on the test report.
• Describe the tested feature and use this description as the name of the Job. If you follow this rule, you provide context for the tests which are run when the job is invoked. In other words, it's easy to see that the tests of the job ensure that the feature X is working as expected.
• Use the expected outcome as the name of the test. If you follow this rule, it's easy to figure out what went wrong when a test case fails.

Let's move on and find out how you can customize the test execution data that's generated by TestProject OpenSDK.

Customizing the Test Execution Data Generated by the TestProject OpenSDK

As you remember, the TestProject OpenSDK generates the name of the project, the name of the job, and the name of the invoked test. If you want to replace the generated values with custom values, you have to follow these steps:

1. Configure the name of the project by invoking the withProjectName() method of the DriverBuilder<T> class.
2. Configure the name of the job by annotating your test class with the @DisplayName annotation.
3. Configure the names of your tests by annotating your test methods with the @DisplayName annotation.

After you have made the required changes to your test class, its source code looks as follows:

package net.petrikainulainen.testproject.opensdk;

import io.testproject.sdk.DriverBuilder;
import io.testproject.sdk.drivers.web.ChromeDriver;
import org.junit.jupiter.api.*;
import org.openqa.selenium.chrome.ChromeOptions;

@DisplayName("Search blog posts")
class BlogSearchTest {

    private static ChromeDriver driver;

    @BeforeAll
    static void configureTestProjectOpenSDK() {
        driver = new DriverBuilder<ChromeDriver>(new ChromeOptions())
                .withCapabilities(new ChromeOptions())
                .withProjectName("TestProject OpenSDK Tutorial")
                .build(ChromeDriver.class);
    }

    @Nested
    @DisplayName("When no search results are found")
    class WhenNoSearchResultsAreFound {

        @Test
        @DisplayName("Should display an empty search result page when no search results are found")
        void shouldDisplayEmptySearchResultPage() {
            //Omitted on purpose
        }
    }

    @Nested
    @DisplayName("When one search result is found")
    class WhenOneSearchResultIsFound {

        @Test
        @DisplayName("Should display search result page that has one search result when one search result is found")
        void shouldDisplaySearchResultPageWithOneSearchResult() {
            //Omitted on purpose
        }

        @Test
        @DisplayName("Should display search result page that has the correct search result when one search result is found")
        void shouldDisplaySearchResultPageWithCorrectSearchResult() {
            //Omitted on purpose
        }
    }

    @AfterAll
    static void shutdownTestProjectOpenSDK() {
        driver.quit();
    }
}

The following figure illustrates the layout of the test report that's generated by the TestProject platform when you run your tests:

Next, you will learn to use manual reporting.

Using Manual Reporting

If you need maximum flexibility, you can use manual reporting. This technique is extremely useful if:

• You want to configure the names of JUnit 5 dynamic tests.
• You want to customize the step descriptions which are shown on a test report.

If you want to replace automatic reporting with manual reporting, you have to make the following changes to your test class:

1. Disable the automatic reporting of test methods by invoking the disableTestAutoReports() method of the Reporter class.
2. Disable the automatic reporting of steps by invoking the disableCommandReports() method of the Reporter class.
3. Use the try-with-resources statement and declare a new ClosableTestReport object by invoking the test() method of the Reporter class. When you invoke the test() method, you have to pass the name of the test as a method parameter. You must use the try-with-resources statement because it ensures that if your test code throws an exception, the test failure is reported to the TestProject platform.
4. Move the code found from your test method to the code block that follows the try-with-resources statement.
5. Identify the steps of your test case by invoking the step() method of the Reporter class. When you invoke this method, you have to pass the description of the step as a method parameter.
6. Mark your test as passed by invoking the setResult() method of the ClosableTestReport class. Remember that this method invocation must be the last line of the code block that follows the try-with-resources statement.

After you have made the required changes to your test class, its source code looks as follows:

import io.testproject.sdk.DriverBuilder;
import io.testproject.sdk.drivers.web.ChromeDriver;
import io.testproject.sdk.internal.reporting.ClosableTestReport;
import org.junit.jupiter.api.*;
import org.openqa.selenium.By;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeOptions;

import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;

@DisplayName("Manual: search blog posts")
class ManualReportingExampleTest {

    private static ChromeDriver driver;

    @BeforeAll
    static void configureTestProjectOpenSDK() {
        driver = new DriverBuilder<ChromeDriver>(new ChromeOptions())
                .withCapabilities(new ChromeOptions())
                .withProjectName("TestProject OpenSDK Tutorial")
                .build(ChromeDriver.class);

        driver.report().disableTestAutoReports(true);
        driver.report().disableCommandReports(true);
    }

    @Nested
    @DisplayName("When no search results are found")
    class WhenNoSearchResultsAreFound {

        @Test
        @DisplayName("Should display an empty search result page")
        void shouldDisplayEmptySearchResultPage() {
            try (ClosableTestReport testReport = driver.report().test("No search results: should display an empty search result page")) {
                driver.get("https://www.petrikainulainen.net/blog/");
                driver.report().step("Open the page: https://www.petrikainulainen.net/blog/");

                WebElement searchField = driver.findElement(By.id("s"));
                driver.report().step("Find the search form from the sidebar");

                searchField.sendKeys("noresults");
                searchField.sendKeys(Keys.ENTER);
                driver.report()
                        .step("Perform the search by using the search term: noresults");

                WebElement noResultElement = driver.findElement(
                        By.cssSelector(
                                ".template-search .content .post_box .archive_content"
                        )
                );
                assertThat(noResultElement.getText()).isEqualTo("No results found.");
                driver.report()
                        .step("Ensure that the search result page displays the text: No results found.");

                testReport.setResult(true);
            }
        }
    }

    //The other tests are omitted on purpuse

    @AfterAll
    static void shutdownTestProjectOpenSDK() {
        driver.quit();
    }
}

The following figure illustrates the layout of the test report that's generated by the TestProject platform when you run your tests:

You understand what kind of test reports are generated by the TestProject platform when you run your tests, and you know how you can customize the generated test reports. Let's summarize what you learned from this blog post.

Summary

This blog post has taught you nine things:

• Because the goal of testing is to share information with different shareholders, you should make your test reports as easy to read as possible by replacing technical language with natural language.
• You should configure the name of the project by invoking the withProjectName() method of the DriverBuilder<T> class.
• You should configure the name of the job by annotating your test class with the @DisplayName annotation.
• You should configure the names of your "normal" (not dynamic) tests by annotating your test methods with the @DisplayName annotation.
• You should use manual reporting if you want to report the test names of JUnit 5 dynamic tests or you want to customize the step descriptions shown on a test report.
• If you are using manual reporting, you must disable the automatic reporting of test methods and test steps.
• If you are using manual reporting, you must use the try-with-resources statement when you declare a new ClosableTestReport object because the try-with-resources statement ensures that if your test code throws an exception, the test failure is reported to the TestProject platform.
• If you are using manual reporting, you can identify the steps of your test case by invoking the step() method of the Reporter class.
• If you are using manual reporting, you can mark your test as passed by invoking the setResult() method of the ClosableTestReport class.

P.S. You can get the example application of this blog post from Github.