How to View Multiple JUnit Assertion Errors in One Test Execution

By IT Admin | June 2, 2020
JUnit assertion errors: How To View Multiple Errors In One Test Execution

Many automated testers use the JUnit Assert library to perform test validations. 

When JUnit Assertion errors (test failures) occur, your test will stop executing and will not perform any remaining Assertions. For tests containing only a single Assertion, as is often the case, this is not an issue. But for tests that include multiple Assertions, you are limited in that your test will not run any Assertions after the first Assertion failure.

The problem becomes more pronounced the more Assertions your test contains. If you are validating hundreds of rows in an Excel sheet or a database table, it would make sense to see all of the Assertion failures at the end of the test execution, not just the first failure and then having to re-execute your test to see if there were any other failures one at a time. This is a limitation of vanilla JUnit Asserts. In this post, we will explore a solution to this problem.

Consider the following Cucumber Scenario example which validates the presence of multiple related fields on a UI:

1
2
3
4
5
6
 
  Scenario: Multiple field validations
    Given I navigate to the Accounts page
    When  I click on the "Create New Account" button
    Then  I verify the username field is present
    And   I verify the password field is present
    And   I verify the confirm password field is present

Here is the code behind our example step definitions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 
public class ExampleSteps {

    @When("^I click on the \"([^\"]*)\" button$")
    public void i_click_on_the_button(String buttonName) {
        //do nothing, example
    }

    @Given("I navigate to the Accounts page")
    public void i_navigate_to_the_accounts_page() {
        //do nothing, example
    }

    @Then("I verify the username field is present")
    public void i_verify_the_username_field_is_present() {
        //will result in an Assertion Error
        WebElement usernameFieldExample = null;
        Assert.assertNotNull("The username field should be present", usernameFieldExample);
    }

    @And("I verify the password field is present")
    public void i_verify_the_password_field_is_present() {
        //will result in an Assertion Error
        WebElement passwordFieldExample = null;
        Assert.assertNotNull("The password field should be present", passwordFieldExample);
    }

    @And("I verify the confirm password field is present")
    public void i_verify_the_confirm_password_field_is_present() {
        //will result in an Assertion Error
        WebElement confirmPasswordFieldExample = null;
        Assert.assertNotNull("The confirm password field should be present", confirmPasswordFieldExample);
    }
}

If we run the above Scenario, an Assertion error will be thrown for the presence of the username field:

java.lang.AssertionError: The username field should be present
at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.junit.Assert.assertNotNull(Assert.java:712)
  at ias.steps.ExampleSteps.i_verify_the_username_field_is_present(ExampleSteps.java:29)
  at ✽.Then I verify the username field is present(C:/Users/jared/IdeaProjects/ddwi_test_automation/src/test/resources/ias/features/ExampleFeatures.feature:7)

As previously described, we are only seeing one Assertion failure: the first one. This test would fail all three of its validations, but we are unable to see this with vanilla JUnit Asserts.

To get around this limitation, I use a simple utility class that I wrote called "AggregatedAsserts". Below is the code for this class (feel free to adapt it as you see fit):

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
 
import org.apache.commons.lang3.exception.ExceptionUtils;
import org.junit.Assert;

import java.util.ArrayList;
import java.util.List;

public class AggregatedAsserts {

    private static final String NEW_LINE = System.lineSeparator();
    private List<Error> errors = new ArrayList<>();
    private boolean printStackTraces = false;

    /**
     * Executes JUnit assertEquals method. If {@link AssertionError} is thrown, error is
     * added to {@link #errors} and test does not fail.
     *
     * @param message  String the message for the {@link AssertionError}.
     * @param expected Object expected value.
     * @param actual   Object actual value.
     */
    public void assertEquals(String message, Object expected, Object actual) {
        try {
            Assert.assertEquals(message, expected, actual);
        } catch (AssertionError e) {
            errors.add(e);
        }
    }

    /**
     * Executes JUnit assertTrue method. If {@link AssertionError} is thrown, error is
     * added to {@link #errors} and test does not fail.
     *
     * @param message   String the message for the {@link AssertionError}.
     * @param condition boolean condition to be checked.
     */
    public void assertTrue(String message, boolean condition) {
        try {
            Assert.assertTrue(message, condition);
        } catch (AssertionError e) {
            errors.add(e);
        }
    }

    /**
     * Executes JUnit {@link Assert#assertNotNull} method. If {@link AssertionError} is
     * thrown, error is added to {@link #errors} and test does not fail.
     *
     * @param message String the message for the {@link AssertionError}.
     * @param object  {@link Object} to verify not null.
     */
    public void assertNotNull(String message, Object object) {
        try {
            Assert.assertNotNull(message, object);
        } catch (AssertionError e) {
            errors.add(e);
        }
    }

    /**
     * Executes JUnit Assert fail method. {@link AssertionError} is added to 
     * {@link #errors} and test will not immediately fail.
     *
     * @param message String Error message for the {@link AssertionError}.
     */
    public void fail(String message) {
        try {
            Assert.fail(message);
        } catch (AssertionError e) {
            errors.add(e);
        }
    }

    /**
     * If list of errors {@link #errors} is not empty, fails test by throwing 
     * {@link AssertionError} and prints the error messages.
     */
    public void processAllAssertions() {
        if (errors.isEmpty()) {
            System.out.println("No Assertion errors were thrown"); //replace with Logger if desired
        } else {
            StringBuilder msgBuilder = new StringBuilder();
            for (Error error : errors) {
                msgBuilder.append(NEW_LINE);
                msgBuilder.append(AssertionError.class.getSimpleName());
                msgBuilder.append(" ");
                msgBuilder.append(errors.indexOf(error) + 1);
                msgBuilder.append(": ");
                if (printStackTraces) {
                    msgBuilder.append(NEW_LINE);
                    msgBuilder.append(ExceptionUtils.getStackTrace(error)); //converts stack trace to String
                } else {
                    msgBuilder.append(error.getMessage());
                }
            }
            Assert.fail(errors.size() + " " + AssertionError.class.getSimpleName() + "(s) were thrown:" + NEW_LINE
                    + msgBuilder.toString());
        }
    }

    /**
     * Includes the stack traces with the error messages when 
     * {@link #processAllAssertions()} is called.
     */
    public void includeStackTraces() {
        this.printStackTraces = true;
    }
}

Using the AggregatedAsserts class is intentionally similar to using the Junit Assert library. This class includes methods that match the JUnit Assert functionality that I use the most, those methods being assertTrue(), assertFalse(), fail(), and assertNotNull(). If we were to re-write the above example Cucumber Scenario using AggregatedAsserts, it could look like this:

1
2
3
4
5
6
7
 
  Scenario: Multiple field validations
    Given I navigate to the Accounts page
    When  I click on the "Create new Account" button
    Then  I verify the username field is present
    And   I verify the password field is present
    And   I verify the confirm password field is present
    And   I process all verifications and print the results

Example code for the steps:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
 
public class ExampleSteps {
    
    private AggregatedAsserts aggregatedAsserts = new AggregatedAsserts();

    @When("^I click on the \"([^\"]*)\" button$")
    public void i_click_on_the_button(String buttonName) {
        //do nothing, example
    }

    @Given("I navigate to the Accounts page")
    public void i_navigate_to_the_accounts_page() {
        //do nothing, example
    }

    @Then("I verify the username field is present")
    public void i_verify_the_username_field_is_present() {
        //will result in an Assertion Error
        WebElement usernameFieldExample = null;
        aggregatedAsserts.assertNotNull("The username field should be present", usernameFieldExample);
    }

    @And("I verify the password field is present")
    public void i_verify_the_password_field_is_present() {
        //will result in an Assertion Error
        WebElement passwordFieldExample = null;
        aggregatedAsserts.assertNotNull("The password field should be present", passwordFieldExample);
    }

    @And("I verify the confirm password field is present")
    public void i_verify_the_confirm_password_field_is_present() {
        //will result in an Assertion Error
        WebElement confirmPasswordFieldExample = null;
        aggregatedAsserts.assertNotNull("The confirm password field should be present", confirmPasswordFieldExample);
    }

    @And("^I process all verifications and print the results$")
    public void i_process_all_verifications_and_print_the_results() {
        //displays all Assertion Errors 
        aggregatedAsserts.processAllAssertions(); 
    }
}

Now, when we run the example Cucumber Scenario, the following is output to the console:

java.lang.AssertionError: 3 AssertionError(s) were thrown:
AssertionError 1: The username field should be present
AssertionError 2: The password field should be present
AssertionError 3: The confirm password field should be present
  at org.junit.Assert.fail(Assert.java:88)
  at common.utilities.AggregatedAsserts.processAllAssertions(AggregatedAsserts.java:97)
  at ias.steps.ExampleSteps.i_process_all_verifications_and_print_the_results(ExampleSteps.java:48)
  at ✽.And I process all verifications and print the results(...features/ExampleFeatures.feature:17)

Instead of seeing only a single JUnit Assertion error, we now see of all our test's failures displayed in the console. The AggregatedAsserts utility class is all that I used accomplish this. I recommend using the AggregatedAsserts class for tests that have multiple Assertion statements so the entirety of the test's results can be viewed when the execution is completed.

Jared H

Jared Hoernemann

Jared Hoernemann is a Senior Automation Engineer specializing in designing and coding test automation scripts and frameworks from the ground up, as well as mentoring other developers. Jared designs, implements, and executes automated tests for web browsers, APIs of all kinds, and databases.

Have a QA question?

Our team would love to help!

Invalid Email
Posted in Automated Testing, Blog, Technology Corner