Expected ‘false’ to be ‘true’ – Optimizing Automated Testing Results

So you feel your team is solid with over 200 automated tests covering all significant functionality of the app. You share the automated testing results link, expecting cheers and congratulatory sentiments but instead you are asked the question, “What does ‘expected false to be true’ mean”?

A fundamental aspect of automation testing is presenting test results that are understandable. Unfortunately, this is often neglected or forgotten about until after test suites have been developed for some time. There is often the mindset from the get-go that “We need to get something built out now!” This situation can result in expensive, time-consuming refactors and modifications that could have been avoided had the issue been seen at the beginning of the automation effort.

There may be an understanding within the organization that the automation engineers will be the ones to interpret test runs. This may appear fine and dandy, but once the test suites expand it may require many more hours of time that could have been spent on better investment returning endeavors. Not to mention, what happens if *shudder* that engineer moves on to another firm?

Optimizing Automated Test Results

The optimal automation solution creates reports with test results that include enough information for a manual tester or non-automation engineer to be able to replicate the issue. Using Mocha as an example, this may include ‘it’ statements with descriptive steps on the test being executed, with assertions that contain custom error messages as opposed to the infamous ‘expected false to be true’, or ‘#tableTopRow was not visible in 60000ms’.

With more descriptive steps in test case titles, console output, or elsewhere, the reader of these results is more likely to be able to determine what happened. Some solutions may also include IDs in the automated test titles that point to test cases included in Test Management tools such as TestRail, VersionOne, or Rally, so that the reader is able to look up the actual manual test case if all else fails.

The fundamental goal is optimization of resources and time, which is the objective of automation in the first place.