Pros and Cons of Commercial and Open Source Test Automation Tools

With software development organizations facing ever-growing pressure to deliver better software, faster, practices such as continuous integration and “shifting left” have gained importance over the past decade. We’re often asked the differences between using commercial and open source test automation tools – and how well they can be used together.

At tapQA, we help organizations build and implement test automation capabilities. This post will explore the pros and cons of commercial and open source test automation tools.

The Advantages of Open Source Test Automation Tools

The number one advantage is cost – particularly the cost of entry. Open source technologies, such as Selenium, are free to use. If you're a cost conscious team, utilizing open source technologies is a great option.

Another advantage is the flexibility that is gained by going open source. Some of the most popular open source technologies come with large communities of support that are behind them. And with that support comes cool libraries and approaches that are built by other engineers with a passion for testing. Whereas commercial tools are often times closed systems with limited flexibility beyond their primary goal.

Additionally, open source technology supports multiple languages. For example, Selenium offers support for many of the most popular languages, from C# to Python to Java to JavaScript. This means you can utilize the technologies that are most widely used among your technology teams to help support the automation frameworks built in open source.

The Disadvantages of Open Source Test Automation Tools

The number one disadvantage in my mind is the skill set necessary to build and maintain the automation suite. Most open source test automation tools come with a “technical barrier of entry.” Some are easier to get past than others, but most have a moderate investment needed to gain a better understanding of the open source technology being used to be able to get to the peak efficiency of that tool.

“Quality” is the underrated disadvantage of going with open source tools. You have to remember that most open source tools are created by people for a specific reason. And when that reason is fulfilled, then the support for that tool can wane. Meaning the lifespan of an open source tool maybe limited. Possibly leaving you with a bug-ridden tool that is just barely getting the job done in the future.

If you were leading a team that didn’t have a dedicated “automation professional” (in other words, a team of QA professionals that may not be particularly strong with programming), a poorly written framework using open source could create some problems – such as ongoing maintenance and support headaches.  In that scenario, a commercial tool may end up being a better choice.

The Advantages of using Commercial Test Automation Tools

The number one advantage of using commercial tools is you don't need as strong of an in-house skill set to get up and running. Many commercial tools use built in workflows out of the box that can make getting off the ground much faster than using an open source tool.

Commercial tools also come with built in support and documentation, which is huge. Much of the documentation provided can be well-written and easy to follow for most engineers. And on top of that, most commercially available tools come with a full support team if any issues arise, so you always have the option  to call them or chat with them if there are issues with their tool.

The built-in reporting capabilities of commercial tools are typically an advantage as well, although the ability to customize the reporting capabilities of open source tools can be really nice.

The Disadvantages of using Commercial Automation Tools

The biggest disadvantage of commercial tools has to be the cost of entry. Commercial tools can be prohibitively expensive and for the most part are subscription-based nowadays. So that means that your team would have to shell out fees on a yearly basis, and those fees aren't small.  Open source frameworks, on the other hand, don’t require licensing – you own the end product that is created.

Flexibility is also a big disadvantage in my mind for commercial tools. These tools are built to do specific tasks and they do those tasks very well, but if you try to do anything outside of those tasks, it may require you to actually enter a change request from the developer of the tool or from the vendor just to be able to get other tasks completed.

The Balanced Approach – utilizing Commercial and Open Source in an Automation Framework

We’ve worked with some great clients on test automation practices that blend open source and commercial tools together in a customized framework. 

Most commercial tools support the ability to intake some open source tools and they can be used together in tandem, and you can really enhance most commercial tools with some open source tools so they can be used together.

For example, Ranorex is a commercial automation tool that allows the ability to use Selenium scripts within their framework. So the point there would be that you would be able to support any legacy automation while you're creating the new stuff within the Ranorex tool.

At tapQA, we recommend prioritizing what items you can use an open source tool for, and then what makes sense to use a commercial tool for. The open source tool would usually be for specific tasks the commercial tool doesn't support.  You’re really itemizing what you need the commercial tool to do and what it can't do. And then, use an open source tool to support the items that the commercial tool can't do would be a great practice to follow.

Best Practices for organizations new to Test Automation

If a company were starting automation for the very first time, it’s a great idea to start with the goal of finding a correct balance with open source and commercial automation tools.  One of the first things you'd want to look at is the technical skill set of your current team.  Of course, your budget to spend on tools is important too.  If you have a flexible budget for tools, and you don’t have strong programmers on your QA team, starting with a commercial tool is a wise choice. Commercial tools are easier to get up and running and it has a lot more thorough documentation around it.

Some of the biggest success stories we've had are building out frameworks using what we call automation accelerators. Our automation accelerators are built with a blend of open source and commercial automation technologies, and allow us to get a head start on spinning up a new framework or environment. So a lot of the customers that we come across may have a scenario where they don't have anything in place and they have to start from scratch. They are able to utilize the tapQA Automation Accelerator, either for API automation, UI automation or integration tests, to be able to get that kickstart and hit the ground running with test automation.  We’ve seen huge improvements as far as the overall quality of the application that's being built. It allows us to get in there faster, work with the team and get better information by being able to focus more on information gathering rather than the fundamentals of building up the framework.