Tool Fight Night

I’ve been doing what I do for over 13 years now and what I am doing is test automation and performance testing. Since my first ever project at DUO (Dutch Student Loan and Educational Service) until my last at a telecom organization, I helped organizations with test automation and performance testing. But the last several years I also help organizations to choose the right test tool to help them improve their testing process.
And this is where the battle between tools starts. First of all, let me give you a definition of a test tool. A test tool is an automated tool that supports one or more test activities. This includes planning and management, specification, test execution and other test activities. I guess if we take this definition, all of us have worked with test tools at one time or another.

So if we want to use tools to help us with our testing activities, it’s important to choose the right tool, choose the tool that suites your situation best. But it’s not always easy to choose the right tool.

If you google on “Test Tool”, within a split second the search result turns up with millions and millions of hits. Consisting of both commercial and open source tools.
Now, first let’s go back to my first ever assignment at a customer. I needed to setup a framework for automated regression testing. This keyword driven framework was built with a so called “Record and Playback tool” and the tool was by a big commercial tool vendor. I didn’t select the tool, but this tool was the tool of choice within this organization. Now, at that moment in time, there were not many test tools around and to me it seemed a good and logical choice.

Was I blinded by the use of this commercial tool? Yes, for several years I was. I did a lot of customer assignments in the next few years, and everywhere I was, commercial test tools were in place and used or not, because sometimes the tools were not fit for purpose and ended up as shelf ware. This again shows the importance of a proper tool selection. But I will not go into this.

Now, the testing world was changing and so were the test tools. After a few years, the software I had to test changed and therefor the tools that were used to support the testing of this software changed. The introduction of web applications made it possible to choose a different type of test tools, open source test tools.

At first, I was rather sceptic about using open source tools, because it was unknown to me and I always relied on the big vendors, to help me with whatever problem I had. But now I was at the hands of others, communities, (hobby) developers etc. And it scared me! I asked myself again and again, why I need open source, I am comfortable with the tools I have and know.

So for years and years I went on using the commercial tools for most of the time and sometimes I had to use a little web recorder, or script tool to help me, because there was no budget to buy tools. And this is one of the first things I want to bring to your attention, budget, funding, find the money. Because, costs are quite often a motivator for using open source tooling instead of commercial tooling.

If we take a look at this, it’s true, the licenses for commercial tools are often costly, but it not only buys you the right to use a tool, but also a lot of other things. What about support, what about trainings. When you make your choice for a tool, be aware that there is more to it than the costs of a tool.

Let’s take a closer look at the fight between the tools and see if someone wins this fight.

Fight Night
“Ladieeeessss and Gentleman! In the right corner, weighing 220 pounds, COMMERCIAL!!! And in the left corner, weighing 130 pounds, Open Source!!!”
“Let’s get ready to Rumble.”

First round: Costs
Let’s take a look at the costs. At first, Open Source doesn’t mean per definition that there are now costs when u use this tool. Ok, maybe, there are no initial costs purchasing this software, but there are others costs, basically the same other costs you have when buying a commercial tool. Startup costs, educational costs, support costs etc. You need to be aware, that when you buy tools from a commercial organization, a lot of costs are already in the license. You buy some kind of security for yourself, but it comes with a price. If you start with open source, there is also need for the same security, but you have to buy it bit by bit.
So, costs can be one of the biggest motivators for using open source, but there are hidden costs in there, which makes it also possible to use commercial tools.
First round ends in a draw.

Second round: Support
As I stated above, when you buy a commercial tool, you also buy support, both functional and technical. If you have problems with your installation, you can use the technical support help desk of the vendor, and they will help you installing the software. When u use open source tooling, quite often the only technical support you have is the installation guide or a forum to help you when you come across problems during installation.

The same thing when you have functional problems. If you don’t know how to use certain functionality of the tool, there usually is comprehensive documentation and support by the vendor. Again, if you use open source this could not be the case. On the other hand, open source software quite often has a large user base, which can function as a support portal. The community has forums, portals, user groups etc. which is basically one big support department.
So again, this round ends into a draw.

Third round: Updates
If you use open source tooling, the rate by which updates are released, can be somewhat frightening. Release after release after release, not often beta releases, are thrown into the world. This can be very difficult to manage, where commercial vendors always use steady releases and a release schedule for their updates. So, is this a uppercut for commercial tools? No, it’s not. While the steady releases are great for buying a secure mind, often new technologies or adaptations to existing technologies take a long while. Open source, because again of the communities and the big user base, are a lot quicker in adapting to changes.
And again, this round ends into a draw.

The-winner-isFourth: Training and Education
This round doesn’t differ too much from the previous round. When you buy commercial tooling, training and education is often well arranged. Online and classical courses on different levels, workshops, training on the job are almost always part of the training program. With open source, you need to look a little bit further and it is a little bit harder to find the trainings, workshops etc. But they are available, often offered by consultancy firms or online.
This looks like a win for commercial, but open source is on his legs again. This round ends in a draw.


And the winner is?

Now, if we take a look at the fight, it looks like there is now winner when you have a fight between open source and commercial tooling. So where is the catch? Why is there always a constant battle between open source and commercial tooling? Well, there is not a battle between the tools, but between the supporters of the tools. There are always people that favor open source and there are always people that favor commercial tools.

My advice to all tool users out there, don’t battle, just select the right tool for the job and select it in a proper way.

This article was published in our May 2014 edition.

https://i0.wp.com/www.testingcircus.com/wp-content/uploads/Tool-fight-night.png?fit=334%2C189&ssl=1https://i0.wp.com/www.testingcircus.com/wp-content/uploads/Tool-fight-night.png?resize=150%2C95&ssl=1Beersma BerndArticlesOpen Source Testing Tools,Testing ArticleTool Fight Night I've been doing what I do for over 13 years now and what I am doing is test automation and performance testing. Since my first ever project at DUO (Dutch Student Loan and Educational Service) until my last at a telecom organization, I helped organizations with test...