I bet you have ever heard this sentence (or similar): “We need to achieve 100% of our software test automation“.🧐
It’s a great idea if we want to produce software (SW) of the highest quality. However, we need to bear in mind that the test automation process can be very costly. This makes it necessary to select the test cases to be automated carefully to ensure the investment we are going to make is profitable. In other words, we need to compare the benefit to be obtained with the cost of the automation.
What is testing automation?
📟 Software test automation consists of creating “scripts” that mechanise SW application test cases and allow their unattended execution. Automation frameworks are used to generate the sequences that a user would perform when running a test, including the following activities:
- Preparation of the data to be used.
- Browsing in the application following the steps necessary to test it.
- Registration of the execution in the test management tool.
- Collection and recording of test evidence.
- Generation of a report with the results.
💻 Automation frameworks allow all these activities to be carried out using programming languages, which may be high-level, zero-code (for automators with limited technical knowledge), or low-level (for those with some technical knowledge).
The lower the programming level, the more sophisticated these automatic programs can be, although, of course, this tends to require more expensive staff profiles… everything has to be taken into account!
In the Agile context, automation assumes special importance due to the need for continuous deliveries which require very frequent testing processes.
Business Case 📋
When deciding how much and what to automate, we need to be clear about the items that contribute to the cost in order to make the comparison with the costs of manual testing.
Los costs include:
- Design of the test cases, which as indicated above is done using programming techniques (the staff profiles required are usually more expensive than those needed for manual testing).
- Maintenance of the test cases is a hugely important aspect that is often overlooked. Since automatic test cases include programming of the actual applications to be tested, any changes to the applications themselves will require reprogramming of the test cases, and this is considerably more expensive than maintaining manual test cases.
- The type of application to be automated also influences the cost: automating user interfaces is much more complex than automating layers closer to the server (such as APIs, direct access to databases, or microservices).
The savings are generally due to the fact that because it is automatic the execution does not require human intervention and all the user has to do is check the execution logs to see if the execution was successful or not.
Automation Criteria 📑
There is no formula telling us exactly how much and what to automate. However, what we can do is provide certain criteria to take into account when deciding on the level of automation. These include the following:
- Stability: the more stable an application is, the less maintenance it will require and therefore the less it will cost to maintain automated test cases. New applications are highly susceptible to change and therefore to high automation costs.
- Criticality: functionalities that have a very high frequency of execution or with a major impact on the business in the event of failure will need to be tested every time new SW is deployed to ensure that they continue to work.
- Fragility: this is a functionality covered by a complex development where it is considered likely that a new version could include defects.
Another resource that provides assistance when deciding what to automate is the Cohn pyramid.
The idea of the pyramid is that the lower you go, the more you should automate:
- Unit tests: by detecting defects at this stage we avoid finding them in later development phases, thereby saving costs.
- API and component tests: normally this automation is not complex, and if we choose stable components the business case is likely to be favourable.
- Graphical user interface (GUI) tests: automation tends to be complex and slow to execute. It is not advisable to automate much of this; exploratory testing is an excellent alternative.
Fundamental points in any project👀
Testing is undoubtedly a decisive step to ensure the quality of SW, and investing in quality is always important.
The concept of the iron triangle is used in Agile developments to refer to the three fundamental elements of any project (scope, scheduling, and cost). In order to achieve a certain level of quality, a combination of these factors is needed.
Changing one of them means having to change one (or both) of the others to compensate. Finding the best combination depending on the specific circumstances of each project is key to management processes with a view to ultimately achieving the desired level of quality.
If you work in the IT world and you are still confused when you hear terms like “Software Testing” or “Software Quality”, this other post where we tell you the differences between the two terms, is of interest to you.
Santander Global T&O is a global company of Santander Group with more than 3,000 employees and based in Madrid, we work to make Santander an open platform for financial services.
Check out the positions we have open here to join this great team and Be Tech! with Santander.
Follow us on LinkedIn and Instagram.
Mariano Saúca
Santander Global T&O
I am active and curious in all areas and I am absolutely convinced that “there’s a will there’s a way”. I am passionate about managing teams and making them grow, as well as speaking in public and spreading (how rare is one…). Without a clear vocation, in recent years I have been dedicated to the world of SW quality, because that’s what I got. The important thing is to add value doing what you have to do.
