Test Automation and What You Need to Consider When Implementing It

QA specialists and software development engineers in test (SDET) help to ensure the quality of the IT product. Using autotests allows the team to cover multiple required checks within a short timeframe, making testing processes much better and simpler. Today we'll discuss when you need test automation and what you need to consider when implementing it.

The benefits offered by test automation

Reduced time-to-market

While it takes about a day to do the manual run of the "find a bug – register – debug – check" process, it is completed in 2 minutes via automated testing. Thus, much less time is spent from task definition to application release. In addition, autotests are often scheduled to run overnight and in multiple threads, and this also contributes to development speed.

Reduced testing costs

Tech experts explore new features and don't spend time on controlling the quality of the ready-to-use features. You have to keep updating the autotests that were already designed and written, only when some changes were made to the features and/or interface of the new versions. Patching takes from 10 minutes to several hours at most. At the same time, the more frequent your planned releases are, the more profitable test automation becomes.

Increased transparency and quality of testing

Implementation of autotesting allows you to explore the performance, simulate various situations of increased system load and predict system behavior in such situations. Automation also ensures the accuracy of the results, as the machines detect errors quickly, without allowing for their own mistakes. Defect reporting is generated automatically and contains complete information on the action steps taken.

Verifying the need to implement automated testing

ROI (Return on Investment) calculators use a number of parameters to help predict the point in the project at which automated testing brings more value than manual one. Note that the resulting estimate will be an approximation, as you cannot anticipate all the emerging risks well in advance. 

At the same time, we first check if the project meets the criteria that indicate it is time to automate the testing process. 

The project duration is at least 6 months 

Firstly, it has to do with the recovery of autotesting implementation costs. Secondly, this is the time when the project accumulates repetitive test scenarios, and you can easily automate their verification. 

Availability of extended regression testing 

Normally, you carry out these tests after you make the changes, to make sure that everything works properly. Their automation saves a lot of time by itself, especially if it takes 3-4 days or more to run them. Automatic quality assurance allows you to run your tests in parallel and overnight, and the generated reporting makes it easy to fix bugs. 

Detecting multiple bugs in late testing stages 

You can run autotests even at the development stage, which is impossible with manual testing. As a result, you can detect bugs at earlier stages, and consequently, reduce the number of bugs at release. In the end, you lower the risks of negative user experience. 

Need to build test data volumes or fill out large report forms for test cases 

You can cover a huge number of checks by automating them. Autotesting will cover every area or bucket without missing any, and accomplish the full scope of required checks. One of the advantages of automation: you can write a single test and configure the bench so that it can check, for example, all versions of the browser and screen resolutions in parallel and promptly. All documentation is generated and filled out automatically. This increases its accuracy, completeness, and speed of creation. 

Performance and/or system stability problems 

You cannot simulate the concurrent work of even a few hundred users without resorting to automation. Automated scripts running on thousands of threads will create the necessary conditions to evaluate service operability and performance. 

What you need to consider when transitioning from manual to automation testing

Team composition 

Experience shows that the testing and automation process can be most effective when QA and SDET staff work in tandem. In this case, QA takes the lead in studying the logic of the application and forming test scripts, while SDET pays more attention to automating existing cases and compose a convenient report based on test runs. 

Project timeline 

The decision to automate the testing for a particular project depends on many factors, development timing being one of them. Experience shows that SDET costs begin to pay off, on average, six months after starting the work. In another six months, resource savings will average 140–150%, and by the end of the second year they may reach 240%. 

Testing Compliance 

Test automation is effective when testing functionality, rather than user experience (UX). An attempt to automate may also lead to the loss of resources, when you keep making changes to the code (except for minor changes) or when you have plans to change the technology stack in the near future. 

Availability of actual test cases 

Available test scripts facilitate the automation process a lot, as the SDET staff does not waste his/her time on studying the requirements and the app logic in order to create a checklist. 

QA and SDET staff need to write or update test scripts for testing automation. If you already have actual test cases, you can move on to the next steps. 

• Selecting priority test cases. Cases that are part of smoke and regress testing usually refer to priority scripts. At the same time, if testing certain scripts is very costly to automate, and these scripts are rarely included in the run, these works can be postponed. 
• Assessing the effectiveness. Calculating effectiveness metrics helps you determine whether or not to automate the testing of certain cases, and estimate the impact. 
• Determining the order and timing of autotest runs. Verification methods depend on the project features and its infrastructure. Once in a while we run non-recurrent tests to check a particular app build, for example, during a regression, although the efficiency of autotesting is smaller in such cases. If CI is available, you can schedule a run at every push to the master branch or a periodic run at a certain time interval: once a day, at midnight, etc. 
• First-hand autotest writing for all the scripts you selected and adapted for automation.

"With autotests, we reduce time-to-market, which means time from one release to the next. IT giants like Amazon focus on frequent and even daily releases. If there is no automation, then it is simply impossible to test the functionality of a fairly large service during the day. With a UI-test, you can run about two dozen fairly long scripts in 5 minutes. With API tests, the number increases to hundreds.

Modern IT solutions tend to be complex, and the cost of error in some areas can be high – for example, in medicine or the financial sector. Testing automation makes it possible to reduce your risks early on in the development stage."

Evgeny Barsukov 

Head of SDET and DevOps 

Learn more about the development culture in our article: Chaos vs. Order: Implementing a Development Culture for Better Business Results 

Join our Medium community to keep up-to-date with our experts' insights: React vs. Vue vs. Angular: Which One Is Better for Business in 2022