Test Procedure for ZDF

How do you automate tests when the applications are used by a wide range of devices - from new to outdated? The desire to automate everything cannot be fulfilled here, as the differences between the various operating systems are too complex. A combination of automation and manual tests manages the balancing act here. Anika and Benedikt take care of quality assurance for ZDF’s mobile applications and share their strategies for finding errors and how they decide which tests are automated and which are carried out manually.

“You have to find a good balance: What do I automate and on how many devices and what do I actually want to map? And how can I close this gap?” - Benedikt Broich, Anika Strake

Benedikt has been part of Appmatics since 2016 and supports the development of QA processes. As team lead, he works at the interface between test management and partners such as Klett, Kicker & Eurowings.

Anika has been working at Appmatics since 2019. Initially, she worked as a QA automation engineer. Today she is TeamLead Automation and supervises test procedures for partners such as ZDF, Migros & BVG.

Highlights of this Episode:

• Anika and Benedikt from Appmatics talk about their test automation project at ZDF
• The project combines manual and automated tests to ensure high test coverage
• A major challenge is the large number of devices and operating systems that need to be covered
• Manual tests are often faster and more flexible to find and isolate specific errors
• Automated tests are particularly useful for recurring, time-consuming tasks
• The combination of both testing methods increases flexibility and efficiency
• The team tests regularly once a week and provides feedback to continuously improve automation
• An important tip for others is to plan time and resources for test maintenance and customization
• The biggest challenge with automation is maintaining and adapting to new software versions and devices
• Benedikt emphasizes the importance of criticality and frequency of errors when deciding which tests to automate

The balancing act of test automation and manual tests at ZDF

In this episode, I talk to Anika Strake and Benedikt Broich about the challenges of a test automation project for ZDF. They emphasize the importance of striking a balance between manual and automated testing to ensure both efficiency and flexibility.

Background of the project

Benedikt begins by explaining how Appmatics was awarded the contract by ZDF. The original tender called for complete automation of the tests. However, it quickly became clear that this was not the most efficient solution. Many apps did not have tags and IDs set correctly, which made automation more difficult. They also had to ensure that older devices were also covered. The solution: a mixture of automated and manual tests.

The role of manual tests

Anika explains in detail the importance of manual tests in the project. Especially with old devices or specific OS versions, they encounter unexpected problems. Manual testing allows them to isolate and fix bugs faster. While automated tests can provide a broad overview, manual tests allow a deeper investigation of specific problems. This two-pronged approach significantly increases test coverage.

Automated tests: opportunities and limitations

Automated tests offer many advantages, especially for repeatable tasks such as checking links or performing standardized actions in apps. However, Benedikt also points out the limitations of this method. Changes to the front end or updates to operating systems can easily interfere with automated tests. This once again demonstrates the value of manual tests in order to be able to react to such changes at short notice.

Evaluation of test cases

An important aspect of the project is the regular evaluation of which test cases should be automated and which should be carried out manually. Anika mentions that they pay attention to several factors: Does automation increase test coverage? Are the functions critical? And how complex is the manual testing effort? These considerations help to optimize the use of resources.

Results and findings

After a year of intensive work, the team was able to make significant progress. The combination of manual and automated tests proved to be extremely effective. They found errors that they would never have discovered through pure automation, especially on edge devices or in specific usage scenarios. This experience clearly shows that a balanced approach to test automation is crucial for success.