Automatic test case selection for regression tests

Silke had a typical problem with regression tests in her company: they ran for a very, very long time. The framework conditions were difficult: the regression tests check a complex test object that is developed in different programming languages. Together with the Technical University of Munich, she and her team were able to solve the problem - through automated test selection across the C++ and Java components. In the interview, we talked about the specific implementation, various concepts for test selection and the impressive figures on the time savings achieved.

“On the Java side, we were previously at a runtime of 2.5 hours (…) In many cases, we can now actually give the developers feedback after 10-15 minutes: Yes, was good - or not” - Silke ReimerOn the Java side, we were previously at a runtime of 2.5 hours (…) In many cases, we can now actually give the developers feedback after 10-15 minutes: Yes, it was good - or not” - Silke Reimer

Silke Reimer came into contact with the concept of Test Driven Development over 20 years ago at her first employer. Since then, she has never really let go of the topic of test automation and software quality assurance in general. She is now Head of Quality Assurance for one of the two business areas at IVU Traffic Technologies AG. An important part of her job involves improvements in the area of test automation at all levels. Prior to her current position, she also devoted many years to this topic as a test architect. It is particularly important to her that test automation is not the goal but one of several measures to give developers quick feedback on the quality of the code.

Highlights of this episode:

• We talk about test case selection for regression tests
• The solution was developed together with the Technical University of Munich to reduce throughput times
• Silke works at IVU and develops software for transportation companies
• The test suite comprises around 25,000 test cases and took up to seven hours without selection
• Up to 50-60% of test cases can be saved through selection
• The solution analyzes which files are opened during test runs in order to select the necessary test cases
• A reduction from hours to minutes was achieved in some runs
• The developers receive significantly faster feedback
• The system has been running for about one and a half to two years
• There were no situations in which important errors were overlooked by the system

How to turn hours into minutes

Silke Reimer shares exciting insights into the world of automatic test case selection. In cooperation with the Technical University of Munich, an innovative solution was developed to significantly reduce the throughput times of regression tests in a complex software environment.

The leap into a new era of test automation

Silke talks about challenges and solutions in a scenario that many companies are very familiar with: long throughput times for regression tests due to a huge code memory and various programming languages. The collaboration between IVU and the Technical University of Munich led to the development of an innovative approach that makes it possible to execute only the necessary test cases and thus achieve massive time savings.

The challenge: A mountain of code and waiting time

Silke gave an overview of the dimensions of the problem. At IVU, a company that develops software for transportation companies, the test suite alone comprises around 25,000 cases. A complete execution of all tests took up to seven hours - an unbearable waiting time for developers after each code check-in. In addition to the complexity caused by different programming languages such as C++ and Java, there was also the maintenance of numerous branches. This setting led to an enormous strain on resources and a delayed feedback loop.

The solution: Intelligent selection instead of mass testing

The turning point came with a project in cooperation with the Technical University of Munich under the leadership of Professor Alexander Pretschner. By employing a PhD student, the team focused on developing a system that only selects relevant test cases based on the changes in the code. This approach not only allowed the team to save significant time, but also to use resources more efficiently and provide faster feedback to developers. Silke shared exciting details about the process of selection, including nightly analysis and automatic mapping of code changes to test cases.

From hours to minutes

The result of this project is impressive: in many cases, feedback to developers was reduced to 10 to 15 minutes - a huge improvement on the original seven hours. What is particularly interesting is the fact that this was achieved across different technologies. The increase in efficiency affects both Java and C++ areas of the company, demonstrating the broad applicability of the system developed.

What’s next?

During our conversation, Silke gave an outlook on possible future improvements to the system. These include, for example, prioritization within the selected tests in order to provide even faster feedback or a more detailed analysis at functional level for further efficiency gains. This continuous optimization clearly shows the potential and necessity for innovation in the field of software testing.

Fazit

The story of IVU and its successful implementation of intelligent test case selection is a great example of technological progress. It impressively demonstrates how bold decisions and innovative collaboration can overcome challenges that many would consider insurmountable.