The planning of a requirements-based system test presupposes that the test effort can be calculated based on the requirements. If test points are used to estimate the effort, it must be possible to derive the test points from the requirements documentation. If test cases are to serve as the basis for the effort estimate, these must be derived from the requirements. Both test points and test cases are counted using the text analyzer tool for English and German texts. They are then incorporated into a modified COCOMO-II formula to estimate the effort and duration of the planned test. Factors such as the testability of the system, the degree of test automation and the test conditions are taken into account. It is recommended to carry out at least three estimates and compare the results. A cost range with lower and upper limits should be created. This is used to negotiate the scope of the test with the user. Ultimately, the scope of the test is a question of ROI.
An essential task in test planning is the estimation of test costs. The forward planning method is used to estimate the effort required to confirm the fulfillment of all requirements. The minimum duration of the test is then calculated. Only then is the amount of the test budget based on the target costs and the end date based on the calculated duration. This is how those responsible for testing would often like to see it.
The reverse planning method is used. First, a test budget and a deadline are set. The aim of the estimate is to determine how many test cases or test points can be tested with the specified budget by the specified deadline. Priorities are then set based on a risk analysis and a benefit analysis. The aim is to test as much as possible within the budgetary and time constraints.
Both methods use the same estimation formula, but with different parameters. Depending on the situation, one or the other method is used.
A prerequisite for both methods is the measurement of the test scope. In the case of a requirements-based system test, this means measuring the requirements. The requirements documents must be analyzed either manually or automatically in order to count the test cases or test points. The test cases are derived either from the requirements or from the use cases. A test case must be specified wherever an action is requested, a state is queried or a condition is set. In the case of conditions, there are even two test cases, one for the positive and one for the negative outcome. In this way, it is possible to count all test cases to cover the requirement text. Of course, more test cases will be added, especially where the requirements are superficially defined and incomplete. But at least you have a starting point. If this count is also automated, you can arrive at a test case count quickly and without major effort.
The same applies to test points, which, like function points, are counted based on inputs, outputs, interfaces and database tables. If the user interfaces and system interfaces are mentioned in the requirements document, they can be classified and weighted. The number of database tables and their attributes is derived from the data model. By combining the interface, interface and database points, you arrive at the test points. This count can also be automated. This gives us two different measurements for the test scope:
Once we have established how large the test project is, the next step is to determine the test productivity, i.e. how many test cases or test points can be processed per person-day. Here we are dealing with empirical values. They cannot be plucked out of the air. The most valuable thing about a professional test organization is that it has a memory, namely the experience from past test projects. This experience must be protected and updated. It is difficult to adopt them from the literature or from other projects, because every environment has a different productivity. This is partly cultural, partly technical and partly operational. For example, we measured an average of 3.8 test cases per day in one project and 8.2 test cases per day in another. In the former, many web interfaces were tested, in the latter batch processes.
In one author’s project, 79,000 test cases were executed within 2 months with 92 testers per release. That makes 21 test cases per tester day. However, it is important to note that this was more of a regression test. Only the changes and enhancements were tested against the specification. The rest was tested against the previous version. Today, the same test organization tests 120,000 test cases within 6 weeks with 24 testers. This results in a productivity of no less than 166 test cases per tester day. This sensational increase in test productivity was made possible by automating the entire test procedure.
This experience shows how difficult it is to define average values for test productivity. An automated test is incomparably more productive than a manual test.
Not all application systems are testable. There are systems that are easy to test and others that are difficult to test. A system in which the application logic is interwoven with the user interface is more difficult to test than a system in which the application logic is separate from the user interface. The components of the second system can be tested via a batch interface, the components of the first system can only be tested via the user interface. Systems with broad import and export interfaces are also more difficult to test because the testers have to generate more parameters with more possible combinations. The size of the databases also influences testability. The more attributes a database has, the more data has to be generated and validated.
The testability can be determined by means of static analysis of the program sources, the interface definitions, e.g. the HTML or XSL sources, the interface definitions, e.g. the XML or IDL sources and the database schemas in SQL or XML. The analysis of the software artefacts results in a rational measure on a scale of 0.1 to 0.9. This measured value, e.g. 0.45, is divided into the mean value 0.5 to obtain the testability multiplier = 1.11. This means that the test effort will be 11% higher due to the below-average testability of the software.
Once the test scope, test productivity and testability have been determined, calculating the test effort is simply a question of estimating the formula. A modified COCOMO formula is used here. The original COCOMO-II formula is:
The system types are Standalone = 0.5, Integrated = 1, Distributed = 2, Embedded = 4.
The system size can be in instructions, function points, object points, use case points or whatever.
Productivity is the number of size units that a developer can generate per month.
The influencing factor is the product of 20 individual influencing factors on a scale of 0.7 to 1.4.
The scaling exponent is the mean value of five different project conditions:
in the range of 0.91 to 1.23.
For the estimation of test projects, the authors suggest expressing the system size and productivity in test cases or test points, replacing the influence factor with the testability multiplier and using the following scaling exponents:
The system type must be reduced to 0.5, 1, 1.5 and 2.
Accordingly, a distributed system with 1200 test cases and a testability of 0.45 from a test team with a productivity of 8 test cases per tester day and a scaling exponent of 1.05 would cost 320 tester days.
This estimate shows what is important when it comes to minimizing test costs. On the one hand, the testability of the software must be as high as possible, but the test team usually has little influence on this. They can only sensitize the user to this. The system type is also a given. What the testers can influence are the test conditions and the test productivity. They can improve the test conditions by using professional testers who are well attuned to each other and who work in a mature test process with a high degree of automation. In turn, they can increase their productivity through more test automation. Ultimately, it is important to run as many test cases as possible in as short a time as possible and uncover as many errors as possible. This means high test automation.
All this speaks of a move away from the traditional, home-grown way of testing software systems. Users must be prepared not only to have their application systems developed by professionals but also to have them tested by well-equipped professionals.