After more than 40 years of experience in software engineering, too many projects are still overrun or fail completely. The reasons for this include incomplete and changing requirements.
Although a lot can be achieved by having a dedicated requirements engineer in the team, this article goes one step further. The quality of the project can be further improved by utilizing the experience of software testers in detecting errors in requirements specifications in early project phases. This combination is very promising, as the current training opportunities for software engineers show. The new successful super-certification QAMP® (QAMP Quality Assurance Management Professional) includes both tester and requirements engineering skills.
The Standish Group’s Chaos Report, which has been tracked since 1994, is synonymous with visualizing the high rates of projects that go over time and over budget or fail. The reasons for the low rate of successful projects have long been the subject of research and are well known to most successful project managers. While some problems have been reduced through the use of engineering techniques and standards, the following are still the most common.
As you can see, requirements are one of the main sources of problems in this list. Not only because they are incomprehensible and change, but also because they affect other sources of problems such as time and resource estimates. As many engineering problems are considered solved by refined technologies from project structuring to testing methods, the gap between customer and developer perspectives has only been subject to increasing standardization in recent years. While there are notations and templates for requirements, for a long time there was no clear concept of how to collect requirements in the first place. Who should be consulted? What should be done to ensure that no important stakeholders or influences are overlooked? How do I identify supposed requirements? Methods for requirements elicitation combined with standards and best practice approaches are part of the “Certified Professional for Requirements Engineering” (CPRE), a relatively new training program for software engineers. With the structure of this certification, the International Requirements Engineering Board (IREB) is closely following the example of the very successful ISTQB Certified Tester.
At the beginning of every project is the goal, however vague and different this goal may be in the minds of all those involved. In the requirements analysis phase, the project goals are refined and finally recorded as concrete requirements. The quality of these requirements very often depends on the skills, personal goals and support of the people recording them. To reduce this influence, the IREB defines a dedicated role for requirements engineers. This results in having a neutral person who can identify and consider all stakeholders and their respective goals in a project. By carefully using their communication skills, templates and engineering methods, the requirements engineer can elicit, document and prioritize all relevant requirements. The validation and verification of requirements is also the responsibility of the requirements engineer. Various documentation techniques and reviews can help to identify incomplete and incorrect requirements. Last but not least, careful requirements management helps to keep track of changing requirements. The benefit is obvious: a complete set of comprehensible and testable requirements details the project goals and facilitates planning, implementation and testing. It is the CPRE methods that we use when we first check a new requirements specification for completeness and testability, and it is not uncommon for us to identify specification gaps and supposed requirements in this way.
Given all the approaches and techniques in requirements engineering, no project should fail due to an incomplete requirements specification. Yet, in our experience, in many projects requirements are still specified by domain experts who are not familiar with refined specification techniques and often do not have the time and support to elicit all relevant stakeholders and details. While the fact that good and exhaustive requirements are a critical factor for project success may not yet be in the focus of project management, the need for independent testing very often is. Some project failures generate a lot of media attention. The public conclusion is very often that systematic testing could have prevented problems. Therefore, a project team must not only consist of a professional requirements engineer, but of several test experts for system and acceptance tests.
In many of our projects, we find that customers still see testing as a phase after implementation. So when we enter the project and are given the opportunity to analyze the requirements specification for the first time, the system is already under development. As certified testers, we then scrutinize requirements that are not detailed enough to specify test cases. Details provided by the customer are usually unknown to the developers and sometimes even include new requirements that have arisen from a deeper reflection on the system. While we may be satisfied with the information we receive, the developers usually are not. How could they be if the customer suddenly shifts the target? The result is often long discussions, excessive budget and time. Nevertheless, the feedback for testers is usually great, as the project management appreciates our deep insight into the requirements. The following diagram shows how many of the errors in a project are due to faulty requirements.
Although requirements-based testing can be very effective in detecting faulty requirements, fixing these bugs is expensive compared to bugs that occur in later stages of the project.
The obvious conclusion from late testing is that errors in the requirements must be detected earlier in the project. An obvious solution is to start testing at the beginning of the project. This is a solution that our customers very often implement after initial experience with requirements-based testing. The fact that this approach is also promoted by the ISTQB’s curriculum for certified testers helps to promote the topic for future projects. As a first step, testers are involved in the review process before the respective requirements specification is released for design and implementation. Due to their experience in quality assurance, testers are very good reviewers of requirements specifications. As it is their job to derive test cases from specifications, they first check the testability of requirements. Are there criteria by which one can recognize whether a requirement is fulfilled or not? Are the requirements exhaustive to describe the behavior of the system? How much a review by testers can contribute is shown by the number of questions raised: during a 9-month functional test phase, which began during implementation, two testers raised around 250 questions, some of which even led to an extension of the functionality and even to change requests, i.e. around one question per tester per day. In another, similarly complex project, two testers reviewed a concept within two days before the start of the implementation phase. As it was the first review, they did not even start to derive test cases. Nevertheless, more than 60 questions were raised in these two days, which could be solved by revising the concept, saving many days of development and testing effort. The special thing about testers in reviews is their different perspective. While the stakeholder often only provides enough information to convey their intentions, the tester’s goal is to obtain clear and testable requirements for the derivation of test cases. If the intended behavior in a test case cannot be determined, a tester will inevitably ask for details. While, for example, a customer demands sufficient performance so that the user can “work without interruptions”, the tester will ask for exact response times. Only with these is it possible to decide whether a test case passes or fails. While testers are good at improving requirements through their experience, training in requirements engineering can improve skills in requirements elicitation and raise awareness of suspected requirements.
The learning curve described is based on our experience in relatively small projects (each realized in less than two years). Often one and the same customer has different projects with different managers. The learning curve therefore includes the general knowledge within a company. This is a typical status quo:
Testing, which was once the starting point of an initiative for quality in projects, has now become an integral part of every project. The established testing process is based on the ISTQB recommendations and qualifications such as a Foundation Level or Advanced Level test certification are required for every new test expert in order to establish a common vocabulary and methodology. More than 100,000 certified testers in more than 40 countries prove that our experience is in line with global trends. In addition, we are seeing an increasing focus on requirements quality. Whereas just a few years ago a domain expert would write down the requirements for a new system based on a template that had grown out of his or his company’s experience, the effort involved in requirements elicitation has increased dramatically. The early involvement of testers and reviews by various stakeholders are a step towards increasing professionalization.
As outlined above, professionalization in requirements engineering is underway. Here we would like to describe what we hope to achieve.
It is a generally recognized truth that independent testing provides a neutral and reliable picture of software quality, as it is neither on the side of development nor on the side of customer interests. The same truth should apply to requirements engineering. Although many customers still demand domain knowledge, a dedicated requirements engineer with a neutral view can avoid mistakes introduced by domain experts who have become jaded by habit. In addition, they can apply the knowledge acquired in special training courses, such as CPRE. A common language when modeling requirements, e.g. in UML, simplifies communication with development and testing.
System and acceptance tests are also referred to as specification-based or requirements-based tests. Many well-known test procedures, such as use case tests or state transition tests, are requirements-based. What these methods have in common is that they are based on models. Although these methods are well known to certified testers, they are often not applied systematically.
Two approaches are necessary to achieve a very good requirements test: Firstly, the use of these model-based methods must be encouraged when specifying test cases. This is the only way to achieve reproducible test coverage. Secondly, the models must already be applied to the requirements during the review phase. Most model-based test methods are able to detect specification errors, a feature that is often underestimated in the specification phase.
One example is the testing of decision tables. Here, the combination of conditions, relationships and constraints is tested. In this way, undefined results from rare combinations can be recognized when constructing test cases.
Another example is the testing of state transitions. While requirements are often documented in textual form, the derivation of a state model can be used to map processes and behavior in different states of a system. In this way, additional requirements can often be identified.
Ultimately, the application of the already known methods not only leads to an improvement in the test coverage of requirements, but also to an improvement in requirements quality.
Another advantage of shared models in requirements engineering and testing is the drastically improved traceability. Especially in safety-critical systems, it is necessary that requirements can be traced to their models and further to the test cases that verify these models and back again. By using common models, the coverage metric can provide a detailed status, starting from the overall test coverage to the coverage of each individual model element and requirement.
The goal of high requirements quality no longer seems far away. Awareness of requirements quality is increasing among customers and software engineering providers. Suitable training for all those involved is essential for a lasting effect. Training must not only address individual techniques, but should focus on the interrelationships between the different project roles. Common modeling languages and an idea of the goals and work of the other team members improve mutual appreciation and enable consideration of each other’s requirements. The requirements of cooperation should also be addressed in training courses. There are already qualification measures for a holistic approach to software quality that integrates requirements engineering and testing. One of these is the new super-certification QAMP® (Quality Assurance Management Professional), which builds on CPRE and Certified Tester Foundation Level (CTFL®). Both trainings follow established approaches and are coherent in themselves. But QAMP goes even further: although CPRE and CTFL form the basis of QAMP, proof of experience and further qualifications such as project management, test management or secure software engineer are required. Current professional experience, research activities or further qualifications are required for periodic recertification. In this way, QAMP ensures that every certificate holder is committed to lifelong learning.
It is obvious that tools and methods are sufficient. Our challenge now is to create awareness of the great benefits that the combination of software testing and requirements engineering can bring.