A software migration requires a different testing approach than a development project. This is because there is usually neither a requirements specification nor a design model as a basis for testing. This means that requirements-based and model-based test approaches are not suitable for migration projects. Instead, the legacy system can be used as a test oracle for the migration test. The decisive test criterion is then the complete functional equality of the legacy and migrated system - and special methods and tools must be used for this.
There are project types where the test effort and time required can be calculated and therefore a fixed price can be offered to the customer for the test processing. For example, if the product created can be tested against the requirements specification. In this case, it is a clearly specified, stable development project. These requirements documents can be analyzed manually in order to obtain an estimate of the effort and costs involved. Alternatively, it is also possible to carry out this specification analysis automatically using a text analysis tool by extracting data (e.g. function or test points).
Another type of project is maintenance or reengineering projects in which the software itself is only slightly modified. For such projects, the testing effort can be estimated based on the difference between the original and modified version.
Migration projects, on the other hand, are a different type of project in which the entire system is tested against the legacy system. The source code of the predecessor system is the decisive basis for the cost estimate here.
The best way to set up a migration test is step by step. The first step is to measure the source code that is to be migrated. This is necessary in order to find out exactly what needs to be tested and with what effort. Only once the software has been measured is it possible to carry out the next step in the migration test process: planning the test.
This second step in the test planning process involves determining the budget and time frame on the one hand; on the other hand, the individual work steps to be carried out and the expected results are already defined at this stage.
The third step is the design of the migration test. It is very important that every test object used by the system is identified at this stage. This includes user interfaces, database tables, system interfaces and reports. The test procedure - i.e. the individual test steps and their sequence - must also be defined in this process step.
In the fourth step, the test data is generated. The test data for a migration is usually extracted from the production database and anonymized. This serves to meet data protection requirements and, if necessary, to protect privacy (e.g. in the case of customer data). This removal of production data is done by running the old system in a controlled environment. Both input and output values as well as the current database statuses are recorded and stored. The biggest challenge lies in correctly handling high volumes of data (as is often the case with a production database). It is just as important to handle the large number of imports, exports, reports and the recording of user interactions carefully.
The actual migration test is carried out in the fifth step using the data generated in step 4. As a rule, there are no explicitly defined, formulated requirements in a migration project. Detailed test cases must therefore be derived in other ways. Online transactions and batch processes are usually suitable for this. The amount of data input that needs to flow into such specific migration test cases can be considerable and therefore difficult to specify. This also applies to the number of test cases in the case of regression tests. Consequently, automation of migration test cases is recommended at this point - after all, test execution in a migration project is mainly concerned with the successful use of tools.
In the sixth step, the results of the migration test are evaluated to determine whether the migrated system fulfills the expectations: namely to provide the same performance (including performance) as the legacy system. To do this, the results of the two systems are compared and any deviations are recorded as non-conformities.
In the seventh and final step, the test coverage is measured according to the objectives agreed in the test plan. This determines whether the program code has been adequately covered by the executed test cases. If this is the case, the test can be completed; otherwise it must be continued until the objectives mentioned have been achieved. These test objectives (coverage of transactions, code and data, etc.) form the basis of the test service agreement (TSA). ## The measurement of the code
“You can’t plan what you can’t measure”. According to this quote from Tom deMarco, when planning a migration test, you have to start by measuring what is to be migrated: the code. After all, it is the code that is transformed into another form in the course of a migration. If a dataset is also to be migrated, the affected data structures must also be measured.
The code offers various areas that can be measured: the complexity of the control flow, data usage, module interactions, reusability, maintainability, conformity or other variables. Which of these aspects are subject to measurement depends on the desired information and target value that is to be obtained with the measurement procedure. Of course, this means that we have to deal with the number of code units or elements, database tables and data interfaces, for example. Because we have to take these into account in the test in order to meet the functional requirements. The desired metrics can be obtained from the source code, the database schemas, the definitions of the user interface and the system interfaces using static analysis. The measurement results then show how many data interfaces with how many database tables are required to test each defined component. A static analysis tool that calculates the testability of the system on a rational scale from 0 to 1 can provide support here.
Experience from past projects shows that the time required to measure source code is usually no more than 1 week, provided that the appropriate measurement tools are used. Over the course of time, one of the authors has developed tools that are suitable not only for new programming languages, but also for old ones. This is because in the vast majority of migration projects, the predecessor system is written in an old programming language such as PL/1, Cobol or C.
Planning a migration test includes defining the test objectives, providing the resources and estimating the time and effort required to actually carry out the migration test. The test effort here is measured by the number of test cases to be executed and the number of test objects to be validated. A test case can be derived from the usage profile of the production environment and can be defined as a simple online transaction, but also as an event, an operation or a batch process. A test object, in turn, can be generated from the production data and can be a user interface, a file, a database table, a system interface or a created report. Test cases and test objects together represent the test scope. It is necessary to align this scope with the test productivity from past projects. This provides a rough estimate for the current migration test project. This estimate can be refined using the product type, the product testability indicator, the factors influencing the project and the test repetition factor.
The result of the test planning is a test concept in accordance with ANSI/IEEE standard 829 and an estimate of the test effort including the time aspect (test duration). On the basis of these 2 parameters, a price offer for the migration test can be submitted to the client. In this, particular attention should be paid to the test objectives and the end-of-test criteria. Only if these two aspects are met can the customer accept the test at the end. It is therefore all the more important that test objectives and test end criteria are stated in the test concept in an easily measurable form. This can be done, for example, by means of key figures on the test cases performed and the validated test objects. The final version of the test concept can be a standardized Word document in accordance with the IEEE 829 table of contents.
To design a migration test, you need to consider what and how to test. The “WHAT” can be described by means of test cases and test objects (see above). These can be obtained by analyzing the current production process. It should be noted that, in principle, every relevant variant of an existing transaction already represents a potential test case.
Unfortunately, you cannot rely on the documentation of the legacy system when designing a migration test, as it is rarely up to date or often not available at all. The only way for a tester to design the test is therefore to sit down with the end users of the system. During these meetings, the tester records how the users are currently using the system. This procedure amounts to subsequent documentation of operational use. Such written documentation also represents a large part of the effort required for the regression tests.
The test objects, on the other hand, which are also recorded via measurement activities, are easier to identify due to their observability. The number of data elements that each data object has is of particular importance. These include, for example, fields, columns, tags (visible characters for structuring data and/or text) and widgets (components of a graphical user interface (GUI) or a small program, often only for displaying information). Equally relevant are the value ranges of these elements, which can be read from the production environment.
How the migration test is designed depends, on the one hand, on how the test objects are actually identified and validated. On the other hand, how the test cases are to be executed also plays a role. The test design must make it predictable where the test objects will be obtained from and by what means the test cases can be recorded. It should be available in a structured, semi-formal format - for example, an Excel spreadsheet, a graphical tool or an XML document can be used for this purpose.
In contrast to the tests in a development project, it is not necessary to generate separate test data in a migration project - as already mentioned - but it is best to obtain it directly from the production system. At this point, however, the question arises: what data do I extract from this system for this purpose? Copying the entire production environment is generally not recommended and is also too time-consuming. In the test design, however, individual transactions, events and batch runs of a representative nature have already been identified. These can now be executed in a controlled environment using the existing system. Before each test, images of the user control panel, the input files and the database content must be archived. After each test, the new images of the same objects as well as additional output messages sent and reports printed must be recorded in the same form. All of this leads to a large volume of test data, which in many previous projects could often only be handled and managed with a special server.
On the one hand, test tools can be used as useful aids here:
On the other hand, the use of a capture-replay tool is also an option. If none was available in the past, the authors simply captured the user interface of the production environment using a screenshot and used this as a specification for testing the migrated system.
Whether with or without tool support: the problem of data acquisition for migration tests is therefore primarily that of managing large volumes of recorded data. Anyone who wants to test a migrated system - regardless of its size - must be up to the task.
Migration projects are characterized in particular by the amount of testing required in relation to other activities. When systems are newly developed, they are tested module by module, component by component, system by system over a long period of time. Each module must be designed, coded and tested. As these processes are closely interwoven, it is not immediately obvious how much total testing effort is involved. Only the system test at the end of the project clearly stands out as a pure test activity.
However, this is not the case in a migration project, as entire systems have to be tested at the same time. As the code migration is usually carried out automatically by people who are not familiar with the functionality of the system, there is no way of finding out where errors will occur. If code is transformed incorrectly, these people will generally not even notice it. This makes it all the more the task of the testers to prove the correctness of the migrated system and to detect transformation errors. This is why 60% to 80% of the total effort in migration projects is devoted to testing. Unless the migration test is automated, it is impossible to complete it within a reasonable time frame.
In order to be able to determine afterwards which part of the migrated program code was tested, probes should be placed in each branch or at least in each method or program code block before the test is started. Automation tools can be used for this task. At the start of the test, the migrated system is then gradually bombarded with data from the old transactions, using test drivers to simulate the human user. If a transaction fails, the next one is started nonetheless. After each transaction or batch process, the contents of the output fields, the output messages and the new appearance of the affected database are also recorded for later evaluation. It would be too time-consuming to validate this during the test phase. If no automated recording tool is available, the content of the output screens should at least be recorded manually as a screenshot and saved for later comparison. This is because the main focus in this phase must be on carrying out the test with as few interruptions as possible. If anomalies occur, they are also recorded so that they can be checked at a later date - because there is no room for creative testing in a migration test.
Only when the migration test has been completed should its results be validated. For validation purposes, the result screens of the migration test are automatically compared with those from the tests of the legacy system. The screen contents, interface data and database tables of the migrated system are compared one after the other with those of the predecessor system. If the formats differ, they must be converted to a standardized format to ensure comparability.
To this end, one of the authors has developed several conversion and comparison tools. The user interfaces, which may previously have been mainframe memory images or UNIX screens, are transformed into XML documents. The new user interfaces (e.g. websites or mashups - they stand for the creation of new content through the seamless (re)combination of existing content) are also converted into similar XML documents. In this way, individual data elements can be selectively compared with each other, regardless of their respective position or display type on the screen.
The same procedure is used for the databases. The contents of the old databases (IMS, IDMS, IDS, ADABAS, DB2, VSAM, sequential files, etc.) are downloaded in the same way as those of the new, relational databases and converted into comma-separated (CSV) files. These 2 CSV files are then compared column by column. Different formats or calculated values can also be checked using defined rules and statements.
In the course of validating reports and system interfaces, a problem may arise: these may be in a completely different format to the old versions. Of course, the values of the individual data elements should be identical: if the old invoice amount was “EUR 19.99”, then the new invoice amount must also have exactly this value. Nevertheless, this value may be placed somewhere completely different on the new screen or paper printout, for example.
To solve this problem, tools can be used that extract individual data elements from any printout, screen or text file and write them to an XML file. The names and types of data elements are extracted from the original data description into the screen language. An XSD schema serves as a basis for comparison. In this way, data from a previous mainframe list (in the sense of a program printout) can now be compared with data in XML or WSDL format.
Data comparison at the level of these elementary units reveals even minimal differences, e.g. if a decimal point is positioned incorrectly. In the case of comparing millions of data outputs, automated comparison is the best way to detect errors in the migrated code. An alternative way is to compare the execution paths via the code. However, this approach requires tracing the tested transactions. But there are also suitable tools for this, such as TestDoc. It not only records the paths taken by the programming code, but also the number of times the methods or procedures have been run.
Monitoring test execution is an important prerequisite for correctly measuring the percentage of code coverage. Since it is the code that is converted and not the functionality itself, it is the code that needs to be measured. Therefore, probes or tracking points are automatically implanted in the code prior to test execution. They are then recorded in a log file during the test execution - should they be run. After the test run of the migration test, these log files are analyzed to find out how efficient the test really was.
Reports can document different types of information: one report, for example, contains the test coverage as a percentage of sensors passed, which is compared with the original test coverage target from the agreed service contract. Another report documents the path of each individual test transaction, showing which code units have been executed and in what order. These execution paths can then also be compared with those of the original system in order to determine where something went wrong in the new system. In a third report, for example, the code units that are affected by certain transactions are recorded. If it becomes necessary to carry out an error correction for a specific code unit (method or procedure), this report provides information on which test cases need to be repeated.
Different types of errors will occur during the migration test than during the test of a completely newly developed system. In the latter, functions may be omitted, data types incorrectly defined, business rules incorrectly interpreted, results incorrectly calculated, etc. When testing a migrated system, however, the errors are more subtle: decimal places are lost, data records are destroyed, data has moved to a different location, conditions have been reversed, etc. The results may seem correct, but they are not. The only way to detect such errors is to compare all data combined with a coverage of the complete program code. Since you can never know which statement has been converted incorrectly, each statement must be tested - this means that repeated execution of the test is necessary.
In addition to measuring test coverage, it is also necessary to measure the accuracy of the output data and the percentage of actual defects relative to the percentage of predicted defects. This information makes it possible to determine when the test can be completed. The defined planning targets for this were anchored in the test agreement before the start of the project.
A migration test requires a different approach than a classic system test. Subjecting a migrated system to a software test is much more concerned with the aspect of mass production. This is because an enormous amount of converted program code and data must be tested almost blindly within a limited time frame. In order to achieve the goal of demonstrating functional equivalence between the old and migrated systems, almost all converted code elements must ideally be tested using a large data sample from the production system. This large amount of data and the vast number of transactions involved, which have to be carried out repeatedly, make a strong case for the use of test automation.
The authors have found that the migration test requires highly specialized testers and highly developed test tools. It is less a question of the number of testers than of the degree of test automation. Equally important is the area of test organization, as a migration test must be planned very carefully down to the smallest detail. This requires experienced test managers. However, as these only exist in a few organizations, a specialized test team is all the more necessary in order to successfully complete a migration project.