Many authorities and companies that work with spatial information are now building geoportals as an essential part of spatial data infrastructures (SDI). Examples of established portals include the Geoportal-Bund in Germany and the geoportal of the Austrian federal states geoland.at. Even if some GI companies today offer “GDI frameworks”, “toolkits” or “suites”, individually tailored software solutions are not available “off the shelf”.
The main reason for this is that geoportals are not created “on a greenfield site”, but have to be integrated into existing environments, e.g. eGovernment infrastructures, with diverse framework conditions. Furthermore, corresponding software solutions are not in mass use, but are rather (further) developed on a project-specific basis. The operator of a geoportal to be set up can therefore not blindly rely on receiving stable standard software. Rather, it will be necessary to test the software in cooperation with the provider and thus qualify it. The costs for this evaluation in the form of tests can easily amount to around 30% of the total costs of software projects, and in some cases even more. Against this background, this article explains which software tests are relevant in connection with geoportals and how they can be carried out.
In the narrower sense, a geoportal is a website that bundles access to distributed (basic) services (map, data and catalog services). For this purpose, geoportals provide various clients (especially map viewers), administration tools and services. With the latter, it is possible to provide basic services connected to the portal for external applications in a processed form and to network portals in this way.
Furthermore, the added value for the user of the aforementioned clients essentially results from the fact that the responses from the basic services are aggregated. Especially if, as is common today, the clients are browser-based, the portal must, for example, display maps from different web map services together or perform coordinate transformations that are not offered by the basic services. Furthermore, the portal must be able to combine the responses from basic services into service chains (e.g. displaying a map for a geographical location).
Tests and checks play an important role throughout the entire life cycle of software applications. The applications are already subjected to unit, component and integration tests during development. System and acceptance tests are carried out later to check the implemented requirements and specifications, and quality is ensured during the use of the software through regression and retests in the area of maintenance (updates, patches). Behind all the test variants are various methods, processes and tools to check different quality criteria of a software, such as the functionality, efficiency or usability of the application.
Established norms and standards, regulations and processes help to carry out test projects in a structured and methodical manner and ensure the quality of the software. The International Software Testing Qualifications Board (ISTQB), which provides testers and test managers with tools, test processes and a common definition of terms with its “Certified Tester” certification series, can serve as a basis for this, for example
The approach presented below relates to the system and acceptance test level, as these types of test focus in particular on the technical requirements. Nevertheless, further upstream and downstream test stages are very important for ensuring the quality of a geoportal.
For the structured and efficient execution of the test project, it is necessary to position the test project within the overall software development and acceptance process on the one hand, and to establish a test process within the test project on the other. The test process based on ISTQB is suitable for the present situation. The individual phases of the test project are explained below.
As with test projects in general, the technical and organizational framework conditions for the test activities must be created at the start of the test project for the test of a web-based geoportal. An essential part of this is the creation of a test concept. The ANSI/IEEE 829 standard provides a comprehensive structure for planning and transparently presenting the test project. This provides everyone involved with a basis on which to build the test project. The following specifications are made in a test concept:
In addition to the test concept, a test controlling system is set up which constantly provides the current status of the test operation. This allows statements to be made during the test case specification phase as to how many test cases still need to be described. During test execution, controlling provides an overview of the test cases performed and those still open, test case coverage and errors found. In addition, it is necessary to establish an error management system to enable traceable documentation of errors and to prevent errors from being “lost”.
The information from test controlling and error management can be prepared for interim and status reports. This ensures constant transparency of the test and problems can be identified at an early stage.
The next step is for the testers to analyze the requirements documents and derive and specify test cases from them.
The methods defined in the test concept for test case determination and test case minimization are used here. The test cases are documented in a test case management tool. The documentation of the test scenarios and test cases includes at least:
When analyzing the requirements, a distinction must be made between the various quality criteria to be tested, for which test cases are specified. For a web-based geoportal, some criteria that are typical for Internet applications are relevant here, including
The functionality test primarily answers the question: “Does the developed and provided geoportal offer the functionalities that are defined in the requirements completely and correctly?” In addition to any standard functions as test objects, a geoportal can also have very complex functionalities, e.g. in connection with coordinate transformations. Some typical test objects for geoportals are listed below as examples:
The second part of this article describes in more detail the special features to look out for in these various test objects.
The usability test is used to evaluate the usability of the geoportal and can cover the following topics, among others:
The efficiency test examines the geoportal for its behavior under load and overload, e.g. with many simultaneous accesses to the portal. Internet applications in particular need to be examined closely here due to the large number of potential users. Here too, it is important to define specifications in the requirements, e.g. how many users can be expected at the same time or over a period of time and how quickly the geoportal must respond to requests in certain load states. The regeneration of the system after an overload is also a possible test aspect here.
It should be noted with these tests that factors that cannot be influenced affect the results, e.g. the integration of WMS services from other providers.
A large number of supporting tools from the commercial and open source sector are available for testing efficiency, especially for Internet applications, which are helpful both for load generation and for evaluating the data.
The topic of security is extremely important, especially for Internet applications, and is very often in the headlines. The system and, above all, the data must be protected against unauthorized access and changes. Users may only see the data for which they are authorized. The security aspect is very comprehensive and requires a high level of technical understanding. Problems with the infrastructure (e.g. the firewall, the network or the web or database server) or newly discovered security holes in applications can make the geoportal vulnerable. Security tests are usually carried out by specialists who also understand the technical background. It is also very important to repeat security tests at certain intervals, as updates, configuration changes and newly discovered security holes can give rise to new risks.
In addition to the technical aspects mentioned above, security must be considered from the perspective of the application. The question here is: Does every user only receive the content they are authorized to use via the geoportal?
A prerequisite for carrying out the test is the classification of users and their assignment to rights. In the actual tests, it must first be checked whether the user is correctly authenticated. Following authentication, the geoportal assigns users to their rights (authorization). In the case of Web Map Services, in addition to the layers, the rights can relate, for example, to scale ranges, the spatial extent or the retrieval of attribute data (GetFeatureInfo Request), whereas in the case of Web Feature Services they relate to the feature types and their attributes.
The correct implementation of these access rights must generally be checked at two levels: Firstly, with regard to the description of the portal services. At interface level, the capabilities of the services must already be filtered according to the user authorizations, i.e. the capabilities of the WMS only contain the layers that the user is allowed to see, for example. On the client side - to stay with the previous example - only the corresponding layers are displayed for selection. This security level corresponds to the concept of client-based security.
The second test level relates to specific requests to the portal services bypassing the portal clients, i.e. at interface level. Unauthorized requests to services must be blocked by the geoportal and answered with a corresponding message.
In preparation for the test execution, the need for test data is also identified in the course of analyzing the requirements and creating the test case. This test data must be available before the test is carried out. For this purpose, it will be necessary to create test data manually or automatically or to modify existing test data, whereby automatic generation is to be preferred. Another option for providing test data is to extract it from the existing system and, if necessary, anonymize it.
The test data for the test cases is stored in a traceable manner to enable the test cases to be repeated.
After preparing the test infrastructure and preparing the test cases, these are carried out by the testers. During the test phase, the functional tests as well as the efficiency, usability and security tests are carried out. In the case of a geoportal, it is important to ensure that all required external services are available during the tests.
The following information is logged for each test case during execution:
If an error is found, an entry must be created in error management. In addition to the exact error description and the metadata, such as the version of the geoportal in which the error occurred, the error is also assigned an error class that describes the severity of the error. The error classes are documented in the test concept.
If errors found in the version to be tested have been corrected, an error retest and a regression test are carried out.
At the end of the test, the recorded test results are evaluated, checked against the acceptance and end-of-test criteria and finally assessed in a final test report. The result of the test report is a recommendation as to whether the geoportal can go into operation or be accepted.
In the event that such a recommendation cannot be made, the reasons for this must be listed in detail.
An important activity at the end of the test project is the analysis and critical review of the entire test in order to incorporate the experience and findings in the course of test process improvement and in further test projects.
Finally, the components, drivers, test data, etc. used for the test are archived so that the test can be repeated or retraced if necessary.
This section describes selected tests and test ideas for the functionality of geoportal clients, services and interfaces.
Due to the complexity of a geoportal, the functionality of the services, interfaces and clients is of particular importance. This also requires an intensive examination of the required functionalities and the underlying norms and standards. This part of a test project is discussed in more detail below.
The architecture of the geoportal is based on a loose coupling between the basic services and the geoportal on the one hand and the portal services and external applications on the other. This presupposes that defined interfaces are supported by the respective components. In the case of geo-services, these interfaces are described by the Open Geospatial Consortium (OGC) as implementation specifications and, if necessary, further specified by profiles. Compliance with these specifications is essential for portal components that are used for external communication and therefore plays a key role in the tests.
The tool of choice for testing compliance with OGC specifications is the CITE program (Conformance & Interoperability Test & Evaluation) offered by the OGC, which has now been completely revised. Among other things, tests for various services such as Web Map Services (WMS), Web Feature Services (WFS) and Catalog Services (CSW) are possible here. The relevant information on the procedure and terms of use can be found on the CITE homepage.
From the perspective of a portal, however, there are some special features that limit the use of CITE tests. The first of these is that the CITE server must have full access to the services to be tested. This can be problematic in development environments or in secure networks. Furthermore, the CITE scenario requires the integration of test data provided by the OGC into the services to be tested. However, portal services generally do not access their own data, but cascade to external services. In this respect, most CITE tests are not applicable to portal services. The tests of client components are not covered by the CITE program. In addition, tests for profiles and tests for the fulfillment of functional requirements, which are specifically tailored to the conditions in the respective geoportal, cannot be carried out with CITE. In this respect, own test ideas, where possible based on CITE, must be defined.
The map service is the most frequently used service of a geoportal and is therefore of central importance.
The tests here are based on the three WMS interfaces GetCapabilities, GetMap and GetFeatureInfo. With a few exceptions, all interface tests must be carried out by calling the interface directly, e.g. by entering it as a browser URL, in order to rule out any errors in the WMS clients. The response from the service is then compared with the expected result. The specific structure of the corresponding requests will not be discussed here.
With regard to the GetCapabilities interface, for example, the following test ideas are useful:
The GetMap interface is the central interface for map services. The tests for correct functioning are very extensive. For this reason, only a few essential test ideas are discussed here:
As a rule, scale hints for individual or all layers are specified in the capabilities for security, performance and/or marketing reasons. In this case, tests should be carried out to ensure compliance. This requires some manual computing effort to generate GetMap requests with the correct content. However, the calculated bounding boxes can be reused in other test cases.
Another test complex relates to the representation of the layers. The following test ideas are to be defined here:
If SLD-capable services are to be integrated into the geoportal, additional tests are carried out to check whether the corresponding SLD parameters (e.g. SLD=) are “passed through” to these services and the result map is rendered in the desired style.
When testing the GetFeatureInfo interface, it should first be checked whether the feature data query is possible for each layer that has been marked with the attribute queryable=“1” in the capabilities of the service. As with the GetMap interface, it should also be possible to generate all specified output formats for displaying information. However, if the corresponding parameter (INFO_FORMAT) is omitted, the output must still be in one of the formats specified in the capabilities.
In addition, it should be tested for each interface whether it works without errors with both the mandatory and optional parameters and whether a valid result is delivered even without vendor-specific parameters.
In geoportals, data services have the task of answering queries for geographical objects such as locations, addresses or regions (gazetteer service) and/or providing raster or vector data (web coverage service or web feature service). There is currently no OGC standard for gazetteer services; the corresponding paper currently has the status of a best practice document.
A gazetteer service is treated as an application profile of a WFS. If the gazetteer service of a geoportal is to use standardized interfaces, only the use of a WFS can be considered. In addition to the interface conformity of the service, the question of whether correct results are delivered is of particular interest for the use and further processing of the results. With this in mind, the following test ideas were selected, whereby only the currently most frequently implemented version 1.0.0 of a basic WFS is referred to here.
The following questions arise in connection with the GetCapabilities interface of a WFS:
With regard to the DescribeFeatureType interface, it must first be checked whether the server response provides a valid GML schema and whether all FeatureTypes of the WFS are also described without specifying the TYPENAME parameter.
From a content perspective, when testing the DescribeFeatureType interface, particular attention must be paid to ensuring that only the attributes of the features that are to be “made public” via the service are described.
In relation to the GetFeature interface, it is then necessary to test whether only the data for these attributes is actually output. To do this, the PROPERTYNAME parameter in the GetFeature request must be given the value * and omitted completely in a second test. The following test ideas should also be used as a minimum:
The last test is relatively time-consuming, as filter encoding queries can be very complex and meaningful queries must be created using the available data, the results of which must then be checked. You should be aware that these tests in particular only represent random samples.
It should also be checked for all WFS operations that both the HTTP request methods GET and POST are supported. While requests can be made using the GET method by entering the browser URL, this is not possible with POST. A specially developed website or application with an input form that is sent to the server via the POST method can help here (requires storage rights on a web server in the case of the website).
Coordinate transformations are required at various points in geoportals. The focus here is initially on server-side transformations; client-side transformations are dealt with elsewhere.
The corresponding functionality is usually not provided externally as an OGC web service, but only used internally. In this respect, the necessary tests are aimed at the results of the transformation and not at interfaces.
Coordinate transformations are most important in connection with the transformation of map images for WMS requests. Transformations can be realized by portal components if SRS are requested on the client side that are not supported by the basic services.
The transformation routine of the portal WMS can be tested using a separate reference data set. The recommended test dataset is, for example, a rectangle that is approximately the size of the geographical area for which the transformation is to be tested.
This rectangle is stored in different layers in the SRS of interest, whereby its (target) coordinates are calculated manually in each case. The GetMap request sent in the browser requests maps in the various SRSs, with the layer that is not to be transformed serving as a reference. Alternatively and more quickly, the bounding box can also be used as a reference.
If the geoportal offers functionalities for geo searches (e.g. locations, addresses, etc.) and for visualizing the search results, a coordinate transformation is required if the SRS of the map does not match that of the search results. If the search is carried out via a WFS, the coordinate transformation is carried out either by a corresponding coordinate transformation service (mandatory up to WFS version 1.0.0) or by the WFS itself (from version 1.1.0, whereby a transformation service can also be addressed internally here). Here too, the coordinate transformation service is only used within the geoportal to convert individual (GML) features in the requested SRS and is not offered as a service to the outside world. In this case, it is not necessary to test the interfaces of the service, but only the correct result of the transformation. This is naturally the case if the map and visualized GML features overlap correctly.
When interpreting the test results, it should be noted that coordinate transformations have different geometric accuracies. In order to be able to identify errors during the transformation, it is therefore necessary to know the accuracy potential of the respective transformation method and parameters: For conversions between Gauss-Krueger strips (e.g. EPSG codes 31466 to 31469), the ellipsoid is not changed and strict formulas are used. The accuracy here is also correspondingly high. However, in the case of transformations where a datum transition is required, e.g. from Gauss-Krueger coordinates on the Bessel ellipsoid to UTM coordinates on the GRS80 ellipsoid (EPSG codes for Germany: 25832 and 25833), the conversion is not carried out strictly, but via transformation parameters. There are a number of different parameter sets for Germany, which differ in their accuracy in relation to a (geographical) area of use. Accordingly, coordinate differences of a few meters can occur if target and actual coordinates were calculated using different parameter sets.
The administration options of the geoportal include the configuration of access rights to the various data, services and functions, as well as the integration and arrangement of various services, layers and specialist data in the portal clients.
While the functionality of geoservices is essentially defined by standardization, the functionality of clients can differ greatly between the various geoportals due to different requirements and implementations. For this reason, only selected test objects with corresponding test ideas can be discussed here.
Map clients can offer a different range of functionalities depending on their design and use. This can also depend on the technology used, so a Java or AJAX-based client will offer more possibilities than an HTML client that may not even use Javascript.
The following typical functions of card clients could be used as test ideas:
Coordinate transformations are also required for client components:
In the simplest case, the transformation of the bounding box coordinates is checked using the map client’s coordinate display. If this is not implemented, tools for analyzing the network protocol can be used to view the incoming and outgoing data traffic on the client via the HTTP protocol and thus the transformed coordinates.
The coordinate display of the client can be checked via a simple target/actual comparison of control points.
Using these clients, requests for geographical regions, addresses or locations are forwarded to decentralized gazetteer services and the results found are usually visualized directly in the map client of the geoportal. The clients are therefore expected to formulate the corresponding requests correctly. In practice, this is checked indirectly by comparing the results of requests via the client with those of HTTP requests to the service.
It should also be tested whether different ways of entering the search terms lead to correct answers. These could be, for example
Further test scenarios then relate to the presentation of the results:
Catalog clients for searching for data and services usually query several ISO 19115/19119-based metadata catalogs. The prerequisite is that the client supports the search via the application profiles implemented in the connected catalog services, e.g. the DE profile of the ISO19115/ISO19119 application profile for OGC Web Catalogue Services (CSW-2.0).
In analogy to the address and location search, the hit list of a search query must also be checked for accuracy. This can be done by submitting identical queries via the geoportal’s metadata search client on the one hand and directly to the externally connected metadata catalogs on the other, e.g. via their clients or (better) as SOAP requests. Both responses must then be compared. Different input variants must also lead to correct results here.
Another test complex relates to the presentation of search results in the client. Here, test ideas are to be developed which test these with regard to
check.
The rapid spread of geoportals and their use even by GIS laypersons, applications in critical areas such as internal security or disaster control and, last but not least, the special features of the underlying service-oriented architecture are some of the aspects that give rise to special quality requirements for the corresponding software solutions. The software tests required to ensure these quality requirements were the subject of this two-part article. In particular, it was shown how general IT testing and GI application know-how interlock in the specification and execution of the tests and thus contribute to the success of the project.
In the future, there will be further opportunities to make the necessary tests even more efficient, particularly in the area of test automation.