Testing a Geoportal

The test process

Motivation

Many authorities and companies that work with spatial information are now building geoportals as an essential part of spatial data infrastructures (SDI). Examples of established portals include the Geoportal-Bund in Germany and the geoportal of the Austrian federal states geoland.at. Even if some GI companies today offer “GDI frameworks”, “toolkits” or “suites”, individually tailored software solutions are not available “off the shelf”.

The main reason for this is that geoportals are not created “on a greenfield site”, but have to be integrated into existing environments, e.g. eGovernment infrastructures, with diverse framework conditions. Furthermore, corresponding software solutions are not in mass use, but are rather (further) developed on a project-specific basis. The operator of a geoportal to be set up can therefore not blindly rely on receiving stable standard software. Rather, it will be necessary to test the software in cooperation with the provider and thus qualify it. The costs for this evaluation in the form of tests can easily amount to around 30% of the total costs of software projects, and in some cases even more. Against this background, this article explains which software tests are relevant in connection with geoportals and how they can be carried out.

Geoportale

In the narrower sense, a geoportal is a website that bundles access to distributed (basic) services (map, data and catalog services). For this purpose, geoportals provide various clients (especially map viewers), administration tools and services. With the latter, it is possible to provide basic services connected to the portal for external applications in a processed form and to network portals in this way.

Furthermore, the added value for the user of the aforementioned clients essentially results from the fact that the responses from the basic services are aggregated. Especially if, as is common today, the clients are browser-based, the portal must, for example, display maps from different web map services together or perform coordinate transformations that are not offered by the basic services. Furthermore, the portal must be able to combine the responses from basic services into service chains (e.g. displaying a map for a geographical location).

The test project

Tests and checks play an important role throughout the entire life cycle of software applications. The applications are already subjected to unit, component and integration tests during development. System and acceptance tests are carried out later to check the implemented requirements and specifications, and quality is ensured during the use of the software through regression and retests in the area of maintenance (updates, patches). Behind all the test variants are various methods, processes and tools to check different quality criteria of a software, such as the functionality, efficiency or usability of the application.

Established norms and standards, regulations and processes help to carry out test projects in a structured and methodical manner and ensure the quality of the software. The International Software Testing Qualifications Board (ISTQB), which provides testers and test managers with tools, test processes and a common definition of terms with its “Certified Tester” certification series, can serve as a basis for this, for example

The approach presented below relates to the system and acceptance test level, as these types of test focus in particular on the technical requirements. Nevertheless, further upstream and downstream test stages are very important for ensuring the quality of a geoportal.

For the structured and efficient execution of the test project, it is necessary to position the test project within the overall software development and acceptance process on the one hand, and to establish a test process within the test project on the other. The test process based on ISTQB is suitable for the present situation. The individual phases of the test project are explained below.

Planning and control

As with test projects in general, the technical and organizational framework conditions for the test activities must be created at the start of the test project for the test of a web-based geoportal. An essential part of this is the creation of a test concept. The ANSI/IEEE 829 standard provides a comprehensive structure for planning and transparently presenting the test project. This provides everyone involved with a basis on which to build the test project. The following specifications are made in a test concept:

• Test objectives: Different test levels have different test objectives, e.g. the objective of an integration test is to detect interface errors. The aim of a system test is to validate the overall implementation against the requirements.
• End-of-test criteria: In order to know when the goal of a test phase has been reached, end-of-test criteria must be defined. These can be, for example, various coverage measures such as “all Prio 1 test cases and at least 90% of Prio 3 test cases completed”.
• Acceptance criteria: Criteria are required to evaluate the system at the end of the test project and represent the minimum requirement for using the system.
• Definition of the test strategy: For each test level or type, it must be determined which methodologies and tools are to be used for test case determination and execution.
• Definition of the framework conditions: Among other things, the requirements on which the test is based must be defined here, i.e. what must be tested against. These can be existing specification documents, such as technical concepts, or standards and norms, such as ISO 19139, which the geoportal must adhere to.
• Identification, structuring and prioritization of the test objects and functions to be tested: A web-based geoportal can be divided into different test objects, e.g. map clients, interfaces, search functions, etc. These must be identified and prioritized in terms of their criticality.
• Identification of the test data
• Definition of the test levels and phases: This determines which types of tests will be carried out or which activities the various test phases, such as test case specification, require.
• Description of the test infrastructure (test environment, test workstations, hardware and software, tools and instruments)
• Definition of error management (error classes, error workflow)
• Definition of interruption and resumption criteria
• Identification of risks and hazards
• Listing of contact persons, responsibilities and resources
• Identification of necessary training/education for test employees
• Scheduling of test activities (availability of testers, services)

In addition to the test concept, a test controlling system is set up which constantly provides the current status of the test operation. This allows statements to be made during the test case specification phase as to how many test cases still need to be described. During test execution, controlling provides an overview of the test cases performed and those still open, test case coverage and errors found. In addition, it is necessary to establish an error management system to enable traceable documentation of errors and to prevent errors from being “lost”.

The information from test controlling and error management can be prepared for interim and status reports. This ensures constant transparency of the test and problems can be identified at an early stage.

Analysis and design

The next step is for the testers to analyze the requirements documents and derive and specify test cases from them.

The methods defined in the test concept for test case determination and test case minimization are used here. The test cases are documented in a test case management tool. The documentation of the test scenarios and test cases includes at least:

• Unique ID for the test case
• Test idea and test objective
• Test case description and expected result of the test case execution
• Test data for the test case
• Prioritization of the test case
• Reference to the requirement from which this test case was created
• Date of test case creation and author of the test case

When analyzing the requirements, a distinction must be made between the various quality criteria to be tested, for which test cases are specified. For a web-based geoportal, some criteria that are typical for Internet applications are relevant here, including

• Functionality
• usability
• efficiency
• Security

Functionality

The functionality test primarily answers the question: “Does the developed and provided geoportal offer the functionalities that are defined in the requirements completely and correctly?” In addition to any standard functions as test objects, a geoportal can also have very complex functionalities, e.g. in connection with coordinate transformations. Some typical test objects for geoportals are listed below as examples:

• File functions
• Administration functions
• Map clients
• Coordinate transformation
• Gazetteer
• Catalog service
• interfaces

The second part of this article describes in more detail the special features to look out for in these various test objects.

Usability

The usability test is used to evaluate the usability of the geoportal and can cover the following topics, among others:

• User groups: Who will use this geoportal? For example, the depth of help and explanations or efficient navigation through the portal depends on who will be using the system. There will be different requirements if technical experts work with the portal or private users. Whether these requirements are implemented is the task of the usability test.
• Styleguide conformity: This tests whether the geoportal has been developed entirely in accordance with the specified style guide.
• Browser compatibility: With which Internet browsers or operating systems must the geoportal be viewable? It should also be noted that the complex structure of a geoportal may mean that support for essential functions may be sufficient.
• Accessibility: The issue of accessibility, i.e. the usability of the application by all users, regardless of physical and/or technical capabilities, must be carefully examined in the case of a geoportal. Due to the use of map material, complete accessibility is not possible. It must therefore be clarified in advance which requirements and functions are to be implemented barrier-free.

Efficiency

The efficiency test examines the geoportal for its behavior under load and overload, e.g. with many simultaneous accesses to the portal. Internet applications in particular need to be examined closely here due to the large number of potential users. Here too, it is important to define specifications in the requirements, e.g. how many users can be expected at the same time or over a period of time and how quickly the geoportal must respond to requests in certain load states. The regeneration of the system after an overload is also a possible test aspect here.

It should be noted with these tests that factors that cannot be influenced affect the results, e.g. the integration of WMS services from other providers.

A large number of supporting tools from the commercial and open source sector are available for testing efficiency, especially for Internet applications, which are helpful both for load generation and for evaluating the data.

Security

The topic of security is extremely important, especially for Internet applications, and is very often in the headlines. The system and, above all, the data must be protected against unauthorized access and changes. Users may only see the data for which they are authorized. The security aspect is very comprehensive and requires a high level of technical understanding. Problems with the infrastructure (e.g. the firewall, the network or the web or database server) or newly discovered security holes in applications can make the geoportal vulnerable. Security tests are usually carried out by specialists who also understand the technical background. It is also very important to repeat security tests at certain intervals, as updates, configuration changes and newly discovered security holes can give rise to new risks.

In addition to the technical aspects mentioned above, security must be considered from the perspective of the application. The question here is: Does every user only receive the content they are authorized to use via the geoportal?

A prerequisite for carrying out the test is the classification of users and their assignment to rights. In the actual tests, it must first be checked whether the user is correctly authenticated. Following authentication, the geoportal assigns users to their rights (authorization). In the case of Web Map Services, in addition to the layers, the rights can relate, for example, to scale ranges, the spatial extent or the retrieval of attribute data (GetFeatureInfo Request), whereas in the case of Web Feature Services they relate to the feature types and their attributes.

The correct implementation of these access rights must generally be checked at two levels: Firstly, with regard to the description of the portal services. At interface level, the capabilities of the services must already be filtered according to the user authorizations, i.e. the capabilities of the WMS only contain the layers that the user is allowed to see, for example. On the client side - to stay with the previous example - only the corresponding layers are displayed for selection. This security level corresponds to the concept of client-based security.

The second test level relates to specific requests to the portal services bypassing the portal clients, i.e. at interface level. Unauthorized requests to services must be blocked by the geoportal and answered with a corresponding message.

In preparation for the test execution, the need for test data is also identified in the course of analyzing the requirements and creating the test case. This test data must be available before the test is carried out. For this purpose, it will be necessary to create test data manually or automatically or to modify existing test data, whereby automatic generation is to be preferred. Another option for providing test data is to extract it from the existing system and, if necessary, anonymize it.

The test data for the test cases is stored in a traceable manner to enable the test cases to be repeated.

Realization and implementation

After preparing the test infrastructure and preparing the test cases, these are carried out by the testers. During the test phase, the functional tests as well as the efficiency, usability and security tests are carried out. In the case of a geoportal, it is important to ensure that all required external services are available during the tests.

The following information is logged for each test case during execution:

• Date/time of test execution
• Test employee: Who carried out the test case?
• Status of the test execution (OK, Not OK, possibly not feasible)
• In the event of an error, the error effect: How does the actual result deviate from the expected result? For example, are incorrect map contents or information displayed?

If an error is found, an entry must be created in error management. In addition to the exact error description and the metadata, such as the version of the geoportal in which the error occurred, the error is also assigned an error class that describes the severity of the error. The error classes are documented in the test concept.

If errors found in the version to be tested have been corrected, an error retest and a regression test are carried out.

Evaluation and report

At the end of the test, the recorded test results are evaluated, checked against the acceptance and end-of-test criteria and finally assessed in a final test report. The result of the test report is a recommendation as to whether the geoportal can go into operation or be accepted.

In the event that such a recommendation cannot be made, the reasons for this must be listed in detail.

Conclusion

An important activity at the end of the test project is the analysis and critical review of the entire test in order to incorporate the experience and findings in the course of test process improvement and in further test projects.

Finally, the components, drivers, test data, etc. used for the test are archived so that the test can be repeated or retraced if necessary.

Testing services and clients

This section describes selected tests and test ideas for the functionality of geoportal clients, services and interfaces.

Test of functional requirements

Due to the complexity of a geoportal, the functionality of the services, interfaces and clients is of particular importance. This also requires an intensive examination of the required functionalities and the underlying norms and standards. This part of a test project is discussed in more detail below.

Services and interfaces

The architecture of the geoportal is based on a loose coupling between the basic services and the geoportal on the one hand and the portal services and external applications on the other. This presupposes that defined interfaces are supported by the respective components. In the case of geo-services, these interfaces are described by the Open Geospatial Consortium (OGC) as implementation specifications and, if necessary, further specified by profiles. Compliance with these specifications is essential for portal components that are used for external communication and therefore plays a key role in the tests.

The tool of choice for testing compliance with OGC specifications is the CITE program (Conformance & Interoperability Test & Evaluation) offered by the OGC, which has now been completely revised. Among other things, tests for various services such as Web Map Services (WMS), Web Feature Services (WFS) and Catalog Services (CSW) are possible here. The relevant information on the procedure and terms of use can be found on the CITE homepage.

From the perspective of a portal, however, there are some special features that limit the use of CITE tests. The first of these is that the CITE server must have full access to the services to be tested. This can be problematic in development environments or in secure networks. Furthermore, the CITE scenario requires the integration of test data provided by the OGC into the services to be tested. However, portal services generally do not access their own data, but cascade to external services. In this respect, most CITE tests are not applicable to portal services. The tests of client components are not covered by the CITE program. In addition, tests for profiles and tests for the fulfillment of functional requirements, which are specifically tailored to the conditions in the respective geoportal, cannot be carried out with CITE. In this respect, own test ideas, where possible based on CITE, must be defined.

Map service (WMS)

The map service is the most frequently used service of a geoportal and is therefore of central importance.

The tests here are based on the three WMS interfaces GetCapabilities, GetMap and GetFeatureInfo. With a few exceptions, all interface tests must be carried out by calling the interface directly, e.g. by entering it as a browser URL, in order to rule out any errors in the WMS clients. The response from the service is then compared with the expected result. The specific structure of the corresponding requests will not be discussed here.

With regard to the GetCapabilities interface, for example, the following test ideas are useful:

• Validity of the Capabilities.xml according to the WMS version
• Correct behavior in the context of version negotiation (answering the request with the desired version, otherwise with the highest implemented version number)
• Correctness of all specified OnlineResource URLs
• Completeness of the metadata for the service (This test case is of secondary importance for the actual functionality of the service. In interoperable environments, however, this metadata is evaluated by catalog services, for example, and should therefore be listed in full in the capabilities of the WMS).

The GetMap interface is the central interface for map services. The tests for correct functioning are very extensive. For this reason, only a few essential test ideas are discussed here:

• Robustness of the service in the event of failure of connected basic services (this case should be brought about explicitly by switching off one of the basic services).
• Presence of the layers specified in the capabilities
• Support of the image formats and coordinate reference systems (SRS) specified in the capabilities
• Correct interpretation of the parameters for transparency and background color
• Provision of a map in the width/height ratio of the image requested by the client (independent of the width/height ratio of the bounding box)
• Test of the coordinate transformation of the WMS

As a rule, scale hints for individual or all layers are specified in the capabilities for security, performance and/or marketing reasons. In this case, tests should be carried out to ensure compliance. This requires some manual computing effort to generate GetMap requests with the correct content. However, the calculated bounding boxes can be reused in other test cases.

Another test complex relates to the representation of the layers. The following test ideas are to be defined here:

• Can each layer be displayed according to the layer styles specified in the capabilities?
• Is the assignment between layer and style correct according to the sequence in the GetMap request?
• Can the layers be requested in the default display (default style) via an empty STYLES parameter (STYLES=) or zero values in the style list (e.g. STYLES=,,,)? Is it possible to combine the latter notation with named styles?

If SLD-capable services are to be integrated into the geoportal, additional tests are carried out to check whether the corresponding SLD parameters (e.g. SLD=) are “passed through” to these services and the result map is rendered in the desired style.

When testing the GetFeatureInfo interface, it should first be checked whether the feature data query is possible for each layer that has been marked with the attribute queryable=“1” in the capabilities of the service. As with the GetMap interface, it should also be possible to generate all specified output formats for displaying information. However, if the corresponding parameter (INFO_FORMAT) is omitted, the output must still be in one of the formats specified in the capabilities.

In addition, it should be tested for each interface whether it works without errors with both the mandatory and optional parameters and whether a valid result is delivered even without vendor-specific parameters.

Data service

In geoportals, data services have the task of answering queries for geographical objects such as locations, addresses or regions (gazetteer service) and/or providing raster or vector data (web coverage service or web feature service). There is currently no OGC standard for gazetteer services; the corresponding paper currently has the status of a best practice document.

A gazetteer service is treated as an application profile of a WFS. If the gazetteer service of a geoportal is to use standardized interfaces, only the use of a WFS can be considered. In addition to the interface conformity of the service, the question of whether correct results are delivered is of particular interest for the use and further processing of the results. With this in mind, the following test ideas were selected, whereby only the currently most frequently implemented version 1.0.0 of a basic WFS is referred to here.

The following questions arise in connection with the GetCapabilities interface of a WFS:

• Is the capabilities response valid and does it correspond to the specified version number?
• Are feature types output correctly and is a valid SRS assigned to each of them?
• Is at least the GML2 format specified as the ResultFormat?

With regard to the DescribeFeatureType interface, it must first be checked whether the server response provides a valid GML schema and whether all FeatureTypes of the WFS are also described without specifying the TYPENAME parameter.

From a content perspective, when testing the DescribeFeatureType interface, particular attention must be paid to ensuring that only the attributes of the features that are to be “made public” via the service are described.

In relation to the GetFeature interface, it is then necessary to test whether only the data for these attributes is actually output. To do this, the PROPERTYNAME parameter in the GetFeature request must be given the value * and omitted completely in a second test. The following test ideas should also be used as a minimum:

• Validity of the returned GML document against the DescribeFeatureType XSchema including the imported GML schemas
• Support of the filter encoding methods specified in the capabilities
• Correctness of the query results when using filter encoding (e.g. comparison with SQL queries)

The last test is relatively time-consuming, as filter encoding queries can be very complex and meaningful queries must be created using the available data, the results of which must then be checked. You should be aware that these tests in particular only represent random samples.

It should also be checked for all WFS operations that both the HTTP request methods GET and POST are supported. While requests can be made using the GET method by entering the browser URL, this is not possible with POST. A specially developed website or application with an input form that is sent to the server via the POST method can help here (requires storage rights on a web server in the case of the website).

Coordinate transformation service

Coordinate transformations are required at various points in geoportals. The focus here is initially on server-side transformations; client-side transformations are dealt with elsewhere.

The corresponding functionality is usually not provided externally as an OGC web service, but only used internally. In this respect, the necessary tests are aimed at the results of the transformation and not at interfaces.

Coordinate transformations are most important in connection with the transformation of map images for WMS requests. Transformations can be realized by portal components if SRS are requested on the client side that are not supported by the basic services.

The transformation routine of the portal WMS can be tested using a separate reference data set. The recommended test dataset is, for example, a rectangle that is approximately the size of the geographical area for which the transformation is to be tested.

This rectangle is stored in different layers in the SRS of interest, whereby its (target) coordinates are calculated manually in each case. The GetMap request sent in the browser requests maps in the various SRSs, with the layer that is not to be transformed serving as a reference. Alternatively and more quickly, the bounding box can also be used as a reference.

If the geoportal offers functionalities for geo searches (e.g. locations, addresses, etc.) and for visualizing the search results, a coordinate transformation is required if the SRS of the map does not match that of the search results. If the search is carried out via a WFS, the coordinate transformation is carried out either by a corresponding coordinate transformation service (mandatory up to WFS version 1.0.0) or by the WFS itself (from version 1.1.0, whereby a transformation service can also be addressed internally here). Here too, the coordinate transformation service is only used within the geoportal to convert individual (GML) features in the requested SRS and is not offered as a service to the outside world. In this case, it is not necessary to test the interfaces of the service, but only the correct result of the transformation. This is naturally the case if the map and visualized GML features overlap correctly.

When interpreting the test results, it should be noted that coordinate transformations have different geometric accuracies. In order to be able to identify errors during the transformation, it is therefore necessary to know the accuracy potential of the respective transformation method and parameters: For conversions between Gauss-Krueger strips (e.g. EPSG codes 31466 to 31469), the ellipsoid is not changed and strict formulas are used. The accuracy here is also correspondingly high. However, in the case of transformations where a datum transition is required, e.g. from Gauss-Krueger coordinates on the Bessel ellipsoid to UTM coordinates on the GRS80 ellipsoid (EPSG codes for Germany: 25832 and 25833), the conversion is not carried out strictly, but via transformation parameters. There are a number of different parameter sets for Germany, which differ in their accuracy in relation to a (geographical) area of use. Accordingly, coordinate differences of a few meters can occur if target and actual coordinates were calculated using different parameter sets.

Administration functions

The administration options of the geoportal include the configuration of access rights to the various data, services and functions, as well as the integration and arrangement of various services, layers and specialist data in the portal clients.

Customers

While the functionality of geoservices is essentially defined by standardization, the functionality of clients can differ greatly between the various geoportals due to different requirements and implementations. For this reason, only selected test objects with corresponding test ideas can be discussed here.

Kartenclients

Map clients can offer a different range of functionalities depending on their design and use. This can also depend on the technology used, so a Java or AJAX-based client will offer more possibilities than an HTML client that may not even use Javascript.

The following typical functions of card clients could be used as test ideas:

• Navigation via maps: Sections of the maps can be enlarged, reduced or moved using certain factors.
• Calculations: Area and length calculations, but also the display of coordinates for specific points.
• Compositions: Adding, removing and rearranging different layers or services.
• File functions: Saving various settings, such as layer combinations, map sections and sizes on the portal or locally. These settings can be reloaded at a later point in time in order to continue working. Saving map sections and technical information is also possible.
• Print functions: The print functions of Internet browsers are not suitable for many situations; print functions of map clients can therefore often contain different print sizes, resolutions and information or already offer the print results as PDF files.

Coordinate transformations are also required for client components:

• Transformation of image coordinates into world coordinates for each map navigation to generate the bounding box of the map request
• Transformation of the bounding box for user-side changes to the SRS
• Display of world coordinates

In the simplest case, the transformation of the bounding box coordinates is checked using the map client’s coordinate display. If this is not implemented, tools for analyzing the network protocol can be used to view the incoming and outgoing data traffic on the client via the HTTP protocol and thus the transformed coordinates.

The coordinate display of the client can be checked via a simple target/actual comparison of control points.

Clients for address and location searches

Using these clients, requests for geographical regions, addresses or locations are forwarded to decentralized gazetteer services and the results found are usually visualized directly in the map client of the geoportal. The clients are therefore expected to formulate the corresponding requests correctly. In practice, this is checked indirectly by comparing the results of requests via the client with those of HTTP requests to the service.

It should also be tested whether different ways of entering the search terms lead to correct answers. These could be, for example

• Exact search by specifying quotation marks (e.g. “Zittauer Gebirge”)
• Consideration of placeholders (? or *)
• (optional) consideration of upper/lower case letters
• Search with umlauts or their replacements (e.g. ä -> ae, ß -> ss)
• Logical linking of search terms with AND, OR, NOT
• Missing address suffixes such as street, path, shore
• Fuzzy search with incorrect spelling, etc.

Further test scenarios then relate to the presentation of the results:

• Correct display of search results (e.g. location displayed correctly on the map),
• Consideration of the criteria established with regard to usability (e.g. sorted hit list with several hits with selection option by the user, “speaking” messages in case of incorrect input or empty result set, limitation of the number of hits, alternative request to refine the search criteria).

Clients to search for data and services

Catalog clients for searching for data and services usually query several ISO 19115/19119-based metadata catalogs. The prerequisite is that the client supports the search via the application profiles implemented in the connected catalog services, e.g. the DE profile of the ISO19115/ISO19119 application profile for OGC Web Catalogue Services (CSW-2.0).

In analogy to the address and location search, the hit list of a search query must also be checked for accuracy. This can be done by submitting identical queries via the geoportal’s metadata search client on the one hand and directly to the externally connected metadata catalogs on the other, e.g. via their clients or (better) as SOAP requests. Both responses must then be compared. Different input variants must also lead to correct results here.

Another test complex relates to the presentation of search results in the client. Here, test ideas are to be developed which test these with regard to

• Correctness of the content of the displayed results (comparison with the direct queries to the distributed metadata catalogs),
• completeness of the displayed metadata elements (depending on the agreed profile of the result set: BRIEF, SUMMARY or FULL) and
• Usability (e.g. meaningful sorting of the hit list, links to service providers or the option of uploading services found to the geoportal)

check.

Fazit

The rapid spread of geoportals and their use even by GIS laypersons, applications in critical areas such as internal security or disaster control and, last but not least, the special features of the underlying service-oriented architecture are some of the aspects that give rise to special quality requirements for the corresponding software solutions. The software tests required to ensure these quality requirements were the subject of this two-part article. In particular, it was shown how general IT testing and GI application know-how interlock in the specification and execution of the tests and thus contribute to the success of the project.

In the future, there will be further opportunities to make the necessary tests even more efficient, particularly in the area of test automation.