Remote and Distributed Validation Testing

Remote and Distributed Testing with the Validation Testing Framework

The validation testing framework supports test configurations where a test manager controls one or more test controllers running on remote target systems. This is the usual configuration, where the test manager and its user interface run in the Eclipse workbench, and the test runs in an OSGi runtime running on a separate JVM, possibly on another system.

Remote and distributed testing provides the following advantages:

There is no overhead of running the test manager and its user interface on the system under test.
Test runs can be coordinated between multiple target system, using the test script's synchronization constructs, for example for distributed integration testing or performance testing.

The validation testing framework supports pluggable remote access of test controllers from the test manager, and an implementation is provided that using Java Remote Method Invocation for remote access. The following diagram shows a configuration where a test manager on a host system runs a test script that executes a specific test on two remote target systems concurrently.

The test script in this example specifies the execution of a test action on two different controllers. The optional controller attribute of the <test> and <configuration> elements is an optional attribute that specifies the remote test controller that will receive the request to execute a test. When the controller attribute is omitted, the default value "" is assumed, designating the default controller. The test manager does not interpret the value of the controller attribute, but delegates controller resolution to controller locater services that have been installed. This allows for configuration-dependent resolution and for multiple implementations of the remote connection.

The test manager uses the following algorithm to resolve the controller:

Delegate controller resolution to each of the installed controller locater services until the first match.
If there is no match, attempt to resolve to a controller service using the OSGi framework. The value of the controller attribute is used to filter on the name attribute of the available OSGI controller services.
If there is still no resolution, use the local controller in the JVM the test manager is running on.

The RMI implementation of the Remote Test Controller

Remote Method Invocation (RMI), a Java Technology that has been part of Java SE platform since the beginning, is a standard technology for distributed Java applications with a relatively low overhead. The RMI protocol makes Java objects available on remote systems via remote interfaces, using TCP/IP and object serialization for sending remote references and other objects over a network. For target systems not capable of running Java SE, add-on implementations of the RMI may exist. For example, IBM provides RMI as an optional package (implementing the JSR-66 RMIOP specification) that can be installed in the J9 runtime of the WebSphere Everyplace environments.

The following diagram shows the components of the Validation Testing Framework in a remote configuration using RMI.

Note that remote objects (called stubs in RMI) will be exported from both host and target system. A test controller running on the target is exported as remote object so that the central test manager can access it, and the test run listener and synchronization interfaces running on the host are exported as remote objects so that the target systems can use them to call back to. This is illustrated in the following class diagram.

Example: Running the Test of the Test Agent Remotely

The org.eclipse.soda.dk.testcontroller.remote.rmi project provides example launch configurations and a test script for running the TestAgentTest on a remote runtime.
The example test script, testcontroller.remote.rmi.target.xml, configures and runs a test agent on a remote runtime, as shown in the following diagram.

To run this example using the test manager in the Eclipse workbench, perform the following steps:

Make sure there is a route to the test system using host name 'target'.
Start the target runtime using the testcontroller.remote.rmi.target launch cofiguration.
Open the example test script testcontroller.remote.rmi.target.xml from project org.eclipse.soda.dk.testcontroller.remote.rmi in the test script editor in the Eclipse workbench on the host system
Click the Run Test Script icon in the main toolbar to execute the remote test.

To run this example using the test manager web-based interface running remotely, perform the following steps:

Make sure there is a route to the test system using host name 'target'.
Start the target runtime using the testcontroller.remote.rmi.target launch cofiguration.
Start the host runtime using the testcontroller.remote.rmi.host launch cofiguration
Open http://localhost/device in a web browser and click on the testmanager servlet.
Select the example test script testcontroller.remote.rmi.target.xml from project org.eclipse.soda.dk.testcontroller.remote.rmi to execute the remote test.

The tests should pass, and you can inspect the debug messages in the console output of the target and host runtimes.

Notes:

On the target system a server socket is created that accepts remote test controller connections from the host system. By default this socket binds to port 2099. The testcontroller.registry.port property, defined as system property or in esc.properties, can be used to override the default port number. In case the default port is not used, the port can be specified as controller="host:port" in the XML script.
It is not necessary to start the RMI registry. The remote stub for the test controller is retrieved directly from the target runtime.
On the target runtime, InetAddress.getLocalHost() must return an IP address at which the target system can be reached from the host, and vice versa. The reason is that commonly RMI implementations use this address in the remote stubs that are exchanged.

Proxy Test Controllers (deprecated)

Version 1.0 of the validation testing framework used a simplified test controller resolution algorithm that did not include the pluggable test controller locater service, but always required a proxy controller services to be created. This approach is still supported and is shown in the following diagram.

The following test script example configures a proxy controller on the host runtime and uses it to create and run a test agent on a remote runtime.