Eclipse.org Eclipse.org - Device Kit

Validation Testing Introduction

Getting Started with Validation Testing

The following steps are needed to create and run a validation test suite.

  1. Implement one or more TestAgent bundles, without an activator, with one TestCase or a TestSuite with a collection of TestCases per TestAgent.
  2. For deployment in an OSGi environment, implement a TestAgentBundle, TestAgentManaged or a TestAgentFactory activator bundle for each TestAgent.
  3. Write an XML test script.
  4. Create a deployment configuration to deploy the application under test, the TestAgents, TestController and optionally the TestManager.
  5. Run the test script and verify the test report.

These steps will be described in more detail in the following sections, and examples will be given.

Step 1: Implement a new TestAgent

The first step is to create a new bundle project that implements a concrete subclass of TestAgent and a specialization of the TestAgentService interface (in order to be able to identify this specific test agent in an OSGi environment). For the subclass of TestAgent the only requirement is to implement the framework methods that return a description and a test class. The test class must follow the JUnit 3 convention, that is, either provide a static suite() method or provide one or more instance methods starting with "test".

import java.util.Dictionary;
import java.util.Hashtable;
import org.eclipse.soda.dk.testagent.TestAgent;
import org.eclipse.soda.dk.testagent.service.TestSynchronizationService;
 
public class ExampleTestAgent extends TestAgent {
 
      public static final String DESCRIPTION = " Example TestAgent";
 
      public String getDescription() {
            return DESCRIPTION;
      }
 
      protected Class loadTestClass() {
            return ExampleTestSuite.class;
      }
 
}

Please note that the Device Kit provides a wizard to create the projects for a new TestAgent with the skeleton code for a TestSuite with a single TestCase.

Figure 1 - New TestAgent wizard

Here is an example of a TestSuite. It is an example of a ConfiguredTestSuite, taken from org.eclipse.soda.dk.testagent.test.factory, which is the self-test for TestAgent. In this example you see that the test suite runs certain test cases only if certain capabilities are available in the execution environment.

public class TestAgentTestSuite extends ConfiguredTestSuite {
 
      public static Test suite() {
            TestAgentTestSuite suite = new TestAgentTestSuite();
            suite.addTest(ConfigurationTestCase.class);
            if (suite.getContext().getSynchronizationService() != null) {
                  suite.addTest(SynchronizationTestCase.class);
            }
            if (suite.getContext().getNotificationService() != null) {
                  suite.addTest(NotificationProbeTestCase.class);
            }
            return suite;
      }
 
}

You can open the source code for ConfigurationTestCase, SynchronizationTestCase, and NotificationProbeTestCase to see basic examples of using the configuration, synchronization and notification features provided by the TestAgent framework.

Following the common coding patterns in the Device Kit, the ExampleTestAgentService interface should be defined as follows:

import org.eclipse.soda.dk.testagent.service.TestAgentService;
public interface ExampleTestService extends TestAgentService {
      public static final String SERVICE_NAME = ExampleeTestService.class.getName();
 
}

Step 2: Implement activators for the new TestAgent

To implement the ExampleTestAgentBundle, ExampleTestAgentManaged and ExampleTestAgentFactory activator bundles, create a new bundle project for each and extend the framework activator class provided by TestAgentBundle, TestAgentManaged and TestAgentFactory. The minimum requirement is to provide an implementation for:

  1. getExportedServiceNames, to list the interfaces that will be used to register the TestAgent.
  2. createProperties, to define the specific registration properties for this service. Note that the SERVICE_PID property will only be used for unmanaged services.
  3. createService, to create a new instance of your TestAgent.

The Device Kit wizard for creating a new TestAgent also creates the desired activator projects.

Figure 2 - Creating Activator projects with the New TestAgent wizard

Here is the example for TestAgentTestFactory.

import java.util.Hashtable;
import org.eclipse.soda.dk.testagent.TestAgent;
import org.eclipse.soda.dk.testagent.factory.TestAgentFactory;
import org.eclipse.soda.dk.testagent.service.TestAgentService;
import org.eclipse.soda.dk.testagent.test.TestAgentTest;
import org.eclipse.soda.dk.testagent.test.service.TestAgentTestService;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.Constants;
 
public class TestAgentTestFactory extends TestAgentFactory implements
            BundleActivator {
 
      public static final String CLASS_NAME = "org.eclipse.soda.dk.testagent.test.factory.TestAgentTestFactory"; //$NON-NLS-1$
 
      public static final String[] EXPORTED_SERVICE_NAMES = new String[] {
            TestAgentService.SERVICE_NAME,
            TestAgentTestService.SERVICE_NAME
      };
 
      public String[] getExportedServiceNames() {
            return EXPORTED_SERVICE_NAMES;
      }
 
      public Hashtable createProperties() {
            final Hashtable properties = super.createProperties();
            properties.put(Constants.SERVICE_PID, TestAgentTestService.SERVICE_NAME);
            properties.put(Constants.SERVICE_DESCRIPTION, TestAgentTest.DESCRIPTION);
            properties.put(Constants.SERVICE_VENDOR, "Eclipse"); //$NON-NLS-1$
            properties.put(Constants.SERVICE_RANKING, "6"); //$NON-NLS-1$
            return properties;
      }
 
      public TestAgent createService() {
            return new TestAgentTest();
      }
}

If it is necessary to acquire additional optional or non-optional services and make them available to the test via the TestContext, one can do so by overriding the applicable methods of the abstract framework activator class that is being used.

Step 3: Write an XML test script

The test script language is introduced with an example. This example is a test script to exercise the TestAgent, TestController and TestManager themselves by running sequential and parallel tests implemented in the TestAgentTest. The example test script is provided in file testagent.test.parallel.xml in project org.eclipse.soda.dk.testagent.test.

 Figure 3 - Example Test Script

To view the example test script, run the Device Kit Samples wizard to create project org.eclipse.soda.dk.testagent.test.factory in the workspace and double-click on file testagent.test.parallel.xml in the project. This opens the validation test script in the Device Kit test script editor, and also lets you run the script in the test manager view in the Eclipse workbench, as shown in the following snapshot.

 Figure 4 - Test Script Editor

Notes:

The Device Kit wizard for creating a new TestAgent creates a sample test script in each of the activator projects for the new agent.

Although not shown in this example, it is not only possible to use wildcards in the configuration OSGi LDAP filter specification, but also in the test filter specification. In case multiple tests match the filter all matching tests are run concurrently if the attribute match="multiple" is specified.

          <test filter="(id=*)" match="multiple">
          </test>

The attributes attributes for the configuration element are the following:

        <configuration factoryPid="..." pid="..." filter="..." bundleLocation="..." match="single|multiple">
          ...
        </configuration>

  1. ConfigurationAction type="create":

  2. ConfigurationAction type="createOrUpdate":

  3. ConfigurationAction type="update" (yes, there also is an update):

  4. ConfigurationAction type="delete":

The type attribute of the <testAction> element is optional and has possible values "sequential" and "parallel". Sequential is the default.

The <test> element has optional attributes: id, filter, class and match.

The <property> element has optional attributes: type and cardinality.

The <title>, <version>, <vendor> and <description> elements are optional and can appear in any order.

Note: the XML schema definition (dktest.xsd) with detailed documentation annotations can be found in the reference section of the Device Kit documentation.

Step 4: Creating a launch configuration for running the new TestAgent

To create a launch configuration for running the test script of the previous section in the Equinox OSGi target runtime, the following bundles must be loaded.

Please note that the Device Kit wizard for creating a new TestAgent also creates a sample launch configuration in each of the activator projects of the new agent.

Step 5a: Running the test script in the OSGi target runtime using the Eclipse workbench view

To run the example test script using the web-based interface of the TestManager, perform the following steps:

  1. Run the target runtime using the Equinox launch configuration created in the previous section (the optional set of bundles for running the test manager and the test manager servlet for the web-based user interface can be omitted). Note that the org.eclipse.soda.dk.testcontroller.remote.rmi bundles require support for Remote Method Invocation in the target Java runtime environment.
  2. Click on the Run Validation Test Script icon in the tool bar of the Eclipse workbench after opening the test script in the editor (see figure 4), or by selecting from the context menu on the test script, as follows:

    3.  Figure 5 - Test script context menu


  3. Navigate to the http://localhost/device/test/manager
    4. servlet.
  4. The validation test view opens in the current workbench perspective and a workbench job progress dialog opens.

    5.  Figure 6 - Test script running in the Eclipse workbench


  5. Observe the execution of the test in the test propgress tree view and interact with the validation test confirmation prompt dialogs. Note that multiple confirmation dialogs may open when running tests in parallel. Any of the confirmation dialogs can be answered, or the test run can be terminated by clicking the Cancel button in the progress dialog or the Stop Test icon in the toolbar of the validation test view.

    6.  Figure 7 - Running and interacting with the test

  6. Verify the outcome of the test. If errors or failures have occurred, click the Next Failed Action or Test or Previous Failed Action or Test icons on the toolbar of the view, and the details of the error and failure appear in the Failure Trace part in the view. Double-click on a line in the Failure Trace to open the Java editor on the corresponding line in the source code.

    7.  Figure 8 - Reviewing the test result

  7. Optionally, rerun the test script or save the report of the test view in a file in the workbench. After saving a test report, double-click to load it into the test run history of the validation test view.

    8.  Figure 9 - Rerunning the test script or saving the test report


Step 5b: Running the test script in the OSGi target runtime using the servlet user interface

To run the example test script using the web-based interface of the TestManager, perform the following steps:

  1. Run the Equinox launch configuration created in the previous section, including the optional set of bundles for running the test manager and the test manager servlet for the web-based user interface. The org.eclipse.soda.dk.testcontroller.remote.rmi bundles are not needed in this case.
  2. Open the main servlet of the Device Kit, http://localhost/device

    3.  Figure 4 - Main servlet of the Device Kit


  3. Navigate to the http://localhost/device/test/manager
    4. servlet.
  4. Select the test script for uploading to the TestManager.

    5.  Figure 10 - Uploading the test script


  5. Observe the execution of the test in the test log panel and interact with the tests through the control panel.

    6.  Figure 11 - Running and interacting with the test


  6. Verify the test summary printed at the end of the test.

Step 5c: Running a test script from the command line as a standard Java application

To run the second example test script in a non-OSGi environment, perform the following steps:

  1. Create a launch configuration or script with the TestManager, TestController and TestAgent jars on the classpath.
  2. Run main class org.eclipse.soda.dk.testmanager.TestManager with the URL or path name of the test script optionally as program argument (test.xml is the default).
    Example:

    java org.eclipse.soda.dk.testmanager.TestManager test.xml

  3. Observe the execution of the test and interact with the tests through the console.
  4. Verify the test summary printed at the end of the test.

 

A version of the example test script that would exercise the TestAgentTest in a non-OSGi environment would looks as follows. Note that in such an environment configuration actions are not supported and TestAgents must be identified by their fully qualified class name.

<?xml version="1.0" encoding="UTF-8"?>
<validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">
  <title>Test of the Device Kit Test Framework</title>
  <version>1.0.0</version>
  <vendor>Eclipse</vendor>
  <description>Validation test for TestManager, TestController and TestAgent</description>
  <actions>
    <testAction type="sequential">
      <tests>
        <test class="org.eclipse.soda.dk.testagent.test.TestAgentTest">
          <properties>
            <property key="id" value="TestAgentTest-1"/>
              <property key="testKey" value="testValue"/>
          </properties>
        </test>
        <test class="org.eclipse.soda.dk.testagent.test.TestAgentTest">
          <properties>
            <property key="id" value="TestAgentTest-2"/>
              <property key="testKey" value="otherValue"/>
              <property key="expectedValue" value="otherValue"/>
          </properties>
        </test>
      </tests>
    </testAction>
    <testAction type="parallel">
      <tests>
        <test id="parA" class="org.eclipse.soda.dk.testagent.test.TestAgentTest">
          <properties>
            <property key="id" value="TestAgentTest-1"/>
              <property key="testKey" value="testValue"/>
          </properties>
        </test>
        <test id="ParB" class="org.eclipse.soda.dk.testagent.test.TestAgentTest">
          <properties>
            <property key="id" value="TestAgentTest-2"/>
              <property key="testKey" value="otherValue"/>
              <property key="expectedValue" value="otherValue"/>
          </properties>
        </test>
      </tests>
    </testAction>
  </actions>

</validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">

Example scripts for Standalone, Bundle-, Managed- and Factory-based testing

Depending on how the test is deployed, the test script will be slightly different. The following examples illustrate the differences for running the TestAgentTest in a non-OSGi environment and using either the Bundle, the Managed or the Factory-based activator in an OSGi environment.

The differences are highlighted in boldface.

Please note that the Device Kit wizard for creating a new TestAgent creates these sample test scripts in each of the activator projects of the new agent.

Sample script for testing in a standard Java runtime environment

<?xml version="1.0" encoding="UTF-8"?>
<validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">
  <title>TestAgentTest</title>
  <version>1.0.0</version>
  <vendor>Eclipse</vendor>
  <description>Static test for TestAgent</description>
  <actions>
    <testAction type="sequential">
      <tests>
        <test id="TestAgentTest" class="org.eclipse.soda.dk.testagent.test.TestAgentTest">
          <properties>
              <property key="testKey" value="testValue"/>
          </properties>
        </test>
      </tests>
    </testAction>
  </actions>
</validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">

Sample script for testing with a Bundle activator

<?xml version="1.0" encoding="UTF-8"?>
<validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">
  <title>TestAgentTest</title>
  <version>1.0.0</version>
  <vendor>Eclipse</vendor>
  <description>Bundle test for TestAgent</description>
  <actions>
    <!-- Invoke the TestAgentTest. -->
    <testAction type="sequential">
      <tests>
        <test id="TestAgentTest" filter="(service.pid=org.eclipse.soda.dk.testagent.test.service.TestAgentTestService)">
          <properties>
              <property key="testKey" value="testValue"/>
          </properties>
        </test>
      </tests>
    </testAction>
  </actions>
</validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">

Sample script for testing with a Managed activator

<?xml version="1.0" encoding="UTF-8"?>
<validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">
  <title>TestAgentTest</title>
  <version>1.0.0</version>
  <vendor>eclipse</vendor>
  <description>Managed test for TestAgent</description>
  <actions>
    <!-- Create or update test configurations. -->
    <configurationAction type="createOrUpdate">
      <configurations>
        <configuration pid="org.eclipse.soda.dk.testagent.test.managed.TestAgentTestManaged">
          <properties>
              <property key="id" value="TestAgentTest"/>
          </properties>
        </configuration>
      </configurations>
    </configurationAction>
    <!-- Invoke the TestAgentTest. -->
    <testAction type="sequential">
      <tests>
        <test id="TestAgentTest" filter="(id=TestAgentTest)">
          <properties>
              <property key="testKey" value="testValue"/>
          </properties>
        </test>
      </tests>
    </testAction>
  </actions>

</validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">

Sample script for testing with a Factory activator

<?xml version="1.0" encoding="UTF-8"?>
<validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">
  <title>TestAgentTest</title>
  <version>1.0.0</version>
  <vendor>Eclipse</vendor>
  <description>Factory-based test for TestAgent</description>
  <actions>
    <!-- Create new test configurations. -->
    <configurationAction type="create">
      <configurations>
        <configuration factoryPid="org.eclipse.soda.dk.testagent.test.factory.TestAgentTestFactory">
          <properties>
            <property key="testId" value="TestAgentTest-1"/>
          </properties>
        </configuration>
      </configurations>
    </configurationAction>
    <!-- Invoke the TestAgentTest. -->
    <testAction type="sequential">
      <tests>
        <test id="TestAgentTest" filter="(testId=TestAgentTest-1)">
          <properties>
              <property key="testKey" value="testValue"/>
          </properties>
        </test>
      </tests>
    </testAction>
    <!-- Delete all created test configurations. -->
    <configurationAction type="delete">
      <configurations>
        <configuration filter="(service.pid=org.eclipse.soda.dk.testagent.test.factory.*)"/>
      </configurations>
    </configurationAction>
  </actions>
</validationTest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="dktest.xsd">

Copyright © 2007 IBM. All Rights Reserved.