Running Tests

To run all tests go to the project build directory and run:

make test

Both test binaries and test reports can be found in:

${PROJECT_DIR}/build/bin

How to Write Tests

Writing tests involves editing at least two files. First, you need to add the test to the CMakeLists.txt of your component. You want to add your tests here using the following line:

armarx_add_test(TestName test/TestName.cpp "${DEPENDENT_LIBRARY_NAMES}")

Second, you need a C++ file test/TestName.cpp containing your tests. This should look as follows:

#define BOOST_TEST_MODULE ArmarX::Some::Unique::Identifier
#define ARMARX_BOOST_TEST
 
// The path to Test.h is depending on the ArmarX package name.
// It is further explained below.
#include <ArmarXCore/Test.h>
 
#include <path/to/component_headerfile>
 
BOOST_AUTO_TEST_CASE(testIntEqualsZero)
{
    int i = 0;
 
    BOOST_CHECK_EQUAL(i, 0);
}
 
BOOST_AUTO_TEST_CASE(testComponentInitialization)
{
    ComponentPtr = new Component();
 
    BOOST_CHECK_EQUAL(ComponentPtr->isInitialized(), true);
}

The most important things are the defines and the includes at the top of the file. Here, the unique name of the current test module must be defined which is then used in the automatically generated <ProjectName/Test.h> file which does all the necessary setup for creating testsuites and log output. The Test.h file is automatically generated in build/testing/<ProjectInstallName>/ and is unique for every ArmarX Package.

Afterwards all the tests are added using BOOST_AUTO_TEST_CASE(name) {} macros.

A list of the most commonly used test macros can be found here: http://www.boost.org/doc/libs/1_48_0/libs/test/doc/html/utf/testing-tools/reference.html

To run the complete testsuite the command 'make test' must be executed. The output of the tests is stored in XML files located in ${PROJECT_DIR}/build/bin where they can be found and evaluated by the continuous integration pipeline.

To get more information about occurred test failures the respective tests must be run manually by executing their test binaries. The console output will contain more detailed descriptions about unmatched conditions, wrong parameters or locations in the source code.

Testing in ArmarX Environment

The component you want to test may depend on the availability of other ArmarX components or you may wish to test the interaction with your component through Ice. In order to test your component within a running ArmarX context you need to create a test environment that looks similar to the following:

#include <path/to/your/component.h>
 
#include <ArmarXCore/core/test/IceTestHelper.h>
#include <ArmarXCore/core/application/Application.h>
#include <ArmarXCore/core/system/FactoryCollectionBase.h>
 
using namespace armarx;
 
class YourComponentTestEnvironment
{
public:
    YourComponentTestEnvironment(const std::string& testName, int registryPort = 11220, bool addObjects = true)
    {
        Ice::PropertiesPtr properties = Ice::createProperties();
        armarx::Application::LoadDefaultConfig(properties);
 
        // If you need to set properties (optional):
        // properties->setProperty("ArmarX.YourComponent.Property1", "Property1");
        // properties->setProperty("ArmarX.YourComponent.Property2", "Property2");
        // properties->setProperty("ArmarX.AnotherComponent.Property1", "Property1");
 
        // The IceTestHelper starts all required Ice processes
        _iceTestHelper = new IceTestHelper(registryPort, registryPort + 1);
        _iceTestHelper->startEnvironment();
 
        // The manager allows you to create new ArmarX components
        _manager = new TestArmarXManager(testName, _iceTestHelper->getCommunicator(), properties);
 
        if (addObjects)
        {
            // This is how you create components.
            _yourComponent = _manager->createComponentAndRun<YourComponent, YourComponentInterfacePrx>("ArmarX", "YourComponent");
            _anotherComponent = _manager->createComponentAndRun<AnotherComponent, AnotherComponentInterfacePrx>("ArmarX", "AnotherComponent");
        }
    }
 
    ~YourComponentTestEnvironment()
    {
        _manager->shutdown();
    }
 
    // In your tests, you can access your component through this proxy
    YourComponentInterfacePrx _yourComponent;
    AnotherComponentInterfacePrx _anotherComponent;
    TestArmarXManagerPtr _manager;
    IceTestHelperPtr _iceTestHelper;
};
 
using YourComponentTestEnvironmentPtr = std::shared_ptr<YourComponentTestEnvironment>;

In your test code all you need to do is import your test environment, create an instance of it and access your component via Ice through the proxy stored in you environment. See

armarx/ArmarXCore/source/ArmarXCore/statechart/test/StatechartIceTest.cpp
armarx/MemoryX/source/MemoryX/libraries/memorytypes/entity/test/KBMTest.cpp
armarx/RobotAPI/source/RobotAPI/components/RobotIK/test/RobotIKTest.cpp

for examples.

Testing Statechart Scenarios

It is possible to write tests for testing whole statechart-scenarios for correctness of their output-values. This requires following steps:

Adding output parameters to your statechart
Creating unit tests that can test your statechart

For adding output parameters to a statechart using the StatechartEditor, you can refer to Basic Statechart Editor Tutorial, Extended Statechart Editor Tutorial and Advanced Statechart Editor Tutorial. In some cases this might not be enough, so you have to modify the cpp files of your states directly. The following example can be found in armarx/Tutorials/WavingTutorial/source/WavingTutorial/statecharts/WavingTutorialGroup/MoveJoints.cpp.

void MoveJoints::run()
{
    // put your user code for the execution-phase here
    // runs in seperate thread, thus can do complex operations
    // should check constantly whether isRunningTaskStopped() returns true
 
    std::map<std::string, float> jointVelocityMap = in.getJointVelocity();
    NameControlModeMap velocityControlModeMap;
    for (const auto & jointVelocity : jointVelocityMap)
    {
        velocityControlModeMap[jointVelocity.first] = eVelocityControl;
    }
    KinematicUnitInterfacePrx kinUnit = getKinematicUnit();
    kinUnit->switchControlMode(velocityControlModeMap);
    kinUnit->setJointVelocities(jointVelocityMap);
 
 
    // Set output values
    RobotStateComponentInterfacePrx robotPrx = getRobotStateComponent();
    SharedRobotInterfacePrx robot = robotPrx->getSynchronizedRobot();
 
    std::map<std::string, float> jointResultMap;
    jointResultMap.insert(std::pair<std::string, float>("Elbow L", robot->getRobotNode("Elbow L")->getJointValue()));
 
    out.setJointValuesResult(jointResultMap);
}

After you set up the output parameters of your statechart, you can write unit test, just like in How to Write Tests.

Note: These tests will take a lot of time and need a lot of ressources, because every test is started in its own new environment, which contains several started components.

Here you can see a detailed example of how to write a statechart-scenario-test (armarx/Tutorials/WavingTutorial/source/WavingTutorial/statecharts/WavingTutorialGroup/test/WavingTutorialScenarioTestExample.cpp):

#define BOOST_TEST_MODULE ArmarX::Tutorials::WavingTutorialScenarioTestExample
#define ARMARX_BOOST_TEST
 
#include <WavingTutorial/Test.h>
 
// Includes the environment for testing whole statechart-scenarios
#include <ArmarXCore/statechart/test/StatechartScenarioTestEnv.h>
 
using namespace armarx;
 
BOOST_AUTO_TEST_CASE(WavingTutorialScenarioTestExample)
{
    // Initialising the testing-environment.
    StatechartScenarioTestEnvironment env("WavingTutorialScenarioTestExample");
 
    // This scenario sets up the simulation for ArmarX-Robots, it should be started before any other robot-related
    // scenario.
    // By passing the third argument, we let the environment wait until that given component is started (or 5000ms
    // have passed), to prevent situations where the simulation is not set up. The component 'Simulator' has
    // proven to be the best choice.
    // env.startScenario("ArmarXSimulation", "Armar3Simulation", "Simulator", 5000); //<-- this simple test case does not need the simulation
 
    // Now we start the scenario we want to test
    env.startScenario("WavingTutorial", "StartStateChart");
 
    // Since we cannot detect if a statechart gets stuck in an infinite loop or something similar,
    // we set a maximal execution time (in milliseconds).
    // If this time is over and the statechart has not finished, an exception is thrown and therefore
    // the whole test fails.
    env.watcher->waitForStateChartFinished(90000);
 
    // After the statechart has finished, we can extract the output parameters of the main-state
    // of the tested statechart and test these parameters afterwards
    StringVariantContainerBaseMap statechartOutput = env.watcher->getStateChartOutput();
 
    // Extracting the actual result values of the output parameters.
    VariantContainerBasePtr counterResult = statechartOutput.find("counterResult")->second;
    VariantPtr counterResultVariant = (SingleVariantPtr::dynamicCast(counterResult))->get();
    int counterResultValue = counterResultVariant->getInt();
 
    VariantContainerBasePtr finalJointValues = statechartOutput.find("FinalJointValues")->second;
    VariantPtr elbowLVariant = (StringValueMapPtr::dynamicCast(finalJointValues))->getVariant("Elbow L");
    float elbowLValue = elbowLVariant->getFloat();
 
    // Example-tests
    // -----------------------
    // Expected results:
    // counterResult -> 5
    // FinalJointValues -> "ElbowL" -> -0.8000000119209290 (the value of the input parameter 'JointMapValueWaveBack' -> "Elbow L")
 
    BOOST_CHECK_EQUAL(counterResultValue, 5);
 
    float expextedElbowLValue = -0.8000000119209290f;
    float eps = 7.0f; // percent
 
    BOOST_CHECK_CLOSE(elbowLValue, expextedElbowLValue, eps);
}

If you want to inspect the memories after the statechart has finished and use their content for your tests, you have to use a different environment, which is shown in the example below (armarx/Tutorials/pick_and_place_tutorial/source/pick_and_place_tutorial/statecharts/PickAndPlaceTutorialGroup/test/PickAndPlaceTutorialScenarioTestExample.cpp).

Note: This example does not work reliably, because the Advanced Statechart Editor Tutorial does not finish always with the same results. They depend on the calculation of the physics of the added object and these might vary.

#define BOOST_TEST_MODULE ArmarX::Tutorials::PickAndPlaceTutorialScenarioTestExample
#define ARMARX_BOOST_TEST
 
#include <pick_and_place_tutorial/Test.h>
 
// Includes the environment for testing whole statechart-scenarios with access to memory components
#include <MemoryX/core/test/StatechartScenarioMemoryTestEnv.h>
 
using namespace armarx;
using namespace memoryx;
 
BOOST_AUTO_TEST_CASE(PickAndPlaceTutorialScenarioTestExample)
{
    // Initialising the testing-environment, but this time with access to memory components.
    // It is possible to set the path to a specific database, that should be used for this test.
    StatechartScenarioMemoryTestEnvironment env("PickAndPlaceTutorialScenarioTestExample", "ArmarXDB", "ArmarXDB/dbexport/memdb");
    env.startScenario("ArmarXSimulation", "Armar3Simulation", "Simulator", 5000);
    env.startScenario("pick_and_place_tutorial", "PickAndPlaceScenario");
 
    env.watcher->waitForStateChartFinished(90000);
 
    // The output parameters of the statechart are not used in this example, but they get accessed like this
    StringVariantContainerBaseMap statechartOutput = env.watcher->getStateChartOutput();
 
 
    // Trying to check the position of the objectInstance "vitaliscereal" after the statechart has finished correctly
    try
    {
        ObjectInstanceMemorySegmentBasePrx objectInstances = env.memoryAccess->workingMemory->getObjectInstancesSegment();
        ObjectInstancePtr vitaliscereal = ObjectInstancePtr::dynamicCast(objectInstances->getObjectInstanceByName("vitaliscereal"));
 
        BOOST_CHECK_EQUAL(vitaliscereal, !false);
 
        float eps = 1.0f; // percent
 
        int expectedXPosition = 2089;
        int expectedYPosition = 5221;
        int expectedZPosition = 788;
 
        BOOST_CHECK_CLOSE(vitaliscereal->getPosition()->x, expectedXPosition, eps);
        BOOST_CHECK_CLOSE(vitaliscereal->getPosition()->y, expectedYPosition, eps);
        BOOST_CHECK_CLOSE(vitaliscereal->getPosition()->z, expectedZPosition, eps);
 
    }
    catch (...)
    {
        handleExceptions();
    }
}

Code Coverage Reports

Generating Code Coverage reports takes quite some time and is therefore disabled by default. To enable it the following steps need to be performed:

install the "lcov" and "Boost unit_test_framework" package
cd ${Package_DIR}/build
cmake -DARMARX_BUILD_TESTS=TRUE -DARMARX_ENABLE_COVERAGE=TRUE ..
make clean all
make coverage

Afterwards, the coverage report in HTML format can be found in:

${Package_DIR}/build/coverage/index.html

Table of Contents

Running Tests

How to Write Tests

Testing in ArmarX Environment

Testing Statechart Scenarios

Code Coverage Reports