Troubleshooting in ArmarX

Table of Contents

Prerequisites: None

As every programmer probably has experienced already, programs do not always do what the programmers intends them to do. The usual procedure to find problems is to attach the debugger and have a closer look what really happens in the program.
Due to the distributed nature of ArmarX, this might not get you far in an ArmarX setup: Programs communicate with other programs over the middleware Ice and the debugger cannot breach this barrier. Thus, the debugger has to treat it like a program-part without debugging information.

In this tutorial, we explain best practices in ArmarX to troubleshoot problems in ArmarX.

Overview

The general order of actions for troubleshooting is as follows:

  1. Inspect log for error messages
  2. Inspect component status with SystemStateMonitor GUI
  3. Attach debbuger to running component

Log Output Inspection

The first step for trouble shooting is always looking at the log output.
All std::cout and std::cerr calls appear in the ArmarX Logging Topic. Further, all ArmarX components use the ArmarX Logging facilities with additional meta information like file and line number and for warning messages (or worse) also a backtrace. The log is easiest inspected in the ArmarX LogViewer GUI. This GUI groups all log messages by application and highlights warnings and errors with reddish colors. Searching and filtering is also possible.
Most problems are reported with a warning or error logging level and this should be the first point to be looked at.

Note
Due to the distributed and multithreaded nature of ArmarX, errors are not always at the end of the log. So look through the complete log for problems. Also have a look at the lines surrounding the error. They sometimes also give more information about the error.

Inspecting the system state

If your setup does not seem to start completetly, you might want to look at the system state.
ArmarX provides the feature to specify required dependencies of components. Inherently, this leads to the fact that components will not start until all dependencies are met. Why objects are not started can have many reasons, which are usually reported in the log (refer back to Log Output Inspection). To find out which components are not started the SystemStateMonitor GUI can be used. This GUI is part of the ArmarXGUI package and should always be available. Every component that is waiting for dependencies or is currently starting is yellow. If it was successfully started, it will be green. The dependencies of components are shown in the subtree of each component. Available dependencies are green while missing dependencies are red. This way it is easily possible to find out why a component is not starting.

It is also possible to inspect all running armarx tasks (armarx::PeriodicTask and armarx::RunningTask) with the ThreadViewer to identify hanging periodic tasks or inspect the caused system load.

Using the C++ Debugger

When the faulty component has been identified and the error is not identifiable it is time to attach the debugger to the application. This works like for any other C++ application. Since ArmarX application are usually started with additional command line parameters it is often useful to start the application normally first, attach the debugger at runtime and trigger then the execution. Of course, this does not work if the problem arises at start up. Then you need to get the command line parameters for that application and add them to the debugger (you can get the command line parameters with ps aux | grep $yourAppName). If you do not know how to use a debugger, a short tutorial is available: Debugging Tutorial .