VisionX: Implementing ImageProviders

Image providers in VisionX allow to send images in the ArmarX framework. The visionx::ImageProvider ManagedIceObject is the basic superclass for image providers. The superclass provides the following features

• image transport via ethernet or shared memory: whether shared memory is used or not is determined by comparing the IP adresses of provider and processor
• wide range of imange formats, including: eBayerPattern, eGrayScale, eRgb, eFloat1Channel, eFloat3Channels
• optional support for frame per second synchronization using the VisionX::CapturingImageProvider as superclass
• visualization of the images using the ImageMonitor plugin for ArmarXGui

Creating an imageprovider

Image providers usually abstract imageing hardware such as cameras, RGB-D sensors, or allow video access through files. How to integrate a new image source as image provider is achieved by subclassing the visionx::ImageProvider or visionx::CapturingImageProvider in the following way:

#include <VisionX/core/CapturingImageProvider.h>
 
namespace visionx
{
    class ExampleImageProvider :
        virtual public visionx::CapturingImageProvider
    {
    public:
        /**
         * @see armarx::ManagedIceObject::onInitComponent()
         */
        virtual void onInitCapturingImageProvider();
 
        /**
         * @see armarx::ManagedIceObject::onConnectComponent()
         */
        virtual void onConnectCapturingImageProvider()  { }
 
        /**
         * @see armarx::ManagedIceObject::onDisconnectComponent()
         */
        virtual void onDisconnectCapturingImageProvider() { }
 
        /**
         * @see armarx::ManagedIceObject::onExitComponent()
         */
        virtual void onExitCapturingImageProvider() { }
 
        /**
         * @see armarx::Component::getDefaultName()
         */
        virtual std::string getDefaultName() const
        {
            return "ExampleImageProvider";
        }
    };
}

The methods that need to be implemented are derived from the armarx::ManagedIceObject :

• The visionx::ImageProvider::onInitCapturingImageProvider method is called once the object is constructred
• The visionx::ImageProvider::onConnectCapturingImageProvider method is called once all network dependencies are resolved
• the visionx::ImageProvider::onDisconnectCapturingImageProvider method is called if a network dependency is lost
• the visionx::ImageProvider::onExitCapturingImageProvider method is called once the application is terminated

As for every ManagedIceObject, the default name of the object needs to be provided using the getDefaultName() method.

Setting up an imageprovider

The setup of the image providers settings such as image format and number of images needs to be performed in the onInitImageProvider method:

void ExampleImageProvider::onInitImageProvider()
{
    // set number of images to provider
    setNumberImages(2);
 
    // set image format
    ImageDimension imageSize(640,480);
    ImageType type = eRgb;
 
    setImageFormat(imageSize, type);
}

Possible image types are:

• eBayerPattern
• eGrayScale
• eRgb
• eFloat1Channel
• eFloat3Channels

For bayerpattern image types an optional parameter is used to set the bayerpattern format from the following list:

• eBayerPatternBg, eBayerPatternGb, eBayerPatternGr, eBayerPatternRg

Providing images

The visionx::ImageProvider offers two ways of providing images:

1. use one of the utility methods from the ImageProvider interfaces
2. use direct access to the image buffer

If possible make use of the first method. For performance reasons it can be necessary to directly access the image buffer without copying which is possible using the second method.

Providing images using utility methods

The following image format specific utility methods are implemented in visionx::ImageProvider:

• visionx::ImageProvider::provideImages(void** inputBuffers)
• visionx::ImageProvider::provideImages(CByteImage** images)
• visionx::ImageProvider::provideImages(CFloatImage** images)

The first method takes an arbitrary image type, the user needs to take care about the proper size and format of the input buffer. All utility methods copy the specified images to the internal buffer.

Providing images via image buffer

For efficiency reasons direct access to the image buffer is possible without the necessity to copy the image (for a second time) in the following way:

{
    armarx::SharedMemoryScopedWriteLockPtr lock = getScopedWriteLock();
 
    ImageFormatInfo imageFormat = getImageFormat();
    int imageSize = imageFormat.dimension.width * imageFormat.dimension.height * imageFormat.bytesPerPixel;
    memcpy(imageBuffers[0], images[0]->pixels, imageSize);
    memcpy(imageBuffers[1], images[1]->pixels, imageSize);
}

Using this method, the user has to take care for setting the write lock on the image buffer as shown in the above code.

Using the capturing image provider

Most image providers will have a capturing thread, that performs the capturing of the images. Such a capturing thread is part of the visionx::CapturingImageProvider:

/**
 * Main capturing function.
 *
 * @param ppImageBuffers    Image pointer array where the captured images
 *                          need to be copied into
 */
virtual bool capture(void** ppImageBuffers) = 0;

The capture method needs to be overwritten in order to fill the image buffers. Thereby, the locking of the write mutex needs to be performed by the user again:

bool ExampleImageProvider::capture(void** ppImageBuffers)
{
    {
        armarx::SharedMemoryScopedWriteLockPtr lock = getScopedWriteLock();
 
        ImageFormatInfo imageFormat = getImageFormat();
        int imageSize = imageFormat.dimension.width * imageFormat.dimension.height * imageFormat.bytesPerPixel;
        memcpy(ppImageBuffers[0], images[0]->pixels, imageSize);
        memcpy(ppImageBuffers[1], images[1]->pixels, imageSize);
    }
 
    return true;
}

Capturing image providers have two different methods of invocing the capture method:

• eCaptureSynchronization: the capture methods itself handles the timing and synchronization
• eFpsSynchronization: the framework will call the capture method with the fps rate as specified in the CapturingImageProvider::startCapture(float fps)

The synchronization type needs to be specified in the onInit method using:

void CapturingImageProvider::setImageSyncMode(ImageSyncMode imageSyncMode)

Stereo image handling

A special case of image providers are stereo cameras. VisionX has special support for this type by means of the StereoCalibrationProviderInterface which allows attaching stereo calibration information to the interface of the ManagedIceObject:

#include <VisionX/core/CapturingImageProvider.h>
#include <VisionX/interface/components/Calibration.h>
 
namespace visionx
{
    class ExampleStereoImageProvider :
        virtual public visionx::CapturingImageProvider,
        virtual public StereoCalibrationProviderInterface
    {
    public:
        /**
         * Part of visionx::StereoCalibrationProvider interface
         *
         * Returns the VisionX StereoCalibration, especially vis ICE
         *
         * @return visionx::StereoCalibration
         */
        visionx::StereoCalibration getStereoCalibration(
                const Ice::Current& c = ::Ice::Current())
        {
            return visionx::StereoCalibration();
        }
 
        /**
         * Part of visionx::StereoCalibrationProvider interface
         *
         * Returns whether images are undistorted
         *
         * @return bool
         */
        bool getImagesAreUndistorted(
                const Ice::Current& c = ::Ice::Current())
        {
            return false;
        }
    };
}