Developing INTER-MW Bridges for IoT Platforms

The intermw_bridge_example repository provides a sample implementation of a bridge called Example bridge which can be used as a template for new bridge projects. Besides bridge implementation it contains an integration test for testing the bridge within the INTER-MW and platform emulator which emulates the Example platform during the integration test.

The project is implemented as a multi-module Maven project and consists of following modules:

  • example-bridge: implementation of the Example bridge
  • example-bridge-integration-test: integration test for the Example bridge for testing the bridge within INTER-MW
  • example-platform-emulator: emulator of the Example platform used in the integration test

Bridge Development

A bridge project requires following INTER-MW Maven dependency:

<dependency>
    <groupId>eu.interiot.intermw</groupId>
    <artifactId>mw.bridges.api</artifactId>
    <version>${intermw.version}</version>
</dependency>

Additional dependencies may be added to the POM file if needed. Bear in mind that these dependencies (JAR files) will have to be added to the INTER-MW runtime classpath and may cause dependency conflicts if not compatible with other INTER-MW dependencies.

The bridge implementation class (e.g. ExampleBridge) have to be annotated with a @Bridge annotation and provide target platform type using platformType attribute. The platform type must be an URI, e.g. http://example.inter-iot.eu/example-platform. The bridge implementation class must extend the AbstractBridge class and thereby implement the Bridge interface:

@Bridge(platformType = "http://example.inter-iot.eu/example-platform")
public class ExampleBridge extends AbstractBridge {

When INTER-MW is started, it scans the classpath for all bridge implementation classes (annotated with a @Bridge annotation) and registers them in the bridges registry to know which bridge class to use for specific platform type. When INTER-MW receives PLATFORM_REGISTER request, it determines appropriate bridge class according to the platform type and creates an instance of it for the specified platform by calling the bridge constructor, e.g.:

public ExampleBridge(BridgeConfiguration configuration, Platform platform)

The configuration parameter provides bridge configuration properties read from property file and the platform parameter provides details of the platform to register extracted from the PLATFORM_REGISTER message.

Bridge Configuration

Bridge configuration can be specified in three levels:

  • global level (common for all bridges)
  • bridge type level (common for all instances of the same bridge type)
  • bridge instance level (for specific bridge)

Global level bridge configuration properties are located in the INTER-MW configuration file intermw.properties. Properties have to start with the bridge. prefix, for example:

bridge.callback.url=http://localhost:8980

Bridge type level and bridge instance level properties have to be specified in a properties file named <bridge-class-name>.properties, for example ExampleBridge.properties. Bridge type level properties doesn't have any prefix, e.g.:

testproperty1=value1
testproperty2=value2

Bridge instance level properties are prefixed with an instance identificator, e.g.:

instanceA.testproperty1=instanceA-value1
instanceA.testproperty3=instanceA-value3

Bridge instance level properties override bridge type level properties. In this example the configuration object passed to the bridge constructor will contain following properties:

testproperty1=instanceA-value1
testproperty2=value2
testproperty3=instanceA-value3

The platforms property contains a list of all platform (bridge) instance identificators which are used as a prefix to specify bridge instance level properties, e.g.:

platforms=instanceA,instanceB

For each instance identificator the corresponding platform ID has to be specified using the id attribute:

instanceA.id=http://test.inter-iot.eu/test-platform1

An example of properties file for the Example bridge (ExampleBridge.properties):

# Configuration properties common for all bridge instances of type ExampleBridge (bridge type-level properties)
testproperty1=value1
testproperty2=value2
testproperty3=value3

# List of platform instance identificators
platforms=instanceA,instanceB

# Configuration properties for specific platform (bridge) instance (bridge instance-level properties)
# Bridge instance-level properties override bridge type-level properties.

# instanceA
instanceA.id=http://test.inter-iot.eu/test-platform1
instanceA.testproperty1=instanceA-value1
instanceA.testproperty2=instanceA-value2
instanceA.testproperty5=value5

# instanceB
instanceB.id=http://test.inter-iot.eu/test-platform2
instanceB.testproperty5=value5

Bridge Operations

A bridge must implement all abstract methods defined in the AbstractBridge abstract class:

Message registerPlatform(Message message) throws Exception;

Message unregisterPlatform(Message message) throws Exception;

Message subscribe(Message message) throws Exception;

Message unsubscribe(Message message) throws Exception;

Message query(Message message) throws Exception;

Message listDevices(Message message) throws Exception;

Message platformCreateDevices(Message message) throws Exception;

Message platformUpdateDevices(Message message) throws Exception;

Message platformDeleteDevices(Message message) throws Exception;

Message observe(Message message) throws Exception;

Message actuate(Message message) throws Exception;

Message error(Message message) throws Exception;

Message unrecognized(Message message) throws Exception;

For detailed description of message types and message format please see JSON-LD Messaging documentation.

Register platform

Registers specified platform in the INTER-MW and creates bridge instance for the platform.

Unregister platform

Unregisters specified platform.

Subscribe

Subscribes client to observations provided by the platform for the specified device.

The subscribe method must implement listener that accepts observation messages sent from the platform to INTER-MW for corresponding subscription. The listener is implemented using Spark framework. It listens on port defined by the configuration property bridge.callback.url (8980 by default) and path matching the conversationId parameter. In the handler (callback) method the observation data is converted to an observation message of type Message and sent upstream through INTER-MW.

Spark.post(conversationId, (request, response) -> {
  // handle the observation
});

Unsubscribe

Cancels specified subscription created by the Subscribe.

Query

Makes a query about a status and last observation made by a specified device.

List Devices

Makes a query to a platform to get all devices managed by it, that it deems discoverable

Platform Create Device

An instruction for a platform to start managing (i.e. create) a new device.

Platform Update Device

An instruction for a platform to update information about a device it is managing.

Platform Delete Device

An instruction for a platform to stop managing (i.e. remove) a device.

Observe

Pushes given observation message from INTER-MW to platform (bridge acting as a publisher for platform)

Actuate

Same as Observe, except the message payload contains actuation instructions.

Error

Sends information about any error that occurred inside bridge or inside INTER-MW.

Unrecognized

Custom message type which was not recognized by INTER-MW.

Device discovery

With device discovery the platform bridge can communicate all its discoverable devices with INTER-MW.

Messages in discovery should adhere to the GOIoTP ontology

There are four message types used in device discovery:

Message type Direction Payload Description
LIST_DEVICES INTER-MW -> Bridge None Will trigger device discovery
DEVICE_REGISTRY_INITIALIZE Bridge -> INTER-MW List of IoTDevices Contains a list of all devices deemed discoverable by the bridge
DEVICE_ADD_OR_UPDATE Bridge -> INTER-MW IoTDevice Contains update data of one IoTDevice
DEVICE_REMOVE Bridge -> INTER-MW IoTDevice Id Contains ID of a removed IoTDevice

INTER-MW recognizes IoTDevices as devices, actuators or sensors. Below is a table representing mandatory fields for each device type.

IoTDevice

Field name Type Description Example
IoTDevice
@id URI Device identifier, each must be unique and fully qualified "@id" : "InterIoT:wso2port/emission/2"
@type List A list of types describing a device. "@type" : [ "InterIoT:LogVPmod#EmissionCabin", "sosa:Sensor", "iiot:IoTDevice" ]
iiot:hasName String Human readable name "iiot:hasName" : "CABINA PUERTO 2"
sosa:isHostedBy Object/String Id of the current device host "sosa:isHostedBy" : { "@id" : "http://www.inter-iot.eu/wso2port" }
Actuator
sosa:actsOnProperty anonymous node ID or URI Property the actuator acts on "sosa:actsOnProperty" : { "@id" : "_:b29" } OR "sosa:actsOnProperty" : { "@id" : "InterIoT:actuatableProperty1" }
Sensor
sosa:observes anonymous node ID or URI Property the sensor is observing "sosa:observes" : { "@id" : "_:b29" } OR "sosa:observes" : { "@id" : "InterIoT:observableProperty1" }

Integration Test

The intermw_bridge_example project provides an integration test for testing the bridge integrated into the INTER-MW and a platform emulator which emulates the Example Platform during the integration test. The integration test is located in the example-bridge-integration-test module of the intermw_bridge_example project and the platform emulator is located in the example-platform-emulator module.

The integration test is a JUnit test class which uses the INTER-MW Java API to execute a standard scenario (register platform, create devices, subscribe to specific devices, etc.) and checks if operations have executed successfully. In the integration test project the INTER-MW dependencies and Example Bridge are put on the same classpath which enables INTER-MW to find and load the bridge.

Prerequisites

The integration test has the following prerequisites:

  • RabbitMQ
  • Parliament triple store
  • Inter-Platform Semantic Mediator (IPSM)
  • Example platform emulator

RabbitMQ

RabbitMQ can be started with the following Docker command:

docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin rabbitmq:3.7-management-alpine

Parliament Triple Store

Parliament can be started with the following Docker command:

docker run -d --name parliament -p 8089:8089 daxid/parliament-triplestore

IPSM

IPSM installation guide is available at ipsm-deployment repository.

The integration test takes care to upload to IPSM alignments required by the Example bridge:

  • Example_Downstream_Alignment version 1.0
  • Example_Upstream_Alignment version 1.0

Alignments are uploaded from following files, respectively:

  • example-downstream-alignment.rdf in example-bridge-integration-test/src/test/resources/alignments/ directory
  • example-upstream-alignment.rdf in example-bridge-integration-test/src/test/resources/alignments/ directory

IPSM channels are managed automatically by INTER-MW. They are created when registering a new platform, updated if necessary when the platform is updated and removed when the platform is unregistered.

Example Platform Emulator

The emulator can be started by running the ExamplePlatformEmulator class from example-platform-emulator module. You can run it from your IDE or by executing following commands:

cd intermw_bridge_example/example-platform-emulator
mvn package
java -jar target/example-platform-emulator-<version>.jar

Configuration

Configuration files are located in the example-bridge-integration-test/src/test/resources directory:

  • intermw.properties: main INTER-MW configuration file
  • kafka-client.keystore.jks and kafka-client.truststore.jks: client's keystore and truststore for publishing messages to the IPSM's Kafka
  • log4j2.xml: logging configuration
  • ExampleBridge.properties: example bridge configuration file

Running the Integration Test

Run the ExampleBridgeIntegrationTest from your IDE or with following Maven command from the intermw_bridge_example (project root) directory:

mvn verify

Note: only one instance of INTER-MW can run at a time on the same machine using the same RabbitMQ, Parliament and IPSM instance. If you run the integration test when another instance of INTER-MW is running, the integration test will fail.

Bridge Installation

Bridge installation to an InterMW instance consists of the following steps:

  • copying bridge JAR file together with its dependencies to the INTER-MW webapp library inside the INTER-MW Docker container
  • copying bridge configuration file (if any) to the INTER-MW configuration directory in
  • restarting INTER-MW

Prerequisites:

  • INTER-MW deployment which version must match the version of mw.bridges.api dependency used in the bridge POM file:
<dependency>
    <groupId>eu.interiot.intermw</groupId>
    <artifactId>mw.bridges.api</artifactId>
    <version>${intermw.version}</version>
</dependency>

The intermw.version variable from dependency tag above is defined in the root POM file of intermw_bridge_example project, for example:

<intermw.version>2.3.0</intermw.version>

Version of the INTER-MW Docker deployment can be checked by running docker ps command and looking for intermw image tag:

docker.inter-iot.eu/intermw:v2.3.0

Installing Bridge JARs

Go into the bridge module:

cd intermw_bridge_example/example-bridge

Build and package the bridge by running following Maven command:

mvn clean package

Copy the bridge JAR file (example-bridge-<version>.jar in this case) created in the previous step to lib directory of INTER-MW webapp inside the INTER-MW Docker container using the following command:

docker cp <path-to-bridge-jar-file> <intermw-container>:/usr/local/tomcat/webapps/ROOT/WEB-INF/lib

In case of Example bridge, run the following command:

docker cp target/example-bridge-<version>.jar <intermw-container>:/usr/local/tomcat/webapps/ROOT/WEB-INF/lib

To check whether bridge JAR file is contained within the INTER-MW library, run the following command:

docker exec <intermw-container> ls -l /usr/local/tomcat/webapps/ROOT/WEB-INF/lib | grep example-bridge

In addition, you also have to copy into the INTER-MW library (i.e. lib directory) all runtime dependencies that are required by the bridge project and don't already exist there. Make sure that dependencies added to INTER-MW library don't cause dependency conflict. Note that the mw.bridges.api dependency together with its sub-dependencies are already included in the INTER-MW library. The syntactic-translators dependency is not included.

You can get a list of all dependencies available in INTER-MW library by listing lib directory content:

docker exec <intermw-container> ls -l /usr/local/tomcat/webapps/ROOT/WEB-INF/lib

To display dependency tree for the bridge project (runtime dependencies), run the following command:

mvn dependency:tree -Dscope=runtime

In case of intermw_bridge_example project, no dependencies were added to POM file except the default mw.bridges.api (dependencies with scope test shall be ignored), so nothing is needed to be copied.

Installing Bridge Configuration File

In case the bridge requires a configuration file (e.g. ExampleBridge.properties for Example bridge), it has to be copied to the INTER-MW configuration directory:

docker cp src/main/resources/<bridge-config-file> <intermw-container>:/etc/inter-iot/intermw

In case of Example bridge, run the following command from example-bridge directory:

docker cp src/main/resources/ExampleBridge.properties <intermw-container>:/etc/inter-iot/intermw

Note: INTER-MW configuration directory is mapped to the intermw_config Docker volume to persist after the container is deleted.

To list the configuration directory content, run:


Restarting INTER-MW

Finally, INTER-MW has to be restarted to detect new bridge. Run following command inside the INTER-MW Docker Compose deployment directory from where the INTER-MW was deployed:

docker-compose restart intermw

To make sure new bridge was detected and registered successfully, search for the string Registering bridges... in INTER-MW log file:

docker-compose logs intermw | grep -A 5 "Registering bridges"

In case of ExampleBridge, the following log statements can be found in the logs:

DEBUG eu.interiot.intermw.bridge.Context.registerBridges(Context.java:208) - Registering bridges...
DEBUG eu.interiot.intermw.bridge.Context.registerBridges(Context.java:210) - Scanning package eu.interiot...
DEBUG eu.interiot.intermw.bridge.Context.registerBridges(Context.java:214) - Following bridges have been found: [class eu.interiot.intermw.bridge.example.ExampleBridge]
DEBUG eu.interiot.intermw.bridge.Context.registerBridges(Context.java:220) - Bridge eu.interiot.intermw.bridge.example.ExampleBridge for platform type http://example.inter-iot.eu/example-platform has been registered.
DEBUG eu.interiot.intermw.bridge.Context.registerBridges(Context.java:224) - Bridge registration has finished successfully.

The term bridge registration means that the bridge class has been related to the corresponding platform type. Bridge is instantiated (an instance of Bridge class is created) when an actual platform is registered for specific platform type. See chapter Registering platform.

Configuring IPSM

IPSM alignments for the bridge can be uploaded using IPSM Swagger UI available at http://localhost:8888/swagger/ (assuming local IPSM deployment on default port).

IPSM channels are managed automatically by INTER-MW. They are created when registering a new platform, updated if necessary when the platform is updated and removed when the platform is unregistered.

Uploading Alignments

Expand Alignments section and use POST /alignments operation to upload new alignment by uploading alignment RDF/XML document.

Example bridge requires following two alignments:

  • Example_Downstream_Alignment
  • Example_Upstream_Alignment

These alignments are available in following files, respectively:

  • example-bridge-integration-test/src/test/resources/alignments/example-downstream-alignment.rdf
  • example-bridge-integration-test/src/test/resources/alignments/example-upstream-alignment.rdf

Creating Channels

If using older version of INTER-MW (prior to v2.2.3), IPSM channels have to be created manually. Go to IPSM Swagger UI, expand Channels section and use POST /channels operation to create downstream and upstream channels for the platform. Use following JSON data as input:

Downstream Channel

For IPSM version 0.4.* use the following format:

{
  "source": "mw-ipsm-downstream-<platform-id>",
  "inpAlignmentId": 0,
  "outAlignmentId": <alignment-id>,
  "sink": "ipsm-mw-downstream-<platform-id>",
  "parallelism": 1
}

For IPSM version 0.5.* use the following format:

{
  "source": "mw-ipsm-downstream-<platform-id>",
  "inpAlignmentName": "",
  "inpAlignmentVersion": "",
  "outAlignmentName": "<name>",
  "outAlignmentVersion": "<version>",
  "sink": "ipsm-mw-downstream-<platform-id>",
  "parallelism": 1
}
Upstream channel

For IPSM version 0.4.* use the following format:

{
  "source": "mw-ipsm-upstream-<platform-id>",
  "inpAlignmentId": <alignment-id>,
  "outAlignmentId": 0,
  "sink": "ipsm-mw-upstream-<platform-id>",
  "parallelism": 1
}

For IPSM version 0.5 onwards use the following format:

{
  "source": "mw-ipsm-upstream-<platform-id>",
  "inpAlignmentName": "<name>",
  "inpAlignmentVersion": "<version>",
  "outAlignmentName": "",
  "outAlignmentVersion": "",
  "sink": "ipsm-mw-upstream-<platform-id>",
  "parallelism": 1
}

where <platform-id> is encoded platform ID. One or more consecutive special characters must be replaced with a single underscore character. For example, http://example.inter-iot.eu/platforms/1 platform ID is encoded to http_example.inter-iot.eu_platforms_1.

Example bridge

For the Example bridge use following channels definition:

IPSM version 0.4.*:

{
  "source": "mw-ipsm-downstream-http_example.inter-iot.eu_platforms_1",
  "inpAlignmentId": 0,
  "outAlignmentId": <alignment-id>,
  "sink": "ipsm-mw-downstream-http_example.inter-iot.eu_platforms_1",
  "parallelism": 1
}

{
  "source": "mw-ipsm-upstream-http_example.inter-iot.eu_platforms_1",
  "inpAlignmentId": <alignment-id>,
  "outAlignmentId": 0,
  "sink": "ipsm-mw-upstream-http_example.inter-iot.eu_platforms_1",
  "parallelism": 1
}

IPSM version 0.5 onwards:

{
  "source": "mw-ipsm-downstream-http_example.inter-iot.eu_platforms_1",
  "sink": "ipsm-mw-downstream-http_example.inter-iot.eu_platforms_1",
  "parallelism": 1,
  "inpAlignmentName": "",
  "inpAlignmentVersion":"",
  "outAlignmentName":"Example_Downstream_Alignment",
  "outAlignmentVersion":"1.0"
}

{
  "source": "mw-ipsm-upstream-http_example.inter-iot.eu_platforms_1",
  "sink": "ipsm-mw-upstream-http_example.inter-iot.eu_platforms_1",
  "parallelism": 1,
  "inpAlignmentName": "Example_Upstream_Alignment",
  "inpAlignmentVersion":"1.0",
  "outAlignmentName":"",
  "outAlignmentVersion":""
}

Bridge Testing

After bridge has been installed to INTER-MW, its operation can be tested by performing the testing scenario provided below. In this testing scenario first a new client is registered, then an Example platform is registered and the client is subscribed to some example devices. If everything is set up correctly, observation messages start arriving to the client queue.

Prerequisites

  • Inter-Platform Semantic Mediator (IPSM) must be running
  • Example platform emulator must be running

Example Platform Emulator

The emulator can be started by running the ExamplePlatformEmulator class from example-platform-emulator module. You can run it from your IDE or by executing following commands:

cd intermw_bridge_example/example-platform-emulator
mvn package
java -jar target/example-platform-emulator-<version>.jar <options>

Example platform emulator supports following options:

  • --delay <N>: delay in seconds between observation messages sent by the emulator
  • --database-file <path>: path to H2 database file, must be an absolute path. If the file doesn't exist yet, it will be created. If specified, emulator will use specified file (as a H2 database file) for persisting its state, otherwise in-memory non-persistent database will be used.
  • --help: show usage
java -jar target/example-platform-emulator-<version>.jar --delay 60 --database-file ~/intermw/example-platform-emulator/db

Testing Scenario

INTER-MW API calls can be performed most easily by using Swagger UI of INTER-MW REST API accessible at http://localhost:8080/swagger/#/. Any other REST client tool can be also used. Swagger UI provides a corresponding curl command for each operation for executing it from the command line.

Registering client

A new client can be registered using POST /mw2mw/clients operation. There are two supported mechanisms (response delivery methods) a client can use for retrieving response messages and observations:

  • client pull
  • server push

POST data for registering a new client depends on the selected response delivery method:

Client Pull response delivery

POST data:

{
  "clientId": "myclient",
  "responseFormat": "JSON_LD",
  "responseDelivery": "CLIENT_PULL",
  "receivingCapacity": 10
}

Response messages and observations can be in this case retrieved from INTER-MW using the POST /mw2mw/responses operation.

Server Push response delivery

POST data:

{
  "clientId": "myclient",
  "responseFormat": "JSON_LD",
  "responseDelivery": "SERVER_PUSH",
  "callbackUrl": "http://172.17.0.1:7070",
  "receivingCapacity": 10
}

INTER-MW will be sending response messages and observations to the endpoint specified by the callbackUrl property. Maximum number of messages sent at a time is specified by the receivingCapacity property.

To accept messages, start some sort of a HTTP listener listening on port specified by the callbackUrl property. You can use a Python script http-listener.py available in the example-bridge/src/test/scripts directory:

cd example-bridge/src/test/scripts
./http-listener.py <port>

Registering platform

Register Example platform with ID http://example.inter-iot.eu/platforms/1 using POST /mw2mw/platforms operation:

POST data:

{
  "platformId": "http://example.inter-iot.eu/platforms/1",
  "type": "http://example.inter-iot.eu/example-platform",
  "baseEndpoint": "http://172.17.0.1:4568",
  "location": "http://test.inter-iot.eu/TestLocation",
  "name": "Example platform 1",
  "downstreamInputAlignmentName": "",
  "downstreamInputAlignmentVersion": "",
  "downstreamOutputAlignmentName": "Example_Downstream_Alignment",
  "downstreamOutputAlignmentVersion": "1.0",
  "upstreamInputAlignmentName": "Example_Upstream_Alignment",
  "upstreamInputAlignmentVersion": "1.0",
  "upstreamOutputAlignmentName": "",
  "upstreamOutputAlignmentVersion": ""
}

Note: baseEndpoint is platform URL which must be accessible from intermw Docker container, e.g. http://172.17.0.1:4568 in case of using Example platform emulator. The 172.17.0.1 is host machine IP address in Docker network on Linux host machine.

The REST API operation returns 202 (Accepted) response code. To make sure the process has executed successfully, check response messages according to selected response delivery method.

Creating Devices

Add some devices to Example platform using POST /mw2mw/devices operation:

POST data:

{
  "devices": [
    {
      "deviceId": "http://example.inter-iot.eu/devices/1",
      "hostedBy": "http://example.inter-iot.eu/platforms/1",
      "location": "http://test.inter-iot.eu/TestLocation",
      "name": "Example Device 1"
    },
    {
      "deviceId": "http://example.inter-iot.eu/devices/2",
      "hostedBy": "http://example.inter-iot.eu/platforms/1",
      "location": "http://test.inter-iot.eu/TestLocation",
      "name": "Example Device 2"
    },
    {
      "deviceId": "http://example.inter-iot.eu/devices/3",
      "hostedBy": "http://example.inter-iot.eu/platforms/1",
      "location": "http://test.inter-iot.eu/TestLocation",
      "name": "Example Device 3"
    }
  ]
}

The REST API operation returns 202 (Accepted) response code. To make sure the process has executed successfully, check response messages according to selected response delivery method.

Subscribing to Devices

Subscribe client myclient to devices using POST /mw2mw/subscriptions operation:

POST data:

{
  "deviceIds": [
    "http://example.inter-iot.eu/devices/1",
    "http://example.inter-iot.eu/devices/2",
    "http://example.inter-iot.eu/devices/3"
  ]
}

The REST API operation returns 202 (Accepted) response code. To make sure the process has executed successfully, check response messages according to selected response delivery method.

Retrieving observations

If everything is working fine, observation message will start arriving. Observation messages can be retrieved using selected response delivery mechanism.

Example platform emulator is sending observations in a batch for all three subscribed devices at a time with a few seconds delay between batches.

Troubleshooting

If anything went wrong, you can check INTER-MW logs for any ERROR logs. Logs can be displayed using the following command:

docker-compose logs intermw

To display last N lines of logs, use the option --tail:

docker-compose logs --tail <N> intermw

To follow the log output, use option -f:

docker-compose logs -f intermw

Additional documentation

There is some additional documentation, which is work in progress, that can help developing bridges: