North Plugins

North plugins are used in North tasks and microservices to extract data buffered in FogLAMP and send it Northbound, i.e. to a server or a service in the Cloud or in an Enterprise data center. We currently have two North plugins, one to send data to an OSIsoft PI Server and one to the OSIsoft Cloud Service.

The OMF Plugin

The OMF Plugin is used by a North task to send data to an OSIsoft PI server via a PI Connector Relay OMF. OMF stands for OSIsoft Message Format, it is the JSON format defined by OSIsoft to send IoT data to a PI server via a Connector Relay server.

The plugin is designed to send two streams of data:

  • The data collected by South microservices and buffered into FogLAMP
  • The statistics generated by FogLAMP

The streams are managed by two different North tasks using the same plugin, but with a different configuration. The two tasks are registered in the list of scheduled jobs and they can be identified using the schedule API call:

$ curl -sX GET   http://locahost:8081/foglamp/schedule
{
  "schedules": [
    { "day": null,
      "time": 0,
      "name": "OMF to PI north",
      "exclusive": true,
      "processName": "North Readings to PI",
      "enabled": false,
      "type": "INTERVAL",
      "repeat": 30,
      "id": "2b614d26-760f-11e7-b5a5-be2e44b06b34" },
    { "day": null,
      "time": 0,
      "name": "Stats OMF to PI north",
      "exclusive": true,
      "processName": "North Statistics to PI",
      "enabled": false,
      "type": "INTERVAL",
      "repeat": 30,
      "id": "1d7c327e-7dae-11e7-bb31-be2e44b06b34" },
  ...
  ]
}

The output of API call above shows three interesting tasks: the two tasks associated to the OMF plugin, the one to send data (OMF to PI north) and the one to send statistics (North Statistics to PI).

The two scheduled tasks are associated to two configuration items that can be retrieved using the category API call. The the items are named SEND_PR_1 and SEND_PR_2.

$ curl -sX GET   http://localhost:8081/foglamp/category/SEND_PR_1
{ "plugin": { "type": "string",
              "default": "omf",
              "value": "omf",
              "description": "Python module name of the plugin to load" }
}
$ curl -sX GET   http://localhost:8081/foglamp/category/SEND_PR_2
{ "plugin": { "type": "string",
              "default": "omf",
              "value": "omf",
              "description": "Python module name of the plugin to load" }
}
$

In order to activate the tasks, you must change their status. First you must collect their id (from the GET method of the schedule API call), then you must use the IDs with the PUT method of the same call:

$ curl -sX PUT http://vbox-dev:8081/foglamp/schedule/2b614d26-760f-11e7-b5a5-be2e44b06b34 -d '{ "enabled" : true}'
{ "schedule": { "day": null,
                "time": 0,
                "name": "OMF to PI north",
                "exclusive": true,
                "processName": "North Readings to PI",
                "enabled": true,
                "type": "INTERVAL",
                "repeat": 30.0,
                "id": "2b614d26-760f-11e7-b5a5-be2e44b06b34" }
}
$ curl -sX PUT http://vbox-dev:8081/foglamp/schedule/1d7c327e-7dae-11e7-bb31-be2e44b06b34 -d '{ "enabled" : true}'
{ "schedule": { "day": null,
                "time": 0,
                "name": "Stats OMF to PI north",
                "exclusive": true,
                "processName": "North Statistics to PI",
                "enabled": true,
                "type": "INTERVAL",
                "repeat": 30.0,
                "id": "1d7c327e-7dae-11e7-bb31-be2e44b06b34" }
}
$

At this point, the configuration has been enriched with default values of the tasks:

$ curl -sX GET http://vbox-dev:8081/foglamp/category/SEND_PR_1
{ "filterRule": {
    "type": "string",
    "default": ".[]",
    "description": "JQ formatted filter to apply (applicable if applyFilter is True)",
    "value": ".[]" },
  "plugin": { "type": "string",
    "default": "omf",
    "description": "OMF North Plugin",
    "value": "omf" },
  ...
}
$
$ curl -sX GET http://vbox-dev:8081/foglamp/category/SEND_PR_2
{ "URL": {
    "type": "string",
    "default": "https://pi-server:5460/ingress/messages",
    "value": "https://pi-server:5460/ingress/messages",
    "description": "The URL of the PI Connector to send data to" },
  "filterRule": {
    "type": "string",
    "default": ".[]",
    "value": ".[]",
    "description": "JQ formatted filter to apply (applicable if applyFilter is True)" },
  ...
$

OMF Plugin Configuration

The following table presents the list of configuration options available for the task that sends data to OMF (category SEND_PR_1):

Item Type Default Description
applyFilter boolean False Whether to apply filter before processing the data
blockSize integer 500 The size of a block of readings
to send in each transmission.
duration integer 60 How long the sending process should run
(in seconds) before stopping
enable boolean True A switch that can be used to enable or disable
execution of the sending process.
filterRule string .[] JQ formatted filter to apply
(applicable if applyFilter is True)
north string omf The name of the north to use to translate the
readings into the output format and send them
OMFHttpTimeout interger 10 Timeout in seconds for the HTTP operations
with the OMF PI Connector Relay
OMFMaxRetry interger 3 Max number of retries for the communication
with the OMF PI Connector Relay
OMFRetrySleepTime interger 1 Seconds between each retry for the communication
with the OMF PI Connector Relay,
NOTE : the time is doubled at each attempt.
plugin string omf OMF North Plugin name
producerToken string omf_north_0001 The producer token that represents this FogLAMP stream
sleepInterval integer 5 A period of time, expressed in seconds, to wait
between attempts to send readings when there are
no readings to be sent.
source string readings Defines the source of the data to be sent
the stream, this may be one of either
readings, statistics or audit.
StaticData JSON { "Location" : "Palo Alto",
"Company"  : "Dianomic" }
Static data to include in each sensor reading
sent to OMF.
stream_id integer 1 Stream ID
URL string https://pi-server:5460/ingress/messages The URL of the PI Connector to send data to

The following table presents the list of configuration options available for the task that sends statistics to OMF (category SEND_PR_2):

Item Type Default Description
applyFilter boolean False Whether to apply filter before processing the data
blockSize integer 500 The size of a block of readings
to send in each transmission.
duration integer 60 How long the sending process should run
(in seconds) before stopping
enable boolean True A switch that can be used to enable or disable
execution of the sending process.
filterRule string .[] JQ formatted filter to apply
(applicable if applyFilter is True)
north string omf The name of the north to use to translate the
readings into the output format and send them
OMFHttpTimeout interger 10 Timeout in seconds for the HTTP operations
with the OMF PI Connector Relay
OMFMaxRetry interger 3 Max number of retries for the communication
with the OMF PI Connector Relay
OMFRetrySleepTime interger 1 Seconds between each retry for the communication
with the OMF PI Connector Relay,
NOTE : the time is doubled at each attempt.
plugin string omf OMF North Plugin name
producerToken string omf_north_0001 The producer token that represents this FogLAMP stream
sleepInterval integer 5 A period of time, expressed in seconds, to wait
between attempts to send readings when there are
no readings to be sent.
source string readings Defines the source of the data to be sent
the stream, this may be one of either
readings, statistics or audit.
StaticData JSON { "Location" : "Palo Alto",
"Company"  : "Dianomic" }
Static data to include in each sensor reading
sent to OMF.
stream_id integer 2 Stream ID
URL string https://pi-server:5460/ingress/messages The URL of the PI Connector to send data to

The last parameter to review is the OMF Type. The call is the GET method foglamp/category/OMF_TYPES, which returns an integer value that identifies the measurement type:

$ curl -sX GET http://vbox-dev:8081/foglamp/category/OMF_TYPES
{
  "type-id": {
    "description": "Identify sensor and measurement types",
    "type": "integer",
    "default": "0001",
    "value": "0001"
  }
}
$

If you change the value, you can easily identify the set of data sent to and then stored into PI.

Changing the OMF Plugin Configuration

Before you send data to the PI server, it is likey that you need to apply more changes to the configuration. The most important items to change are:

  • URL : the URL to the PI Connector Relay OMF. It is usually composed by the name or address of the Windows server where the Connector Relay service is running, the port associated to the service and the ingress/messages API call. The communication is via HTTPS protocol.
  • producerToken : the token provided by the Data Collection Manager when the PI administrator sets the use of FogLAMP.
  • type-id : the measurement type for the stream of data.
  • source : this parameter should be set to readings (default) when the plugin is used to send data collected by South microservices, and to statistics when the plugin is used to send FogLAMP statistics to the PI system.

An example of the changes to apply to the plugins to send data to the PI system is available here here.

Data in the PI System

Once the North plugins have been set properly, you should expect to see data automatically sent and stored in the PI Server. More specifically, the process of the plugin is the following:

  • Assets buffered in FogLAMP are stored as elements in the PI System. - PI Asset Framework is automatically update with the new assets. - JSON objects captured as part of the reading in FogLAMP become attributes in the PI Data Archive
  • The Producer Token is used to authenticate and create the hierarchy of elements in the PI Asset Framework
  • The configuration object named as Static Data is added as a set of attributes in the PI Data Archive
System Object Value
FogLAMP Producer Token readings_001
OMF Type 0001
Static Data { “Company” : “Dianomic”, “Location” : “Palo Alto” }
Asset fogbench/accelerometer
Reading [{“reading”:{“y”:1,”z”:1,”x”:-1}, “timestamp”:”2018-05-14 19:27:06.788}]
PI System Element Template [OMF.readings_001 Connector.0001_fogbench/accelerometer_typename_sensor]
Attribute Template Company Configuration Item, Excluded, String
Location Configuration Item, Excluded, String
x Excluded, Int64
y Excluded, Int64
z Excluded, Int64
Element foglamp > readings_001 > fogbench/accelerometer
Attributes Name Value Timestamp
Company Dianomic 1970-01-01 00:00:00
Location Palo Alto 1970-01-01 00:00:00
x -1 2018-05-14 19:27:06.788
y 1 2018-05-14 19:27:06.788
z 1 2018-05-14 19:27:06.788