South Plugins

South plugins are used to communicate with sensors and actuators, there are two modes of plugin operation; asyncio and polled.

Polled Mode

Polled mode is the simplest form of South plugin that can be written, a poll routine is called at an interval defined in the plugin configuration. The South service determines the type of the plugin by examining at the mode property in the information the plugin returns from the plugin_info call.

Plugin Poll

The plugin poll method is called periodically to collect the readings from a poll mode sensor. As with all other calls the argument passed to the method is the handle returned by the initialization call, the return of the method should be the JSON payload of the readings to return.

The JSON payload returned, as a Python dictionary, should contain the properties; asset, timestamp, key and readings.

Property Description
asset The asset key of the sensor device that is being read
timestamp A timestamp for the reading data
key A UUID which is the unique key of this reading
readings The reading data itself as a JSON object

It is important that the poll method does not block as this will prevent the proper operation of the South microservice. Using the example of our simple DHT11 device attached to a GPIO pin, the poll routine could be:

def plugin_poll(handle):
    """ Extracts data from the sensor and returns it in a JSON document as a Python dict.

    Available for poll mode only.

    Args:
        handle: handle returned by the plugin initialisation call
    Returns:
        returns a sensor reading in a JSON document, as a Python dict, if it is available
        None - If no reading is available
    Raises:
        DataRetrievalError
    """

    try:
        humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle)
        if humidity is not None and temperature is not None:
            time_stamp = str(datetime.now(tz=timezone.utc))
            readings =  { 'temperature': temperature , 'humidity' : humidity }
            wrapper = {
                    'asset':     'dht11',
                    'timestamp': time_stamp,
                     'key':       str(uuid.uuid4()),
                    'readings':  readings
            }
            return wrapper
        else:
            return None

    except Exception as ex:
        raise exceptions.DataRetrievalError(ex)

    return None

Async IO Mode

In asyncio mode the plugin inserts itself into the event processing loop of the South server itself. This is a more complex mechanism and is intended for plugins that need to block or listen for incoming data via a network.

Plugin Start

The plugin_start method, as with other plugin calls, is called with the plugin handle data that was returned from the plugin_init call. The plugin_start call will only be called once for a plugin, it is the responsibility of plugin_start to install the plugin code into the python event handling system for asyncIO. Assuming an example whereby the interface to a sensor is via HTTP and the sensor will make HTTP POST calls to our plugin in order to send data into FogLAMP, a plugin_start for this scenario would create a web application endpoint for reception of the POST command.

loop = asyncio.get_event_loop()

app = web.Application( middlewares=[middleware.error_middleware] )
app.router.add_route( 'POST', '/', SensorPhoneIngest.render_post )
handler = app.make_handler()
coro = loop.create_server( handler, host, port )
server = asyncio.ensure_future( coro )

This code first gets the event loop for this Python execution, it then creates the web application and adds a route for the POST request. In this case it is calling the render_post method of the object SensorPhone. It then goes on to create the handler and install the web server instance into the event system.

Async Handler

The async handler is defined for incoming message has the responsibility of taking the sensor data and ingesting that into FogLAMP. Unlike the poll mechanism, this is done from within the handler rather than by passing the data back to the South service itself. A convenient method exists for ingesting readings, Ingest.add_readings. This call is passed an asset, timestamp, key and readings document for the asset and will do everything else required to make sure the readings are stored in the FogLAMP buffer.
In the case of our HTTP based example above, the code would create the items needed to generate the arguments to the Ingest.add_readings call, by creating data items and retrieving them from the payload sent by the sensor.

try:
    if not Ingest.is_available():
        increment_discarded_counter = True
        message = {'busy': True}
    else:
        payload = await request.json()

        asset = 'SensorPhone'
        timestamp = str(datetime.now(tz=timezone.utc))
        messages = payload.get('messages')

        if not isinstance(messages, list):
                raise ValueError('messages must be a list')

        for readings in messages:
             key = str(uuid.uuid4())
await Ingest.add_readings(asset=asset, timestamp=timestamp, key=key, readings=readings)

except ...

It would then respond to the HTTP request and return. Since the handler is embedded in the event loop this will happen in the context of a coroutine and would happen each time a new POST request is received.

message['status'] = code
return web.json_response(message)

A South Plugin Example: the DHT11 Sensor

Let’s try to put all the information together and write a plugin. We can continue to use the example of an inexpensive sensor, the DHT11, used to measure temperature and humidity, directly wired to a Raspberry PI. This plugin is also available in the FogLAMP project on GitHub, in the contrib folder.

First, here is a set of links where you can find more information regarding this sensor:

The Hardware

The DHT sensor is directly connected to a Raspberry PI 2 or 3. You may decide to buy a sensor and a resistor and solder them yourself, or you can buy a ready-made circuit that provides the correct output to wire to the Raspberry PI. This picture shows a DHT11 with resistor that you can buy online.

The sensor can be directly connected to the Raspberry PI GPIO (General Purpose Input/Output). An introduction to the GPIO and the pinset is available here. In our case, you must connect the sensor on these pins:

  • VCC is connected to PIN #2 (5v Power)
  • GND is connected to PIN #6 (Ground)
  • DATA is connected to PIN #7 (BCM 4 - GPCLK0)

This picture shows the sensor wired to the Raspberry PI and this is a zoom into the wires used.

The Software

For this plugin we use the ADAFruit Python Library (links to the GitHub repository are above). First, you must install the library (in future versions the library will be provided in a ready-made package):

$ git clone https://github.com/adafruit/Adafruit_Python_DHT.git
Cloning into 'Adafruit_Python_DHT'...
remote: Counting objects: 249, done.
remote: Total 249 (delta 0), reused 0 (delta 0), pack-reused 249
Receiving objects: 100% (249/249), 77.00 KiB | 0 bytes/s, done.
Resolving deltas: 100% (142/142), done.
$ cd Adafruit_Python_DHT
$ sudo apt-get install build-essential python-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
build-essential python-dev
...
$ sudo python3 setup.py install
running install
running bdist_egg
running egg_info
creating Adafruit_DHT.egg-info
...
$

The Plugin

This is the code for the plugin:

""" Plugin for a DHT11 temperature and humidity sensor attached directly
    to the GPIO pins of a Raspberry Pi

    This plugin uses the Adafruit DHT library, to install this perform
    the following steps:

        git clone https://github.com/adafruit/Adafruit_Python_DHT.git
        cd Adafruit_Python_DHT
        sudo apt-get install build-essential python-dev
        sudo python setup.py install

    To access the GPIO pins foglamp must be able to access /dev/gpiomem,
    the default access for this is owner and group read/write. Either
    FogLAMP must be added to the group or the permissions altered to
    allow FogLAMP access to the device.
    """

from datetime import datetime, timezone
import Adafruit_DHT
import uuid
import copy

from foglamp.common import logger
from foglamp.services.south import exceptions

__author__ = "Mark Riddoch"
__copyright__ = "Copyright (c) 2017 OSIsoft, LLC"
__license__ = "Apache 2.0"
__version__ = "${VERSION}"

_DEFAULT_CONFIG = {
    'plugin': {
        'description': 'Python module name of the plugin to load',
        'type':        'string',
        'default':     'dht11pi'
    },
    'pollInterval': {
        'description': 'The interval between poll calls to the device poll routine expressed in milliseconds.',
        'type':        'integer',
        'default':     '1000'
    },
    'gpiopin': {
        'description': 'The GPIO pin into which the DHT11 data pin is connected',
        'type':        'integer',
        'default':     '4'
    }

}

_LOGGER = logger.setup(__name__)
""" Setup the access to the logging system of FogLAMP """

def plugin_info():
    """ Returns information about the plugin.

    Args:
    Returns:
        dict: plugin information
    Raises:
    """

    return {
        'name':      'DHT11 GPIO',
        'version':   '1.0',
        'mode':      'poll',
        'type':      'south',
        'interface': '1.0',
        'config':    _DEFAULT_CONFIG
    }


def plugin_init(config):
    """ Initialise the plugin.

    Args:
        config: JSON configuration document for the device configuration category
    Returns:
        handle: JSON object to be used in future calls to the plugin
    Raises:
    """

    handle = config['gpiopin']['value']
    return handle


def plugin_poll(handle):
    """ Extracts data from the sensor and returns it in a JSON document as a Python dict.

    Available for poll mode only.

    Args:
        handle: handle returned by the plugin initialisation call
    Returns:
        returns a sensor reading in a JSON document, as a Python dict, if it is available
        None - If no reading is available
    Raises:
        DataRetrievalError
    """

    try:
        humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle)
        if humidity is not None and temperature is not None:
            time_stamp = str(datetime.now(tz=timezone.utc))
            readings =  { 'temperature': temperature , 'humidity' : humidity }
            wrapper = {
                    'asset':     'dht11',
                    'timestamp': time_stamp,
                    'key':       str(uuid.uuid4()),
                    'readings':  readings
            }
            return wrapper
        else:
            return None

    except Exception as ex:
        raise exceptions.DataRetrievalError(ex)

    return None


def plugin_reconfigure(handle, new_config):
    """ Reconfigures the plugin, it should be called when the configuration of the plugin is changed during the
        operation of the device service.
        The new configuration category should be passed.

    Args:
        handle: handle returned by the plugin initialisation call
        new_config: JSON object representing the new configuration category for the category
    Returns:
        new_handle: new handle to be used in the future calls
    Raises:
    """

    new_handle = new_config['gpiopin']['value']
    return new_handle


def plugin_shutdown(handle):
    """ Shutdowns the plugin doing required cleanup, to be called prior to the device service being shut down.

    Args:
        handle: handle returned by the plugin initialisation call
    Returns:
    Raises:
    """

The configuration

Since the plugin is still experimental, it works only in a build environment, the snap version will be available in the next release.

The configuration must be set manually in the FogLAMP metadata. in the repository, the file cmds.sql in the contrib/plugins/south/dht11pi folder must be executed with psql (or another PostgreSQL client) to add the configuration to the FogLAMP metadata.

Let’s see the SQL commands:

--- Create the South service instannce
INSERT INTO foglamp.scheduled_processes ( name, script )
     VALUES ( 'dht11pi', '["services/south"]');

--- Add the schedule to start the service at system startup
INSERT INTO foglamp.schedules ( id, schedule_name, process_name, schedule_type,schedule_interval, exclusive )
     VALUES ( '543a59ce-a9ca-11e7-abc4-cec278b6b11a', 'device', 'dht11pi', 1, '0:0', true );

--- Insert the config needed to load the plugin
INSERT INTO foglamp.configuration ( key, description, value )
     VALUES ( 'dht11pi', 'DHT11 on Raspberry Pi Configuration',
              '{"plugin" : { "type" : "string", "value" : "dht11pi", "default" : "dht11pi", "description" : "Plugin to load" } }' );

Building FogLAMP and Adding the Plugin

If you have not built FogLAMP yet, follow the steps described here. After the build, you can optionally install FogLAMP following these steps.

Once the Storage database has been setup, let’s update the configurarion to include the new plugin:

$ psql -d foglamp -f cmds.sql
INSERT 0 1
INSERT 0 1
INSERT 0 1
$

Now it is time to apply a workaround and include our new plugin.

  • If you intend to start and execute FogLAMP from the build folder: copy the structure of the contrib folder into the python folder:
$ cd ~/FogLAMP
$ cp -R contrib/plugins python/foglamp/.
$
  • If you have installed FogLAMP by executing sudo make install, copy the structure of the contrib folder into the installed python folder:
$ cd ~/FogLAMP
$ sudo cp -R contrib/plugins /usr/local/FogLAMP/python/foglamp/.
$

Note

If you have installed FogLAMP using an alternative DESTDIR, remember to add the path to the destination directory to the cp command.

Using the Plugin

Now you are ready to use the DHT11 plugin. If stop and restart FogLAMP if it is already running, or start it now.

  • Starting FogLAMP from the build folder:
$ cd ~/FogLAMP
$ export FOGLAMP_ROOT=$HOME/FogLAMP
$ scripts/foglamp start
Starting FogLAMP................
FogLAMP started.
$
  • Starting FogLAMP from the installed folder:
$ cd /usr/local/FogLAMP
$ bin/foglamp start
Starting FogLAMP................
FogLAMP started.
$

Let’s see what we have collected so far:

$ curl -s http://localhost:8081/foglamp/asset | jq
[
  {
    "count": 158,
    "asset_code": "dht11"
  }
]
$

Finally, let’s extract some values:

$ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq
[
  {
    "timestamp": "2017-12-30 14:41:39.672",
    "reading": {
      "temperature": 19,
      "humidity": 62
    }
  },
  {
    "timestamp": "2017-12-30 14:41:35.615",
    "reading": {
      "temperature": 19,
      "humidity": 63
    }
  },
  {
    "timestamp": "2017-12-30 14:41:34.087",
    "reading": {
      "temperature": 19,
      "humidity": 62
    }
  },
  {
    "timestamp": "2017-12-30 14:41:32.557",
    "reading": {
      "temperature": 19,
      "humidity": 63
    }
  },
  {
    "timestamp": "2017-12-30 14:41:31.028",
    "reading": {
      "temperature": 19,
      "humidity": 63
    }
  }
]
$

Clearly we will not see many changes in temperature or humidity, unless we place our thumb on the sensor or we blow warm breathe on it :-)

$ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq
[
  {
    "timestamp": "2017-12-30 14:43:16.787",
    "reading": {
      "temperature": 25,
      "humidity": 95
    }
  },
  {
    "timestamp": "2017-12-30 14:43:15.258",
    "reading": {
      "temperature": 25,
      "humidity": 95
    }
  },
  {
    "timestamp": "2017-12-30 14:43:13.729",
    "reading": {
      "temperature": 24,
      "humidity": 95
    }
  },
  {
    "timestamp": "2017-12-30 14:43:12.201",
    "reading": {
      "temperature": 24,
      "humidity": 95
    }
  },
  {
    "timestamp": "2017-12-30 14:43:05.616",
    "reading": {
      "temperature": 22,
      "humidity": 95
    }
  }
]
$

Needless to say, the North plugin will send the buffered data to the PI system using the PI Connector Relay OMF. Do not forget to set the correct IP address for the PI Connector Relay, as it is described here.

DHT11 in PI