Postman calls for EdgeX Foundry

Viewing data, creating rules and export topics in EdgeX foundry can easily be done in Postman. I’ve put together a small collection of REST calls which may be useful for those who are starting out with EdgeX Foundry and want to leverage Postman for the API interaction.

The API calls are based on the contents of this API walkthrough: https://nexus.edgexfoundry.org/content/sites/docs/staging/master/docs/_build/html/Ch-Walkthrough.html

In case the below “Run in Postman” link doesn’t work the collection can be downloaded and imported from here: link

Download, install and run EdgeX Foundry in 5 min on Ubuntu 18.04 server

It’s extremely quick to get started with EdgeX Foundry. Less than 5 minutes – including installing Docker and Docker-compose (provided you have a reasonable internet connection).

Note: This is for the Edinburgh 1.01 release. Other releases can be downloaded from here: link

For the impatient: All required commands: link

Install docker-ce

vagrant@EdgeXblog:~$ sudo apt update
 Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
 Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease  
 Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                                       
 Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [380 kB]   
 Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]                     
....
vagrant@EdgeXblog:~$ sudo apt install apt-transport-https ca-certificates curl software-properties-common
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 ca-certificates is already the newest version (20180409).
...
vagrant@EdgeXblog:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK
vagrant@EdgeXblog:~$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
 Get:1 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
 Get:2 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [8,880 B]                                                
 Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease                                                 
 Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease                                   
 Hit:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease  
...                           
vagrant@EdgeXblog:~$ sudo apt update
 Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
 Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease                                   
 Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
...                  
vagrant@EdgeXblog:~$ sudo apt install docker-ce
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 The following additional packages will be installed:
...
vagrant@EdgeXblog:~$ sudo usermod -aG docker ${USER}

Download and install docker-compose

vagrant@EdgeXblog:~$ sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
 100   617    0   617    0     0   1804      0 --:--:-- --:--:-- --:--:--  1804
 100 15.4M  100 15.4M    0     0  4251k      0  0:00:03  0:00:03 --:--:-- 5278k
vagrant@EdgeXblog:~$ sudo chmod 755 /usr/local/bin/docker-compose

Download the EdgeX Foundry docker-compose.yml file

This is for the Edinburgh release, version 1.0.1. Others, including older releases, can be found here: link

vagrant@EdgeXblog:~$ wget https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/releases/edinburgh/compose-files/docker-compose-edinburgh-no-secty-1.0.1.yml
 --2019-10-11 08:46:28--  https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/releases/edinburgh/compose-files/docker-compose-edinburgh-no-secty-1.0.1.yml
 Resolving raw.githubusercontent.com (raw.githubusercontent.com)… 151.101.108.133
 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.108.133|:443… connected.
vagrant@EdgeXblog:~$ cp docker-compose-edinburgh-no-secty-1.0.1.yml docker-compose.yml

Pull the docker images from docker hub

vagrant@EdgeXblog:~$ sudo docker-compose pull
 Pulling volume         … done
 Pulling consul         … done
 Pulling config-seed    … done
 Pulling mongo          … done
 Pulling logging        … done
 Pulling system         … done
 Pulling notifications  … done
 Pulling metadata       … done
 Pulling data           … done
 Pulling command        … done
 Pulling scheduler      … done
 Pulling export-client  … done
 Pulling export-distro  … done
 Pulling rulesengine    … done
 Pulling device-virtual … done
 Pulling ui             … done
 Pulling portainer      … done

Start EdgeX Foundry

vagrant@EdgeXblog:~$ sudo docker-compose up -d
 Creating network "vagrant_edgex-network" with driver "bridge"
 Creating network "vagrant_default" with the default driver
 Creating volume "vagrant_db-data" with default driver
 Creating volume "vagrant_log-data" with default driver
 Creating volume "vagrant_consul-config" with default driver
 Creating volume "vagrant_consul-data" with default driver
...

List containers and ports

vagrant@EdgeXblog:~$ sudo docker-compose ps
            Name                          Command               State                                 Ports                              
 edgex-config-seed             /edgex/cmd/config-seed/con …   Exit 0                                                                   
 edgex-core-command            /core-command --registry - …   Up       0.0.0.0:48082->48082/tcp                                        
 edgex-core-consul             docker-entrypoint.sh agent …   Up       8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp,               
                                                                         0.0.0.0:8400->8400/tcp, 0.0.0.0:8500->8500/tcp,                 
                                                                         0.0.0.0:8600->8600/tcp, 8600/udp                                
 edgex-core-data               /core-data --registry --pr …   Up       0.0.0.0:48080->48080/tcp, 0.0.0.0:5563->5563/tcp                
 edgex-core-metadata           /core-metadata --registry  …   Up       0.0.0.0:48081->48081/tcp, 48082/tcp                             
 edgex-device-virtual          /device-virtual --profile= …   Up       0.0.0.0:49990->49990/tcp                                        
 edgex-export-client           /export-client --registry  …   Up       0.0.0.0:48071->48071/tcp                                        
 edgex-export-distro           /export-distro --registry  …   Up       0.0.0.0:48070->48070/tcp, 0.0.0.0:5566->5566/tcp                
 edgex-files                   /bin/sh -c /usr/bin/tail - …   Up                                                                       
 edgex-mongo                   docker-entrypoint.sh /bin/ …   Up       0.0.0.0:27017->27017/tcp                                        
 edgex-support-logging         /support-logging --registr …   Up       0.0.0.0:48061->48061/tcp                                        
 edgex-support-notifications   /support-notifications --r …   Up       0.0.0.0:48060->48060/tcp                                        
 edgex-support-rulesengine     /bin/sh -c java -jar -Djav …   Up       0.0.0.0:48075->48075/tcp                                        
 edgex-support-scheduler       /support-scheduler --regis …   Up       0.0.0.0:48085->48085/tcp                                        
 edgex-sys-mgmt-agent          /sys-mgmt-agent --registry …   Up       0.0.0.0:48090->48090/tcp                                        
 edgex-ui-go                   ./edgex-ui-server                Up       0.0.0.0:4000->4000/tcp                                          
 vagrant_portainer_1           /portainer -H unix:///var/ …   Up       0.0.0.0:9000->9000/tcp                           

Access EdgeX Foundry

Either access directly via the API’s or use the console on port 4000: “http://<ubuntu ip>:4000”.

  • Username: “admin”
  • Password: “admin”

Shut down EdgeX Foundry

Not that you would ever want to, but just in case: Stopping EdgeX Foundry containers can be done as per the below. Make sure the command is executed in the same directory as the “docker-compose.yml” file is located in.

vagrant@EdgeXblog:~$ sudo docker-compose stop
 Stopping edgex-device-virtual        … done
 Stopping edgex-ui-go                 … done
 Stopping edgex-support-rulesengine   … 
 Stopping edgex-export-distro         … 
 Stopping edgex-support-scheduler     … 
 Stopping edgex-core-command          … 

EdgeX Foundry version 1.0 is now available

The Open Source IoT solution called EdgeX Foundry has just had it’s first Long-Term Support (LTS) release. This is a big deal and a real milestone since it’s finally out of 0.x versions and into the first, big 1.x release. After a journey of over 2 years it’s finally ready for broad adoption. EdgeX Foundry is an official Linux Foundation project and part of LF Edge.

Why is EdgeX Foundry so relevant?

Data ingestion

  • It is a native speaker of the protocols and data formats of a myriad edge devices
  • Without the need for agents, it ingests data from most edge devices and sensors
  • It converts the data to XML or JSON for easy processing
  • It streams the data in real-time to internal or external cloud and big-data platforms for visualization and processing

Control and automation

  • It supports the native commands of edge devices and can change camera angles, fan speeds, etc.
  • It has rules that can act on input and trigger commands for instant action and automation

Architecture

  • EdgeX Foundry is cloud-native
  • It’s open source and can be downloaded by anyone free of charge
  • It’s made up of microservices running in docker containers
  • It’s modular, flexible and can be integrated into other IoT management systems

How does it fit in with the rest of the IoT world?

While the Internet of Things is fairly new and very much a buzzword, the concept of connected devices runs back much longer through the Machine 2 Machine era.

Oftentimes these solutions are vendor-specific, siloed off and lack any unified layer for insight, control and management. EdgeX Foundry bridges not only the old M2M with the new IoT solutions but also connects the Edge to the core DC to the Cloud. It’s the glue that holds the world of IoT together.

It can favorably be used both stand-alone, as a part of a larger IoT solution (containers can easily be integrated as they contain individual services) or together with a commercial IoT solution such as VMware Pulse IoT Center 2.0.

How to get started

Many resources are available for those looking to get started with EdgeX Foundry. There are starter guides and tutorials on the project page as well as docker-compose files on GitHub as per the below

Pulled images from Docker hub

Version 1.0.0 as far as the eye can see 🙂

Tentacle Tales from Tokyo

A story of IoT and Octopods in the far-east (or, if preferred: How to leverage EdgeX Foundry for a successful customer demonstration).

A normal day in Shinjuku: The EdgeX Foundry Cthulhu lookalike taking over the high-rise district

Last autumn we had ongoing discussions with a customer regarding their IoT efforts. Primarily the discussions had been focused around deployment and management of the edge devices from an infrastructure standpoint. Working at Dell Technologies, that part comes naturally. However, once the discussion broadened it came across the data management side. How does one best deal with the data flow from a myriad IoT devices? How would that data be stored long-term and later used for analytics? Could data be intelligently acted upon in real-time based on alerts from a sensor cluster somewhere

As the hardware management had already been taken care of by EDM (Edge Device Manager) and Edge Gateway OS status and alerting covered by VMware Pulse, we found ourselves looking to a new solution for the data management side. At the back of the room the pre-sales manager said, “Maybe we should look into EdgeX Foundry?”. At that time nobody had much information about this open source, micro-service solution for IoT, but that was about to change.

I took it upon myself to do some research and get it going. It turned out to be exactly what we had been looking for all along.

What is EdgeX Foundry?

EdgeX Foundry functionality can (very simplified) be boiled down to the following:

  • Ingest data from a wide variety of IoT and M2M devices as well as control them (it speaks many languages!)
  • Act upon sensor data in real-time (it has a rules engine that can automate actions)
  • Format, compress & encrypt data prior to forwarding it for storage and analytics (for example Azure)

So, Receive, Process and Forward data with optional actions taken along the way. It’s essentially the glue that holds the world of IoT devices and both internal and external cloud services together. With all its flexibility, adaptability and far-reaching access to almost any IoT device or cloud service it is plain to see why it’s represented by an Octopus 🙂

It elegantly bridges the gap between the edge and the cloud and can reside anywhere along the data path. The preferred location is close to the edge, to minimize latency and network load, but as it’s based on micro-services running in Docker containers it can (and should) be spread out across the infrastructure layer so that each service can make itself useful when and where it’s needed the most.

https://www.edgexfoundry.org/

EdgeX Foundry has two ends

  • Southbound: Where the IoT Edge is and the data is generated. Thanks to the many protocols it speaks it can interface with both old and new IoT devices.
  • Northbound: Where the external / internal cloud services reside. Where data warehousing can store the vast volumes of data created and where Machine Learning tools can be leveraged to gain insights from data over time.

In between these two the core and supporting services reside, with a temporary NoSQL data storage (30 min by default), the Scheduling service as well as the Rules Engine for actions and a command module to interface with machinery at the edge when action need to be taken in real-time.

Device templates

To make it easy to add new IoT devices there are ready-made templates which help integrate sensors with EdgeX Foundry. These are made by Dell EMC or the respective manufacturer and can be downloaded from GitHub.

The device templates play an important role in identifying the types of data being sent to EdgeX Foundry. For example, when receiving a value like “51” from an IoT device it’s important to clarify to the system what that data represents. Is it 51 degrees C? Is it 51% humidity or is it 51 vibrations / sec?

The templates also allow for commands to be sent and status codes to be exchanged. Commands like setting the sensitivity of a camera or motion detector may be important to certain IoT devices and EdgeX Foundry device templates include the valid commands for each device.

Demonstrating the solution

So, as Octopods go this one is quite capable and a great fit for our customer! The next step is to showcase this as seeing is believing and we’re all about being hands-on. After all, it’s most interesting when the rubber hits the road (or when the tentacles grab the Edge Gateways as is the case for us). How do we get this going?

Demonstration goals

  • Explain what EdgeX Foundry all about and show how it can be leveraged to propel their IoT efforts
  • Have a real microservice environment going including all the EdgeX Foundry goodness
  • Show real-time data collection, processing and export to external cloud services
  • Keep it interesting and relevant for the customer

Getting the base system up and running is surprisingly easy. The EdgeX Foundry website has a happy “Get started” button and it links to a hands-on tutorial. All that’s needed is a system with Docker Compose. There’s a simple text file (yaml) which docker-compose will use to bootstrap all required Docker images and get them up and running on the same virtual network. Once that’s done the system can be interfaced with using REST clients like, for example, Postman. Most of the micro service comes with its own REST endpoint which can be interfaced with. Either to retrieve or send data as well as modifying settings.

Postman used to directly interact with the EdgeX Foundry microservice APIs

The tutorial goes through a fictional scenario with a smart-camera along with simulated data generated. While this is nice, for our demo we want real data from real sensors. As such we take the example of the fictional smart camera and modify it to fit two scenarios:

  1. Humidity and Temperature sensors (we have no IoT devices, so we build one using a Raspberry Pi)
  2. Smart camera powered by machine learning (we use a simple webcam + a Python script from the Tensorflow tutorial for image recognition)

Providing a graphical interface

While we’re quite happy working on the command line, it doesn’t always make for a good customer demo to just show text on a screen. We need a GUI of some sort. After talking to the creator of EdgeX Foundry – Jim White, we get introduced to Keith from IOTech in the UK.  

IOTech has just what we need – a GUI for EdgeX Foundry that can be used to, among other things, display sensor data and graph it. Much easier on the eyes for our customer, so we head over to IOTech’s GitHub page where we find a Docker file for their UI. It’s quick work to build a Docker image and subsequently add it to the docker-compose.yml file used to initiate all EdgeX Foundry services.

Now it starts up along with all other micro services and can communicate across the same network. We’re excited and we’re hopeful our customer will be as well once they get to see it.

EdgeX Manager from IOTech

Generating sensor data as input

Next, we need some data input. A Raspberry Pi + a breadboard and a DHT11 sensor will provide both Humidity and Temperature data. It’s interfacing with the Raspberry Pi using the GPIO pins and a little help of AdaFruit’s DHT library. We’re a big fan of Lady Ada and her libraries.

So, now we have sensor data. However, to get it sent to EdgeX Foundry we need to use one of the many protocols supported by the EdgeX Foundry device service. Since we’ve been working with Redfish and the REST protocol for server automation and since the Dell EMC GitHub page contains plenty of examples of this (courtesy of Texas Roemer) we pull down some Python files and borrow from there.

A little tweaking later our Raspberry Pi is humming along and sending sensor data via REST every 2 seconds to the EdgeX Foundry device service. To make it interface correctly with the device service we have modified the smart camera template provided in the initial EdgeX Foundry API tutorial to match the data sent by the Raspberry Pi. Instead of counting Humans and Dogs as in the original we’re now sending “Humidity in Percent” and “Temperature in C”.

RaspberryPi with a breadboard and DHT-11 humidity and temperature sensor

Looking at the data through IOTech’s GUI we can easily graph the data and provide the customer with a visually pleasing representation of data that is generated in real-time right in front of them. This also has the added benefit that we can ask the customer to touch or breathe on the sensor to change temperature and / or humidity during the demo. This creates an impactful element which helps convey the realization that this system is live, right here, right now.

Exporting data to external cloud services

But wait. How about data export? Wasn’t EdgeX Foundry supposed to be able to export data to external services? Well, the tutorial also contains information on this topic. We create an account at CloudMQTT and configure EdgeX Foundry to export data there. CloudMQTT is essentially a MQTT broker with a live GUI where data can be seen entering the system as it happens. EdgeX Foundry authenticates with the CloudMQTT service and start forwarding on our data. We can view it as it comes in in XML format. This can easily be changed to JSON on the fly and we do this during the live demo later. Jim White provide helpful input on this topic to make sure it functions as expected.

Exporting EdgeX Foundry data stream to CloudMQTT in real-time

Adding the smart camera

This is all good and well for simple sensor data. What about our ML powered smart camera? The Tensorflow tutorials come with a GitHub repository full of useful tools. Among them the classify_image.py script which takes an image as input and then tells you what it sees in plain text. We hook it up to a web camera, run it in a loop and export the top scoring items the Tensorflow script finds after running inferencing on our live camera input. The resulting data is streamed using REST, just like the Humidity and Temperature data from the Raspberry Pi earlier, to EdgeX Foundry device service and then onward to the public cloud over MQTT where it can be viewed entering the system live.

All preparation is done and we’re ready for our demo.

Customer feedback and next steps

So the day of the demonstration came and went. What was the customer’s impression of the solution? In short: They’re very excited about the capabilities, flexibility and potential of the EdgeX Foundry system. They’re now looking to get their own environment up and running to pursue further tests which will drive the integration of EdgeX Foundry into their IoT initiative. This is done alongside Dell EMC EDM and VMware Pulse as well as their home-grown orchestration platform.

Tips for a successful setup

While the process was made fairly straight forward by the online tutorials for both EdgeX Foundry and Tensorflow as well as the Dell EMC Redfish GitHub repository there were some areas which warranted more attention to get right.

The device templates

For the demo custom device templates were created for both the Smart Camera as well as the Raspberry Pi sensors. It can be challenging to create a device template which will be accepted by the system. If the fields don’t match the data already having been sent do the metadata subsystem an upload attempt will result in errors. Carefully compare the tutorial templates with the REST API calls to ensure the fields that need to be the same, actually match.

Running the REST API calls via Postman vs. Python

Personally, I found using Postman to be handy for troubleshooting or the occasional API call but gradually becoming tedious as more calls via REST were required. Instead I made a Python script which automates the process and can be run more easily from the command line. This may simply be a matter of preference though.

The IOTech GUI

If the IOTech GUI is desired the Dockerfile can be downloaded from here:

https://github.com/edgexfoundry-holding/edgex-ui-clojure

Simply create a Docker image and add it to the docker-compose.yml file to have it start / stop with the rest of the EdgeX Foundry micro services.

Microservices and their description

Micro serviceDescription
iotech-managerUI provided by IOTech in the UK
edgex-export-distroExport Distribution microservice, receives data from Core Data, filters, transforms, and formats the data, then distributes it through REST, MQTT, or 0MQ to client endpoints of choice.
edgex-support-schedulerPerforms data housekeeping tasks, scrubs exported / stale events and data
edgex-core-commandConduit for other services to trigger action on devices and sensors through their managing Device Services
edgex-export-clientService doing the actual exporting of XML or JSON data to external services
edgex-support-notificationsIssues notifications via SMS, Email, REST, AMQP or MQTT. Sometimes triggered by the Rules engine noticing values out of bounds
edgex-core-metadataHolds IoT Device information, including Device Services and Device Profiles
edgex-core-dataCentral, optionally persistent, datastore
edgex-support-loggingLogging services
edgexfoundry_del_portainerEdgeX deployment and orchestration
edgex-mongoPersistent, NoSQL datastore
edgex-core-consulConfiguration and Registry, Key-Value store. EdgeX services report to Consul when starting up and Consul continuously monitor their status
edgex-support-rulesengineEdge-side event triggering mechanism for quick action at the edge. Based on Drools rules engine