Voice controlled Docker container deployment system using AWS and a Raspberry Pi

Playing around with AWS Lambda, Rekognition, Polly, DynamoDB, Lex, S3, etc. to create a system for deploying Docker containers by talking to a Raspberry Pi. The containers are deployed locally on a PC running the “p4docker” service while the other two services (p4security and p4voiceui) are running on the Raspberry Pi.

This was part of a project for an internal Pied Piper course here at Dell Tech earlier this year: https://bigdatadownunder.com/2019/10/11/innovating-ground-up-project-piper/

The code can be found here:

Download, install and run EdgeX Foundry in 5 min on Ubuntu 18.04 server

It’s extremely quick to get started with EdgeX Foundry. Less than 5 minutes – including installing Docker and Docker-compose (provided you have a reasonable internet connection).

Note: This is for the Edinburgh 1.01 release. Other releases can be downloaded from here: link

For the impatient: All required commands: link

Install docker-ce

vagrant@EdgeXblog:~$ sudo apt update
 Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
 Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease  
 Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                                       
 Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [380 kB]   
 Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]                     
....
vagrant@EdgeXblog:~$ sudo apt install apt-transport-https ca-certificates curl software-properties-common
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 ca-certificates is already the newest version (20180409).
...
vagrant@EdgeXblog:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK
vagrant@EdgeXblog:~$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
 Get:1 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
 Get:2 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [8,880 B]                                                
 Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease                                                 
 Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease                                   
 Hit:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease  
...                           
vagrant@EdgeXblog:~$ sudo apt update
 Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
 Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease                                   
 Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
...                  
vagrant@EdgeXblog:~$ sudo apt install docker-ce
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 The following additional packages will be installed:
...
vagrant@EdgeXblog:~$ sudo usermod -aG docker ${USER}

Download and install docker-compose

vagrant@EdgeXblog:~$ sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
 100   617    0   617    0     0   1804      0 --:--:-- --:--:-- --:--:--  1804
 100 15.4M  100 15.4M    0     0  4251k      0  0:00:03  0:00:03 --:--:-- 5278k
vagrant@EdgeXblog:~$ sudo chmod 755 /usr/local/bin/docker-compose

Download the EdgeX Foundry docker-compose.yml file

This is for the Edinburgh release, version 1.0.1. Others, including older releases, can be found here: link

vagrant@EdgeXblog:~$ wget https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/releases/edinburgh/compose-files/docker-compose-edinburgh-no-secty-1.0.1.yml
 --2019-10-11 08:46:28--  https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/releases/edinburgh/compose-files/docker-compose-edinburgh-no-secty-1.0.1.yml
 Resolving raw.githubusercontent.com (raw.githubusercontent.com)… 151.101.108.133
 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.108.133|:443… connected.
vagrant@EdgeXblog:~$ cp docker-compose-edinburgh-no-secty-1.0.1.yml docker-compose.yml

Pull the docker images from docker hub

vagrant@EdgeXblog:~$ sudo docker-compose pull
 Pulling volume         … done
 Pulling consul         … done
 Pulling config-seed    … done
 Pulling mongo          … done
 Pulling logging        … done
 Pulling system         … done
 Pulling notifications  … done
 Pulling metadata       … done
 Pulling data           … done
 Pulling command        … done
 Pulling scheduler      … done
 Pulling export-client  … done
 Pulling export-distro  … done
 Pulling rulesengine    … done
 Pulling device-virtual … done
 Pulling ui             … done
 Pulling portainer      … done

Start EdgeX Foundry

vagrant@EdgeXblog:~$ sudo docker-compose up -d
 Creating network "vagrant_edgex-network" with driver "bridge"
 Creating network "vagrant_default" with the default driver
 Creating volume "vagrant_db-data" with default driver
 Creating volume "vagrant_log-data" with default driver
 Creating volume "vagrant_consul-config" with default driver
 Creating volume "vagrant_consul-data" with default driver
...

List containers and ports

vagrant@EdgeXblog:~$ sudo docker-compose ps
            Name                          Command               State                                 Ports                              
 edgex-config-seed             /edgex/cmd/config-seed/con …   Exit 0                                                                   
 edgex-core-command            /core-command --registry - …   Up       0.0.0.0:48082->48082/tcp                                        
 edgex-core-consul             docker-entrypoint.sh agent …   Up       8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp,               
                                                                         0.0.0.0:8400->8400/tcp, 0.0.0.0:8500->8500/tcp,                 
                                                                         0.0.0.0:8600->8600/tcp, 8600/udp                                
 edgex-core-data               /core-data --registry --pr …   Up       0.0.0.0:48080->48080/tcp, 0.0.0.0:5563->5563/tcp                
 edgex-core-metadata           /core-metadata --registry  …   Up       0.0.0.0:48081->48081/tcp, 48082/tcp                             
 edgex-device-virtual          /device-virtual --profile= …   Up       0.0.0.0:49990->49990/tcp                                        
 edgex-export-client           /export-client --registry  …   Up       0.0.0.0:48071->48071/tcp                                        
 edgex-export-distro           /export-distro --registry  …   Up       0.0.0.0:48070->48070/tcp, 0.0.0.0:5566->5566/tcp                
 edgex-files                   /bin/sh -c /usr/bin/tail - …   Up                                                                       
 edgex-mongo                   docker-entrypoint.sh /bin/ …   Up       0.0.0.0:27017->27017/tcp                                        
 edgex-support-logging         /support-logging --registr …   Up       0.0.0.0:48061->48061/tcp                                        
 edgex-support-notifications   /support-notifications --r …   Up       0.0.0.0:48060->48060/tcp                                        
 edgex-support-rulesengine     /bin/sh -c java -jar -Djav …   Up       0.0.0.0:48075->48075/tcp                                        
 edgex-support-scheduler       /support-scheduler --regis …   Up       0.0.0.0:48085->48085/tcp                                        
 edgex-sys-mgmt-agent          /sys-mgmt-agent --registry …   Up       0.0.0.0:48090->48090/tcp                                        
 edgex-ui-go                   ./edgex-ui-server                Up       0.0.0.0:4000->4000/tcp                                          
 vagrant_portainer_1           /portainer -H unix:///var/ …   Up       0.0.0.0:9000->9000/tcp                           

Access EdgeX Foundry

Either access directly via the API’s or use the console on port 4000: “http://<ubuntu ip>:4000”.

  • Username: “admin”
  • Password: “admin”

Shut down EdgeX Foundry

Not that you would ever want to, but just in case: Stopping EdgeX Foundry containers can be done as per the below. Make sure the command is executed in the same directory as the “docker-compose.yml” file is located in.

vagrant@EdgeXblog:~$ sudo docker-compose stop
 Stopping edgex-device-virtual        … done
 Stopping edgex-ui-go                 … done
 Stopping edgex-support-rulesengine   … 
 Stopping edgex-export-distro         … 
 Stopping edgex-support-scheduler     … 
 Stopping edgex-core-command          … 

Photon OS on Raspberry Pi 3 model B+

Introduction

Photon OS is a VMware initiative to create a lightweight Linux based OS with container support. I have to admit my initial reaction to Photon OS was: “y tho?”

It’s a reasonable reaction. There are MANY Linux based OS options out there already and essentially all of them have container support. The reason for creating Photon OS would seem to be that VMware wants their own rubber-stamped Linux OS as part of an ecosystem under their control.

Photon OS’s redeeming feature is the fact that it’s really lightweight. Not as lightweight as Ubuntu Core though. Photon OS for Raspberry Pi weighs in at 512Mb while Ubuntu Core is 450Mb. Still, given the influence of VMware in virtualization and their (our) inroads into IoT / M2M with Pulse, it’s likely that Photon OS will take off eventually.

Currently the main barrier to widespread adoption of Photon OS is a lack of commercial support. At the moment it is simply available as an unsupported download from GitHub (here). This could change in the future though and in that case we may see it being utilized more broadly and also outside the lab environments it is currently inhabiting.

Note that unlike Raspbian, which is 32bit, Photon OS is a 64bit operating system. That too may be something that’ll help float the boat for some.

Getting started with Photon OS on the Raspberry Pi

First download the image from here: http://dl.bintray.com/vmware/photon/3.0/GA/rpi3/photon-rpi3-3.0-26156e2.tar.xz

Deflate the zx compressed image and save to a micro-SD card:

tar xf photon-rpi3-3.0-26156e2.tar.xz 
cd rpi3/
sudo dd if=photon-rpi3-3.0-26156e2d.raw of=/dev/mmcblk0 bs=4M;sudo sync

In this example the SD card device is /dev/mmcblk0. This may differ on other systems of course. Please check with “lsblk” or so and please do be careful. Linux / Unix folks don’t refer to dd as “Disk Destroyer” for nothing.

Boot the Raspberry Pi and log in. The default credentials are: root / changeme

DHCP and SSH are both enabled by default and should make it possible to access the Pi across the network if using a wired connection (I haven’t tried though). With a Raspberry Pi it’s likely a wireless connection would be more convenient however. Configuring Wi-Fi is easy and is described in the section that follows.

Photon OS Wi-Fi configuration

There are a few steps to go through for Wi-Fi connectivity but it’s not difficult.

Start the wpa_supplicant service

systemctl start wpa_supplicant@wlan0

Enable the wpa_supplicant service (so it starts with the Pi)

systemctl enable wpa_supplicant@wlan0

Check the service status

systemctl status wpa_supplicant@wlan0

Edit the dhcp settings to get DHCP for wlan0 and not eth0

root@photon-rpi3 [ ~ ]# cat /etc/systemd/network/99-dhcp-en.network 
[Match]
Name=e*

[Network]
DHCP=yes
IPv6AcceptRA=no
root@photon-rpi3 [ ~ ]# 

Change “Name=e*” to “Name=w*” to capture the wlan0 interface instead of the wired eth0 interface

root@photon-rpi3 [ ~ ]# vi /etc/systemd/network/99-dhcp-en.network

It should now look something like this:

root@photon-rpi3 [ ~ ]# cat /etc/systemd/network/99-dhcp-en.network 
[Match]
Name=w*

[Network]
DHCP=yes
IPv6AcceptRA=no
root@photon-rpi3 [ ~ ]# 

Restart networking

systemctl restart systemd-networkd

Configuring the wpa supplicant

WordPress changes the “>” signs regardless of what I do. The actual command can be found here for reference: https://pastebin.com/raw/gB5FkuhC

wpa_passphrase yourSSID yourPassword &gt;&gt; /etc/wpa_supplicant/wpa_supplicant-wlan0.conf
reboot

Installing Docker

Photon OS comes in a few different sizes and in the larger ones both Docker and Kubernetes are preinstalled. Not so with the Raspberry Pi version though, so we need to install Docker manually.

Packages are installed with either “yum” or “tdnf”. Docker is available from tdnf so we’ll use that to run the install below.

Refresh the cache but don’t update the packages

We need to refresh the tdnf cache to find the docker package. However, this process can also be used to update all packages. I found that this breaks Wi-Fi. So, if you use Wi-Fi I recommend:

root@photon-rpi3 [ ~ ]# tdnf update

Then select "n" to just refresh the cache without updating any packages.  

Search for Docker packages

root@photon-rpi3 [ ~ ]# tdnf list | grep docker
docker.aarch64                              18.06.2-2.ph3       photon-updates
docker-doc.aarch64                          18.06.2-2.ph3       photon-updates
docker.aarch64                              18.06.1-2.ph3             photon
docker-doc.aarch64                          18.06.1-2.ph3             photon
ovn-docker.aarch64                          2.8.2-3.ph3               photon
docker-py.noarch                            3.5.0-1.ph3               photon
docker-py3.noarch                           3.5.0-1.ph3               photon
docker-pycreds.noarch                       0.3.0-1.ph3               photon
docker-pycreds3.noarch                      0.3.0-1.ph3               photon
root@photon-rpi3 [ ~ ]# 

Install Docker

root@photon-rpi3 [ ~ ]# tdnf install docker

Installing:
libapparmor                    aarch64         2.13-7.ph3           photon-updates   66.57k 68168
libsepol                       aarch64         2.8-1.ph3            photon          611.89k 626576
libselinux                     aarch64         2.8-1.ph3            photon          174.16k 178338
libseccomp                     aarch64         2.3.3-1.ph3          photon          286.28k 293153
libltdl                        aarch64         2.4.6-3.ph3          photon           35.53k 36384
device-mapper-libs             aarch64         2.02.181-1.ph3       photon          315.39k 322960
docker                         aarch64         18.06.2-2.ph3        photon-updates  154.39M 161893076

Total installed size: 155.85M 163418655
Is this ok [y/N]:y

Downloading:
libapparmor                              39330    100%
libsepol                                275180    100%
libselinux                               84756    100%
libseccomp                               80091    100%
libltdl                                  24218    100%
device-mapper-libs                      149078    100%
docker                                43826910    100%
Testing transaction
Running transaction
Installing/Updating: libsepol-2.8-1.ph3.aarch64
Installing/Updating: libselinux-2.8-1.ph3.aarch64
Installing/Updating: device-mapper-libs-2.02.181-1.ph3.aarch64
Installing/Updating: libltdl-2.4.6-3.ph3.aarch64
Installing/Updating: libseccomp-2.3.3-1.ph3.aarch64
Installing/Updating: libapparmor-2.13-7.ph3.aarch64
Installing/Updating: docker-18.06.2-2.ph3.aarch64

Complete!

Start and Enable the docker service

root@photon-rpi3 [ ~ ]# systemctl start docker
root@photon-rpi3 [ ~ ]# systemctl enable docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
root@photon-rpi3 [ ~ ]# 

Verify the Docker installation

root@photon-rpi3 [ ~ ]# docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
3b4173355427: Pull complete 
Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535
Status: Downloaded newer image for hello-world:latest
root@photon-rpi3 [ ~ ]# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@photon-rpi3 [ ~ ]# 

That’s all! Photon OS is installed, Wi-Fi configured, Docker installed and verified. Ready to rock.