dnsmasq: failed to create listening socket for port 53: Address already in use

Ubuntu 19:10: systemd-resolved blocks port 53 and thereby preventing any service using port 53 (like dnsmasq) from starting

 Jan 29 03:31:58 ubuntupxe02 dnsmasq[2386]: dnsmasq: failed to create listening socket for port 53: Address already in use
 Jan 29 03:31:58 ubuntupxe02 dnsmasq[2386]: failed to create listening socket for port 53: Address already in use
 Jan 29 03:31:58 ubuntupxe02 dnsmasq[2386]: FAILED to start up
 Jan 29 03:31:58 ubuntupxe02 systemd[1]: dnsmasq.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
 Jan 29 03:31:58 ubuntupxe02 systemd[1]: dnsmasq.service: Failed with result 'exit-code'.
 Jan 29 03:31:58 ubuntupxe02 systemd[1]: Failed to start dnsmasq - A lightweight DHCP and caching DNS server.

Verify that the port is used by systemd-resolve

jonas@ubuntupxe02:~$ sudo lsof -i -P -n | grep LIST
 systemd-r  784 systemd-resolve   13u  IPv4  19378      0t0  TCP (LISTEN)
 sshd       859            root    3u  IPv4  23918      0t0  TCP *:22 (LISTEN)
 sshd       859            root    4u  IPv6  23920      0t0  TCP *:22 (LISTEN)
 apache2   1705            root    4u  IPv6  27900      0t0  TCP *:80 (LISTEN)
 apache2   1706        www-data    4u  IPv6  27900      0t0  TCP *:80 (LISTEN)
 apache2   1707        www-data    4u  IPv6  27900      0t0  TCP *:80 (LISTEN)

Stop the service

jonas@ubuntupxe02:~$ sudo systemctl stop systemd-resolved

Edit the systemd-resolved config file

 jonas@ubuntupxe02:~$ sudo vi /etc/systemd/resolved.conf
 jonas@ubuntupxe02:~$ cat !$ | grep DNS
 cat /etc/systemd/resolved.conf | grep DNS

Create symlink to /etc/resolv.conf

jonas@ubuntupxe02:~$ sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Start systemd-resolved service

jonas@ubuntupxe02:~$ sudo systemctl start systemd-resolved

Start dnsmasq

jonas@ubuntupxe02:~$ sudo systemctl start dnsmasq
jonas@ubuntupxe02:~$ sudo systemctl status dnsmasq
 ● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
    Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
    Active: active (running) since Wed 2020-01-29 03:56:12 UTC; 6s ago
   Process: 1312 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
   Process: 1314 ExecStart=/etc/init.d/dnsmasq systemd-exec (code=exited, status=0/SUCCESS)

Verify that dnsmasq is now the user of port 53

jonas@ubuntupxe02:~$ sudo lsof -i -P -n | grep LISTEN
 sshd     823     root    3u  IPv4  23016      0t0  TCP *:22 (LISTEN)
 sshd     823     root    4u  IPv6  23018      0t0  TCP *:22 (LISTEN)
 apache2  874     root    4u  IPv6  22454      0t0  TCP *:80 (LISTEN)
 apache2  875 www-data    4u  IPv6  22454      0t0  TCP *:80 (LISTEN)
 apache2  876 www-data    4u  IPv6  22454      0t0  TCP *:80 (LISTEN)
 dnsmasq 1331  dnsmasq    5u  IPv4  28097      0t0  TCP *:53 (LISTEN)
 dnsmasq 1331  dnsmasq    7u  IPv6  28099      0t0  TCP *:53 (LISTEN)

Kudos to Nitin Gurbani for the solution

Kubernetes home lab: Upgraded edition with functional LoadBalancer and external access to pods

In a previous post we covered the deployment of a home k8s lab, but this post will show a much better way to do it as well as improving on the end result – a fully functional local cluster.

The installation is done using Vagrant with Flannel networking and MetalLB for load balancing.

Commands for the session can be downloaded from here: https://jonamiki.com/wp-content/uploads/2019/11/kubectl-demo-commands.txt


Why go through the trouble of setting up a home lab for k8s? Well, while using public cloud services is a quick and easy way it will cost money to deploy and run. It will also rely upon predefined cloud formation templates which have already been created. Doing it locally can provide both a more economical way to use k8s as well as give more insight into the internal workings and how it’s actually set up.

Why not use Minikube? Because it’s overly simplified. Using a cluster deployment like this is not only a better learning and testing experience but it also provides an overall more realistic experience of a “real” k8s installation.

Configuration files

Download the Vagrant, Flannel and MetalLB files from GitHub or clone with Git

git clone https://github.com/jonas-werner/k8s-home-lab-with-vagrant.git

Video: Editing the config files and standing up the cluster

Video: Getting started using the new K8s cluster

Enjoy your new Kubernetes powers!

Ubuntu: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY

Error when adding repo to install kubectl

jonas@octo:~$ sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
Hit:1 http://jp.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 https://nvidia.github.io/libnvidia-container/ubuntu18.04/amd64  InRelease [1,106 B]                                                                                                  
Get:3 https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/amd64  InRelease [1,103 B]                                                                                             
Hit:4 http://jp.archive.ubuntu.com/ubuntu bionic-updates InRelease                                                                                                                         
Hit:5 https://download.docker.com/linux/ubuntu bionic InRelease                                                                                                                            
Get:6 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease [1,096 B]                                                                                                        
Hit:7 http://jp.archive.ubuntu.com/ubuntu bionic-backports InRelease                                                                                                                       
Hit:9 http://security.ubuntu.com/ubuntu bionic-security InRelease                                                                                                                 
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]                                                                   
Err:8 https://packages.cloud.google.com/apt kubernetes-xenial InRelease                              
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
Hit:10 https://cf-cli-debian-repo.s3.amazonaws.com stable InRelease                             
Reading package lists… Done                                        
W: GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
E: The repository 'http://apt.kubernetes.io kubernetes-xenial InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Solved by adding the missing key as follows

jonas@octo:~$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 6A030B21BA07F4FB

Executing: /tmp/apt-key-gpghome.JKpxFjtwsU/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys 6A030B21BA07F4FB
gpg: key 6A030B21BA07F4FB: public key "Google Cloud Packages Automatic Signing Key <a href="mailto:gc-team@google.com">gc-team@google.com</a>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Voice controlled Docker container deployment system using AWS and a Raspberry Pi

Playing around with AWS Lambda, Rekognition, Polly, DynamoDB, Lex, S3, etc. to create a system for deploying Docker containers by talking to a Raspberry Pi. The containers are deployed locally on a PC running the “p4docker” service while the other two services (p4security and p4voiceui) are running on the Raspberry Pi.

This was part of a project for an internal Pied Piper course here at Dell Tech earlier this year: https://bigdatadownunder.com/2019/10/11/innovating-ground-up-project-piper/

The code can be found here:

Download, install and run EdgeX Foundry in 5 min on Ubuntu 18.04 server

It’s extremely quick to get started with EdgeX Foundry. Less than 5 minutes – including installing Docker and Docker-compose (provided you have a reasonable internet connection).

Note: This is for the Edinburgh 1.01 release. Other releases can be downloaded from here: link

For the impatient: All required commands: link

Install docker-ce

vagrant@EdgeXblog:~$ sudo apt update
 Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
 Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease  
 Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]                                       
 Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [380 kB]   
 Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]                     
vagrant@EdgeXblog:~$ sudo apt install apt-transport-https ca-certificates curl software-properties-common
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 ca-certificates is already the newest version (20180409).
vagrant@EdgeXblog:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
vagrant@EdgeXblog:~$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
 Get:1 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
 Get:2 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [8,880 B]                                                
 Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease                                                 
 Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease                                   
 Hit:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease  
vagrant@EdgeXblog:~$ sudo apt update
 Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
 Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease                                   
 Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
vagrant@EdgeXblog:~$ sudo apt install docker-ce
 Reading package lists… Done
 Building dependency tree       
 Reading state information… Done
 The following additional packages will be installed:
vagrant@EdgeXblog:~$ sudo usermod -aG docker ${USER}

Download and install docker-compose

vagrant@EdgeXblog:~$ sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
 100   617    0   617    0     0   1804      0 --:--:-- --:--:-- --:--:--  1804
 100 15.4M  100 15.4M    0     0  4251k      0  0:00:03  0:00:03 --:--:-- 5278k
vagrant@EdgeXblog:~$ sudo chmod 755 /usr/local/bin/docker-compose

Download the EdgeX Foundry docker-compose.yml file

This is for the Edinburgh release, version 1.0.1. Others, including older releases, can be found here: link

vagrant@EdgeXblog:~$ wget https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/releases/edinburgh/compose-files/docker-compose-edinburgh-no-secty-1.0.1.yml
 --2019-10-11 08:46:28--  https://raw.githubusercontent.com/edgexfoundry/developer-scripts/master/releases/edinburgh/compose-files/docker-compose-edinburgh-no-secty-1.0.1.yml
 Resolving raw.githubusercontent.com (raw.githubusercontent.com)…
 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)||:443… connected.
vagrant@EdgeXblog:~$ cp docker-compose-edinburgh-no-secty-1.0.1.yml docker-compose.yml

Pull the docker images from docker hub

vagrant@EdgeXblog:~$ sudo docker-compose pull
 Pulling volume         … done
 Pulling consul         … done
 Pulling config-seed    … done
 Pulling mongo          … done
 Pulling logging        … done
 Pulling system         … done
 Pulling notifications  … done
 Pulling metadata       … done
 Pulling data           … done
 Pulling command        … done
 Pulling scheduler      … done
 Pulling export-client  … done
 Pulling export-distro  … done
 Pulling rulesengine    … done
 Pulling device-virtual … done
 Pulling ui             … done
 Pulling portainer      … done

Start EdgeX Foundry

vagrant@EdgeXblog:~$ sudo docker-compose up -d
 Creating network "vagrant_edgex-network" with driver "bridge"
 Creating network "vagrant_default" with the default driver
 Creating volume "vagrant_db-data" with default driver
 Creating volume "vagrant_log-data" with default driver
 Creating volume "vagrant_consul-config" with default driver
 Creating volume "vagrant_consul-data" with default driver

List containers and ports

vagrant@EdgeXblog:~$ sudo docker-compose ps
            Name                          Command               State                                 Ports                              
 edgex-config-seed             /edgex/cmd/config-seed/con …   Exit 0                                                                   
 edgex-core-command            /core-command --registry - …   Up;48082/tcp                                        
 edgex-core-consul             docker-entrypoint.sh agent …   Up       8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp,               
                                                               ;8600/tcp, 8600/udp                                
 edgex-core-data               /core-data --registry --pr …   Up;48080/tcp,;5563/tcp                
 edgex-core-metadata           /core-metadata --registry  …   Up;48081/tcp, 48082/tcp                             
 edgex-device-virtual          /device-virtual --profile= …   Up;49990/tcp                                        
 edgex-export-client           /export-client --registry  …   Up;48071/tcp                                        
 edgex-export-distro           /export-distro --registry  …   Up;48070/tcp,;5566/tcp                
 edgex-files                   /bin/sh -c /usr/bin/tail - …   Up                                                                       
 edgex-mongo                   docker-entrypoint.sh /bin/ …   Up;27017/tcp                                        
 edgex-support-logging         /support-logging --registr …   Up;48061/tcp                                        
 edgex-support-notifications   /support-notifications --r …   Up;48060/tcp                                        
 edgex-support-rulesengine     /bin/sh -c java -jar -Djav …   Up;48075/tcp                                        
 edgex-support-scheduler       /support-scheduler --regis …   Up;48085/tcp                                        
 edgex-sys-mgmt-agent          /sys-mgmt-agent --registry …   Up;48090/tcp                                        
 edgex-ui-go                   ./edgex-ui-server                Up;4000/tcp                                          
 vagrant_portainer_1           /portainer -H unix:///var/ …   Up;9000/tcp                           

Access EdgeX Foundry

Either access directly via the API’s or use the console on port 4000: “http://<ubuntu ip>:4000”.

  • Username: “admin”
  • Password: “admin”

Shut down EdgeX Foundry

Not that you would ever want to, but just in case: Stopping EdgeX Foundry containers can be done as per the below. Make sure the command is executed in the same directory as the “docker-compose.yml” file is located in.

vagrant@EdgeXblog:~$ sudo docker-compose stop
 Stopping edgex-device-virtual        … done
 Stopping edgex-ui-go                 … done
 Stopping edgex-support-rulesengine   … 
 Stopping edgex-export-distro         … 
 Stopping edgex-support-scheduler     … 
 Stopping edgex-core-command          …