Telemetry Streaming with Dell EMC PowerEdge 14G servers, Python, InfluxDB and Grafana

In early 2020 a new feature was added to the PowerEdge 14G servers called “Telemetry Streaming’. This feature makes it possible to send a continuous stream of telemetry containing in-depth information about the state of the server and its various components including, but not limited to, the following:

  • CPU, Memory and Fans
  • FPGA and GPU
  • PCIe slots
  • Airflow inside server
  • Power usage information

Since the level and depth of information collected with this method FAR exceeds what has been previously possible using IPMI or other tools, this feature can help in several areas. For example:

  • Power ML algorithms for Anomaly Detection
  • Provide detailed inventory, usage and status information
  • Assist with security and auditing

Blog posts in this series

Introductory video and demo

Links to useful documents / scripts published elsewhere:

Enable Telemetry Streaming with RACADM, Scripts and/or Redfish and Postman

This post aim to describe three methods with which to enable the Telemetry Streaming feature in the iDRAC9 on Dell EMC 14G PowerEdge servers:

  • Enable using RACADM / SSH
  • Enable using provided GitHub scripts
  • Enable using Redfish and Postman

Enabling using RACADM and Redfish are selective methods while using the GitHub script enables ALL reports in one go. Personally I’d recommend being selective to start with until it is clear what data is required / desired.

Note that enabling everything will result in just shy of 3M data points / 24h / server

Blog posts in this series:

License

Get a 30 day trial of the iDRAC9 Datacenter license here: https://www.dell.com/support/article/en-us/sln309292/idrac-cmc-openmanage-enterprise-openmanage-integration-with-servicenow-and-dpat-trial-licenses?lang=en

Enable using RACADM / SSH

Enable using GitHub script

Enable using Redfish and Postman

URI and payload for Postman

URI:
https://IDRAC_IP/redfish/v1/Managers/iDRAC.Embedded.1/Attributes

Auth: Basic (root / calvin by default)

Methods: 
GET for viewing current settings
PATCH for changing settings

Payload for enabling streaming telemetry (indentation doesn't work properly in Wordpress, sorry):
{
"Attributes": {
"Telemetry.1.EnableTelemetry": "Enabled"
}
}

Payload for enabling / disabling reports (indentation doesn't work properly in Wordpress, sorry)
{
"Attributes": {
"TelemetryCPUSensor.1.EnableTelemetry": "Enabled",
"TelemetryPowerStatistics.1.EnableTelemetry": "Disabled",
"TelemetrySensor.1.EnableTelemetry": "Enabled"
}
}

Configuring Telemetry Streaming

This article contains the practical steps to set up and configure Telemetry Streaming. It assumes it has already been enabled using one of the methods described in the previous article here. In this blog post we use the following:

  • Python script to collect the data
  • InfluxDB for storing the data
  • Grafana for visualizing the data

Blog posts in this series

Overview of the architecture

For the experienced user

Those with experience running containers, installing Python modules, etc., please refer to the below quick start

  • Capture the data from the iDRAC with this Python script: link
  • Run InfluxDB with the following settings: link
  • Create a Grafana instance and connect to InfluxDB to visualize the data

For those who prefer step-by-step instructions

To set this up, start with an Ubuntu server VM. The video below goes through all steps to get started from scratch, including installation of:

  • Python virtual environment
  • Python modules
  • Docker
  • InfluxDB
  • Grafana

Summary of all commands

The commands used below are also summarized in this text file for easy copy & paste: link

URL to get all metrics:

https://IDRAC-IP/redfish/v1/SSE?$filter=EventFormatType%20eq%20MetricReport

Setting up the environment

Update and install: 
sudo apt update
sudo apt upgrade -y
sudo apt install python3-venv python3-pip jq -y

Create a virtual environment:
python3 -m venv NAME-OF-ENV
source ./NAME-OF-ENV/bin/activate

Download the repositories from GitHub:
git clone https://github.com/jonas-werner/idrac9-telemetry-streaming.git
git clone https://github.com/dell/iDRAC-Telemetry-Scripting.git

Install the Python modules:
cd idrac9-telemetry-streaming
pip3 install -r requirements.txt

Command for viewing the JSON data:
cat aaa | sed 's/\x27/"/g' | jq

Installing Docker

Installing prerequisite packages:
sudo apt install apt-transport-https ca-certificates curl software-properties-common -y

Adding the key for Docker-CE:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Adding the repository for Docker-CE
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu eoan stable"

Installing Docker-CE
sudo apt update
sudo apt install docker-ce -y

Adding user to docker group: 
sudo usermod -aG docker ${USER}

Installation and commands for InfluxDB

Download the container image:
docker pull influxdb

Run the image, create DB and add credentials:
docker run \
-d \
--name influxdb \
-p 8086:8086 \
-e INFLUXDB_DB=telemetry \
-e INFLUXDB_ADMIN_USER=root \
-e INFLUXDB_ADMIN_PASSWORD=pass \
-e INFLUXDB_HTTP_AUTH_ENABLED=true \
influxdb

View data in the container using the "influx" client:
docker exec -it influxdb influx -username root -password pass

Commands for the "influx" client:
show databases
use DB_NAME
show measurements
select * from MEASUREMENT
show field keys from MEASUREMENT
drop measurement MEASUREMENT **DELETES THE DATA**

Downloading and running Grafana

Download the container image:
docker pull grafana/grafana

Run the Grafana instance:
docker run -d --name=grafana -p 3000:3000 grafana/grafana

Minecraft ore lamp

This is a 3D printed lamp in the shape of a Minecraft ore block. Powered by an Arduino Uno and four Neopixel LEDs. Model from Thingiverse: link

The button on top will step through different ore types:

  • Redstone
  • Emerald
  • Gold
  • Lapis Lazuli
  • Diamond

A final button push will turn it off.

The design on Thingiverse only contained code for RGB LEDs. In case somebody wants to use Neopixel (WS2812B) LEDs instead, the code for this project has been uploaded to Github: https://github.com/jonas-werner/minecraft-ore-lamp

Since the Neopixel LED’s are very bright I made a dome-shaped diffuser of transparent PLA. The sides are made of painted white PLA (brand: Priline) and the “windows” are transparent PLA. To give it some stability and to easily be able to mount / remove from the Arduino I used a prototype board to create a “hat” for the Arduino. That way it can easily be detached if required.

Diagram

Some more pics of the finished cube