Author Archives: Marc

m2ag.labs iot framework reloaded

I realize it has been quite sometime since my last post detailing how to install the m2ag.labs iot framework. After that post I looked over my progress and goals and did research in to existing projects. I came across Mozillia’s IoT project and was happy to find something I can work with.

Part of my goal was to provide for automatic ui configuration of common controls, easy web access and standard way of defining IoT devices. Mozilla’s project does that and more while supporting proposed w3c standards. This covers a large part of my development effort so I am definitely adopting the webthings and gateway for my project.

I still have the management interface to workout, including a point and click configuration mechanism and easy certificate generation and installation. These will be my milestone 1 goals. I will also be developing examples and documentation for the project.

The new architecture will use webthings as a standard

m2ag.lab iot framework

Installing the pre-alpha m2ag.labs IoT framework

The m2ag.labs IoT framework currently targets the Raspberry PI and Raspbian. The framework is in pre-alpha but us useable with custom modules installed. Custom modules will be the topic of next weeks post, along with some example code.

The usual warnings apply — this framework may not be compatible with the alpha or later versions of the framework. Some the of the internals need some work, but the framework can be made to work for you now.

Steps:

  1. Install Raspbian Lite
  2. Perform initial start up
  3. Configure WIFI, SSH, i2c,  change password and hostname
  4. Reboot and login via SSH
  5. Update OS via apt
  6. Generate and place X509 certificates
  7. Get  installer and run

Install Raspian:

Use the Raspberry PI imager  to write Raspbian Lite to your sd card. I use 32 gb Samsung Evo cards. There is direction for choosing a micro sd at the Raspberry PI site.

I use Raspbian Lite unless there is a need for desktop UI on the device. Generally the stuff I do runs headless and is controlled via a web app.

After the image is written do the initial start up for the PI. The easiest way is to add a ssh file to the boot directory of the image we just created and connect the device to an ethernet hub. Just ssh to pi@raspberrypy.local and go. Another way is to plug in a monitor and keyboard but I am too lazy to go get one. The RaspberryPI  site has  instructions on setting up headless with wifi but I haven’t had much luck getting this to work lately (the wifi won’t start) .

After we get logged in (pi/raspberry) we need to configure wifi and enable ssh (if we haven’t already) It’s a good idea to go a head a change:

  •  the default password
  • the time zone
  • enable i2c

Next –  install the X509 certificates. This has been discussed previously on m2aglabs.com. I haven’t decided if I will add a configuration option to run without ssh. I’m not thinking it is a good idea. But — ssh can be turned off with a little manual file editing. For the device, it is in the file device/comm/comm.py. The logger is just one file. For the client look in device/api/static/js/comm.js. I don’t think it is a good idea to not use ssh, even in a controlled network.

To create and deploy the certificates follow the instructions in this link or use the tool at  https://github.com/dakshshah96/local-cert-generator to generate the certs. Remember — the root CA has to be imported to each OS that wants to access the IoT device securely. The blog post at m2aglabs.com contains links to instructions for the most common systems. I use Mac, IOS and Linux around here but Windows and Android are all good.

After the certificates are generated place them in the pi users home directory in a .certs subdirectory. Both the Flask app and Mosquitto MQTT server will access them from here. We are looking for the filenames server.crt and server.key.

Next – cd to home/pi and get the install.sh with wget :

wget https://raw.githubusercontent.com/m2ag-labs/m2ag-iot-installer/master/install.sh
chmod +x install.sh
/install.sh

This script does apt update and upgrade first, then the install. It can take quite a while to complete. When the script is complete the system should be installed and running. Check the script results for errors. Then navigate to your device on port 5000 (raspib.local:5000) and get the client page opened (be sure to have imported the root CA so ssl will work.

You will get a warning about no user name password. Click on ‘credentials’ in the upper right and enter the default username/password (pi/raspberry).

provide default credentials (pi/raspberry) and click ‘close’

The device will use the hostname as a login for mqtt, default password is raspberry. When we get the page and the login we can see that the api app is running and ssl is correctly recognized by the browser.

Click on ‘Get Device’ list to populate the list. You should see a listing for the device that you can click on:

Select the device to proceed.

This selects the device setup us up for an mqtt connection to the IoT core of or device. Commands can be sent via the Communications tab:

command to sound a sample alert

If you have a piezo buzzer attached to pin 13 the preceding command will make it play a tone. In the command — the prm object can carry your custom command to your IoT device. Returned data will be available in the output area. I have a post here that details how I set my buzzer up.

Limited DB access via Database tab,

registry and users are currently implemented.

The code is pre-alpha so there are plenty of warts. There is a problem with the HTML5 app that make the initial mqtt connection a little contrary but it can be made to work. Just try sending a command a couple of times if you can’t get the initial command to work. Much work needs to be done to HTML5 app ( would you like to help? ).

The UI and accompanying API are going to get some usability improvements next, followed by some work on the IoT device.

The repos are here.

If you are working on COVID related projects and would like to use this framework to boot strap your effort I would be happy to assist in getting you going. Contact me here.

If you find this work helpful perhaps you would consider supporting m2ag.labs open source efforts by buying us coffee. Any amount would be appreciated. Please use this link to do so:

Small Donate Button

Thanks for stopping by.

m2ag.labs iot framework block diagram

m2ag.labs IoT framework

The m2ag.labs IoT framework is a collection of techniques for running very capable IoT devices completely disconnected from any specific cloud provider. The framework, currently in pre-alpha, is an moderately opinionated implementation that seeks to cover the basics of communication, security and configuration of most IoT devices needs. The implementor will have a consistent framework with which to start projects with and only need to focus on the IoT functionality to be implemented.

The framework itself is intended to be used inside a controlled firewall with no access to the internet. Because of this the self signed X509 certificates are used to provide SSL and MQTTS connectivity for the device.

The framework consists of several distinct parts:

  • API
  • UI
  • Action Engine
  • Logger
  • Device
  • COMM module
  • Device manager
  • Hardware interface
  • Multiple component modules to implement functionality

Installer:

A basic install script will ensure required elements are installed and put in a basic configuration. Initially there will be some configuration that is manually completed but as the framework matures a more complex installation and configuration system will be implemented.

API:

A REST API to allow access to logs database and configuration pages for the device. In general configuration is set through this API.

The API includes the device database which serves as the persistent memory for the device as well as configuration registers.

UI:

HTML5 based app to provide default access to both the config API and the Device. If device is a HUB, will coordinate a fleet of devices.

Initially just basic connectivity will be implemented. As the framework matures preconfigured modules will be used to allow basic UI of common components to displayed.

Device:

A device is the implementation of the IoT functionality. As the python app service is concerned – it manages MQTT comm, configuration changes and general live activities the specific application requires. Monitor a temperature or pressure and send an alarm. Did a position sensor change — send a message. Basic IoT stuff. Other aspects of the device implementation – the support services – are basic Linux/Raspberry PI configurations. The non-python functions of the device will require separate setup steps. These steps will be documented in the installer.

The device coordinates:

COMM module:

Is a generic interface to allow MQTT access. MQTTs is the mechanism for interacting with the IoT functionality.

Each device may be its own MQTT server, or it may connect to a central MQTT server so that events can be aggregated.  The architecture supports connecting to cloud services in a on your own terms manner. The implementor has complete control of what data gets sent to the cloud, if any.

Device manager:

The device manager coordinates activity between the COMM, API and Hardware. Events are monitored and generated here (TBD a better location — probably on the logger service)

Hardware manager:

Handles command processing amongst the python monitored modules. This module contains a list of submodules that comprise the actual implementation of functionality. An object representing the exposed functionality of a module can be addressed in a simple yet expressive manner that can cause any state change that may be required for the implementation.

Component Modules:

 Generally the object will wrap an hardware item – like a bmp280. The framework will generally wrap an Adafruit Circuit Python library for the module with helpers to facilitate standardized data transfer over the mqtt interface.

Component modules may aggregate components to abstract an more complicated component. The comm framework strives to be agnostic to message traffic.

In general — gpizero will be wrapped to implement desired functionality. One of the framework goals is to be as simple as possible to allow easy adoption of the framework with a minimum of technical ability. You won’t need to necessarily be a software engineer to be able to make use of the framework.

Action Engine:

A separate process that monitors MQTT topics for programmed events. Can collect data and trigger actions. This process also logs the MQTT traffic. Each device in a fleet can run it’s own action engine, or they can defer to a central point.

Logger:

A  logger task monitors MQTT traffic to gather telemetry about the IoT system. Will aggregate traffic and log to a SQLite database of the hub.  This is a separate python app that is running as a service on the hub device (a solo device is its own hub).

The idea is to provide a standardized implementation of the common aspects of getting a system running. Control and monitoring, combining components and events can be coordinated in a easy to implement pattern. 

An installer process will do the preliminary install. This will get the api, logger and device services installed as and enabled as well as getting mosquitto mqtt, circuit python, etc.

Some critical tasks — like generating and staging X509 certificates need to take place before install. The frame work strives to be reasonably secure without too much effort.

Alpha versions of the framework will be available in the upcoming weeks.

If you find this work helpful perhaps you would consider supporting m2ag.labs open source efforts by buying us coffee. Any amount would be appreciated. Please use this link to do so:

Small Donate Button

Thanks for stopping by.

Embedded audio alerts with gpizero and simple circuit

For the devices I tend to make the Raspberry PI is intended to be a controller or sensor of some sort that does not usually have a visual interface attached. It is nice to have the device setup so that all critical functions are easily accessible, and the device be able to give indications of type. Certainly LEDS are easy to implement on the PI. A simple transistor allows us to interface a wide assortment of devices. In this case I want to interface a piezo buzzer to provide audible alerts for device activity.

Now granted, I could attach a plain speaker to the appropriate port and play audio files. There is a lot of overhead to doing that and it would be way overkill for what I am trying to accomplish. Generally I am looking for a beep at power up, maybe a tone when the network is down, an alarm for a sensor – you know.

A simple circuit is all you need –

A few pennies worth of discrete components implements the buzzer

6.8 K worked for me, but you should match R1 to the transistor you are using. This makes for an audible tone that is not too loud. If you need a reference to calculate R1 this is a good source: https://www.electronics-tutorials.ws/transistor/tran_4.html

Python code to implement tones could not be simpler. Using gpiozero‘s buzzer can give basic beep or tonal buzzer can be used to create compelling tones. Pwmled can also be used to give an attention getting alert.

from gpiozero import PWMLED
import time

buzzer = PWMLED(13)
buzzer.pulse(fade_in_time=0.5, fade_out_time=0.2, n=10, background=True)
time.sleep(10)

Vary the fade_in_time and fade_out_time to experiment with tones. I’ll be using some variation of this implementation in the upcoming m2ag.labs IoT framework for general device side alerting. As I develop tones I will add them the the m2ag.labs example repo. I’d be happy to include your tones of you create a pull request.

If you find this work helpful perhaps you would consider supporting m2ag.labs open source efforts by buying us coffee. Any amount would be appreciated. Please use this link to do so:

Small Donate Button

Thanks for stopping by.

a simple circuit to implement a run indicator and shutdown button

Raspberry PI running indicator and shutdown button

One of the things I want for my IoT devices is a smart power button. When off or in low power I would like to be able to press the button and have the device power on. If the device is on I want to press the power button and have the device shutdown safely and either power off or indicate that power can be shut off.

The PI has built in support for the seed of my solution in the form of device tree overlays  (dto) that can activate kernel support for a shut down signal and a powered down signal. I will be using these two features to implement a simple shutdown button and running indicator led. Not quite the feature set I am looking for, but this may be useful to pull out for some circumstances. Besides, if I can control a led at shut down I could send a signal to put the PI in low power or have it’s UPS shutdown.

There are many sources on the web that detail these two dto’s . The two I found most helpful were:

shutdown signal:
https://www.raspberrypi.org/forums/viewtopic.php?t=217442
 
toggling gpio at shutdown:
https://www.raspberrypi.org/forums/viewtopic.php?t=223157

It should be noted that specifying the gpio pin to use is different in these two overlays. My implementation looks like this (in /boot/config.txt):

dtoverlay=gpio-shutdown,gpio_pin=17,active_low=1,gpio_pull=up
dtoverlay=gpio-poweroff,gpiopin=19,active_low=1

With this configuration I am monitoring pin 17 for a low signal to initiate system shut down. When the system has halted, pin 19 will go low. Pin 19 will go from low to high on startup.

When pin 19 signals shut down there are two low to high transitions that look like this:

the has shut down signal toggles low to high once before staying low
The has shut down signal toggles low to high once before staying low

So — using this line to control a running led would have the led on while running. When the system shuts down the led will blink once and then go out indicating it is safe to power off. Keeping the led lit adds 10 ma to the measured current draw of the PI. If this is a concern the logic could be reversed and the led could be made to illuminate at system shutdown. I’m going to be using the led as a running indicator, on while running.

These simple circuits are all that we need to implement our running indicator and shutdown switch.

a simple circuit to implement a run indicator and shutdown button
For a dollars worth of parts we can add some helpful features.

In case it is not clear — those upside diamonds are grounds at the bottom of the schematic. I like the other style ground symbols better.

D1 is a generic led I had on my desk. R1 is 500 ohms because I have so many of them. R3 and C1 are for switch debounce. You can leave these out if you want. I added them because my switch was glitchy when activated:

Without r3/c1:

glitchy shutdown signal
With out R3/C1 my shutdown signals usually look like this

With r3/c1

glitch free shutdown signal
With r3/c1 installed the shutdown signal is glitch free

And there you have it. With a simple file edit and circuit we now have a shutdown switch and a running indicator for our PI. We no longer have to log in and issue a shutdown command and have to assume it is safe to power off our PI. This is a good first step, but it would be handy if we could get this to be a smart power switch. We will be looking at this in an up coming post.

These links are relevant for what was done here:

These links are relevant for what was done here:
 
https://www.onsemi.com/pub/Collateral/2N3903-D.PDF
 
https://www.thegeekpub.com/246471/debouncing-a-switch-in-hardware-or-software/
 
https://hackaday.com/2015/12/09/embed-with-elliot-debounce-your-noisy-buttons-part-i/
 
https://www.electronics-tutorials.ws/transistor/tran_4.html
 
Logic captures by the awesome Saleae logic analyzer

If you find this work helpful perhaps you would consider supporting m2ag.labs open source efforts by buying us coffee. Any amount would be appreciated. Please use this link to do so:

Small Donate Button

Thanks for stopping by.

sqlite.remote version 1.0

March 22, 2020

A while back I released some simple code to allow access to a SQLite database on my embedded devices. A python/html5 app that provided an HTML5 interface to a python api. My goal is a simple to use method that allows quick and easy access to  databases on raspberry pi (or similar) used for configuration and logging on my IoT devices. The requirements are minimal :

  1. User login via http auth
  2. The ability to change a user password.
  3. SSL via self signed certificates
  4. Basic database access via a browser based app

Today version 1.0 of the sqlite.remote tool is being released. Available via git hub, the app meets the goals specified above.

Some assembly is required — mainly in the form of generating self signed certificates for the devices and browsers that will be used to access them. Check out my previous post about securing local IoT devices.  Generate the certs that you need before installing the sqlite.remote tool.

Installation:

Since I use this on the Raspberry PI it is installed in the pi user’s home directory. Any system that supports python should work, as should any user.

First clone the git hub repository:

 git clone https://github.com/m2ag-labs/m2ag-sqlite-remote.git

Then cd to the install directory and chmod the installer script:

chmod +x install.sh

Then ./install.sh to install dependencies. This app requires flask and flask_httpauth. Depending on your current update status the dependencies may already be installed.

After the install is completed the install can be tested by changing to the project root and running:

python3  api.py sqlite-remote.sqlite

This should start the api server and show a  message like this in the console:

As you can see we are using the barebones development server to run our api. Despite the message to not use this in production, I am using this in production. My usage only calls for one or two users at a time infrequently hitting the api. I decided to trade a simple implementation for a more robust web server and more complicated install. Plus, the web server would generally sit idle for day to day use.

It should also be noted that the api is being served over https. This is requires a self signed certificate to be installed on the device. This is configured at the bottom of api.py:

Adjust the paths to the correct certificate files. The api can be run without ssl if desired. For me, I want to provide at least a minimal level of security to my IoT devices. I intend this to be a config and control interface, but the devices will generally be inside a controlled firewall. Self signed certs  and a basic login meets my needs.

As part of the install a service a file is and copied to the systemd system directory. If the default location (/home/pi/m2ag-sqlite-remote) is changed, change it in this file.  To enable the service:

sudo systemctl m2ag-sqlite-remote enable
and 
sudo systemctl m2ag-sqlite-remote start

After the api is running navigate to your device at:

https://your-device.local:5001  to get the app page:

The default user is “pi” and the password is “raspberry”. These can be set in the credentials popup:

Select close to get back to query screen. If the query button is pressed with an empty query an “Ok” message will be returned. This tells us the configuration is correct:

To change the password use e the credentials popup:

‘Ok’ will appear in the status on a successful change.

To add new users first select * from users and get the hash for pi’s password and then insert a user using the same password hash. You can then use the new user’s credentials and change the password with the set password dialog.

General usage:

This app should handle most updates and edits to tables. Since the services setup specifies the database at startup additional tables to be added to the sqlite-remote.sqlite database. The only requirement for the app is the user table with a text username and password fields. This table could be added to any database. Just insert the pi user (or any other user) into the user tables via the command line app or some other tool.

It is hoped that you find this app useful, please feel free to open issues on github or comment here if there are problems with the app. Keep in mind it is barebones on purpose, all I need it to do is update tables remotely for me. If more complex usage is envisioned it may be modifications will need to be made.

If you find this work helpful perhaps you would consider supporting m2ag.labs open source efforts by buying us coffee. Any amount would be appreciated. Please use this link to do so:

Small Donate Button

Thanks for stopping by.

Securing Local IoT devices

March 13, 2020

The goal of my development efforts lately has been the implementation of an IoT framework for  embedded devices that requires no connection to the cloud. I want a framework that I can use within my firewall that does not need to connect anywhere. I can then allow only the data I want to make it to the cloud.  This makes security a little tricky for me. I intend to use web interfaces and Mosquitto mqtt on the devices, but I still want to ensure secure access to the device. I will be targeting Google Chrome as the web browser

Stack overflow has several discussions on the topic of securing devices on an internal network.

This one is pretty good ->

https://security.stackexchange.com/questions/121163/how-do-i-run-proper-https-on-an-internal-network

When it comes down to it there are only two ways to proceed:

  1. Self signed certificates
  2. Public certificates for internal addresses

The stack overflow discussion mentioned previously can give a overview of some of the options.

For this project’s needs option 2 is not attractive because the need to involve an outside entity for certificate management. Option 2 also requires a more advanced approach to network routing and a hard requirement for at least occasional internet access for certificate validation. It has the benefit of effortless compatibility with web browsers. 

Self signed certificates  can provide a complete disconnect from outside entities if desired and still allow for web interfaces on the IoT devices. The downside is that the self generated root certificate will need to be installed on every device that is used to access the systems via SSL. Steps to do so on each O/S varies but is just a Google search away.

There is an excellent writeup here on how to  generate self signed certificates. The focus is for local software development, but the result generates certificates that will meet the projects needs.

There are two things to keep in mind here. First: server SSL certificates can only have a life span of 2 years. Trying to generate one with a longer life will succeed but Chrome will reject it.  Secondly — the root certificate will need to be imported in each O/S that will be used to access the IoT device. This is actually kind of a benefit as we an control release of the root certificate authority according to our security situation.

Instructions to create a trusted certificate are here:

https://www.freecodecamp.org/news/how-to-get-https-working-on-your-local-development-environment-in-5-minutes-7af615770eec/

https://github.com/dakshshah96/local-cert-generator/

When I set up my certificates I gave the root a life time of 20 years. I don’t want to have to touch a machine more than once. The server certificates had to be set to 2. The certs can be used for the Mosquitto server as well. This site has a good explanation of how to set up a Mosquitto server  for SSL

Managing users:

Each O/S has a specific way this must be done.

Mac O/S:

The original article details how to import the root certificate into the Mac keychain.

Windows/Linux:

This article details the steps: https://thomas-leister.de/en/how-to-import-ca-root-certificate/. The Windows instructions worked with out issue. The Linux instructions are iffy.

I0S:

The root ca must be emailed to an IOS device and imported. This article details the steps –> https://medium.com/collaborne-engineering/self-signed-certificates-in-ios-apps-ff489bf8b96e . These instructions had no issues.

Android:

This site has instructions on how to install self signed certificates. The steps were not tested.

Some other links that may be useful:

Enable SSH for PAHO JavaScript client:

https://stackoverflow.com/questions/53051679/how-can-i-use-tls-with-paho-mqtt-over-javascript. The PAHO client will not use client certificates (which are the same as the sever certificates) and will support SSL only rather than full encryption. To do this in a browser mqtt.js will need to be used. 

Securing raspberry pi:

https://www.raspberrypi.org/documentation/configuration/security.md

https://www.raspberrypi.org/documentation/configuration/