sabato 19 novembre 2016

Docker command list

Docker is an open source project for the management of software containers that contain complete ecosystems to run software, with the main goal of isolating software components and simplify their deployment.
Accordingly, Docker provides a virtualization over the Linux kernel without the overhead of running virtual machines. This means that non-Linux systems run a Linux-based virtual machine Docker to run Docker containers. Docker exploits kernel namespaces and the cgroups to isolate and virtualize system resources for a collection of processes (e.g. process tree, CPU, network and mounted file systems), as well as union mounting which allows for the combination of separate directories into a one that is managed individually. This provides with a lightweight self-contained environment where software artifacts and their dependencies can be deployed. 

A docker file is a text file that describes the steps that Docker has to undertake in order to prepare a Docker image. This includes for instance the installation and setup of third party packages and libraries, the creation of the runtime environment (e.g, variables, filesystem folders). An image can then be created from a Dockerfile using the command docker build. See this page for further information for the creation of a Dockerfile and Docker images.
A docker image is an immutable template defining a snapshot of execution environment in terms of parameters and filesystem, while a container is the running instantiation of an image.
The Docker engine translates the image to a running execution environment, by initializing requested resources (e.g., CPU, memory) and starting specified processes inside it.
This means that after stopping the container changes are lost, unless the container state is saved (i.e. committed) to an image.

We can run a container from a local or remote Docker image as follows:
docker run -i -t <containerImageName> /bin/bash

with -it starting the container in interactive mode providing a terminal. Alternatively, -d can be used to run the container as a background service. In this case, it is possible to later start an interactive session by using the docker attach <containerID> command. However, this connects to the same session used to start the processes in the container. To actually start a new shell on the container the exec command can be used (i.e., docker exec -it <containerId> bash). The container can be given a name, for instance --name <cname>.  Using the --link and -p options, the container can be also started so as to reach the ports exposed by another container. The commands docker inspect and docker port can be used to list the port mappings of the container.
In addition, the -v option can be use to share a host folder as a specific folder on the Docker container. For instance, -v "$PWD":/current shares the current work directory (i.e., pwd) as the /current folder on the container.

Alternatively, data can be moved between a container and its host with:
docker cp <containerName>:/root/file.extension .
docker cp file.extension <containerName>:/root/file.extension

with the first copying from the host to the current work directory, and the second copying from the host to the container.

The processes running on a specific container can be monitored with:
docker top <containerId>

The process logs can instead be visualized using:
docker logs <containerId>
docker logs -f <containerId>

with f returning a continuous log stream.  While this works well, in practice for several containers a solution like logspouts ( is preferable to collect logs on a log server.

Similarly, resource usage of the container can be retrieved with:
docker stats <containerId>

which provides a CLI stream that can be closed with a Ctrl-C.
The container also exposes a REST api which can be polled by any HTTP client.

The container can then be gracefully stopped (i.e, SIGTERM) with:
docker stop <containerId>

Alternatively, the container can be interrupted (i.e., SIGKILL) with:
docker killed <containerId>

All available containers can be listed with:
docker ps -a

List only the running Docker containers:
docker ps

List latest created containers:
docker ps -l

To show the changes made over a container:
docker diff <containerId>

The Docker container can be committed to a Docker image to possibly restore the state in future:
docker commit <containerId> <containerImageName>

Similarly, a container can be exported to a tarball as follows:
docker export <containerId> > tarball.tar

The tarball can consequently be imported with:
docker import <containerImageName> < tarball.tar

Alternatively, the container can be deleted with:
docker rm <containerId>

The following commands remove respectively all containers and all images available:

docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
The list of Docker images available locally can be retrieved with:
docker images

An image can be looked on the Docker hub with:
docker search <containerImageName>

An image can be downloaded from the hub with:
docker pull <containerImageName>

A Docker image can be deleted from disk as follows:
docker rmi <imageId>

The list of Docker images available locally can be retrieved with:
docker images

An image can also be renamed and associated to tags (e.g., version) with:
docker tag <repository>:<currentTag> <newRepository>
docker tag <repository>:<currentTag> <newRepository>:<newTag>

To list all available commands
docker help

Docker provides a handy tool to ease the management of distributed systems.

Moreover, the introduced commands can be easily integrated into a continuous-integration (CI) / continuous-deployment (CD) pipeline, thus providing a complete solution for the setup, deployment and monitoring of distributed applications.

mercoledì 30 dicembre 2015

German Battery Maker Launches Clean Energy Trading | MIT Technology Review

"The German company Sonnenbatterie has launched a trading platform for distributed renewable energy by offering a way for owners of small solar and wind generation capacity to buy and sell power across the utility grid." More here

domenica 16 agosto 2015

Bridging Local-Area Networks (LAN) across buildings

Lately I had the need for bridging a second building placed 80/100 m away, in order to share the internet connection and interface remote cameras.

The CPE mounted on a TV Antenna
The best quality-price solution I have found is the Ubiquity Nanostation M serie. Main products are the M2 and M5, which differ for their operating frequency, 2.4GHz and 5 GHz respectively. In particular, I opted for the "loco" version which is slightly less powerful and cheaper than the full M2.

A quick installation intro is available here and here. What's required is basically to configure one of the CPEs as Access Point, which creates a new IEEE 802.11ac wireless network which can be secured with a passkey. It's also important to correctly refer to the gateway in the network mask.

Similarly, the other CPE can be set as station to connect to the network just created. That's it.

I am using a switch to connect to the station. Apparently the CPE also forwards DHCP requests, which allows the gateway to seamlessly distribute IP addresses back to the second network.

Stepwise guides are available here and here.

So far so good, although I am curious to see how it behaves under bad weather conditions, especially for prolonged time. The case does not really look that robust for outdoor applications, still I am quite satisfied with the performance of the whole deployment.

lunedì 27 luglio 2015

Mjölnir - the open source energy advisor

We recently announced the release of the stable version 0.2 of our open source energy management system Mjölnir at

While the tool targeted so far mostly "disaggregated" device-level energy and power usage, we have recently introduced full support for circuit-level measurements (buildings, rooms) which unlocks a considerable potential for further data analysis.
The DIN-RAIL module running the measurement system

As most of energy meters use the industrial automation system ModBus, we have been looking for possible shields to extend our open hardware solution with RS485 communication. We finally selected this RPi hat from Libelium while the meter is the Carlo Gavazzi EM24. The implementation is eased by the Libelium ArduPi library, which makes the C code written for Arduino compatible with the Raspberry Pi. The data is then being sent to our servers through a REST interface.

The overall component shown on the picture is therefore a low cost solution able to retrieve remote measurements via the RS485/ModBus and the USB/ZigBee network. This opens for the future integration of other measurement units, such as water and gas meters.

The support of circuit-level measurements required changes on the Mjölnir system.

The system is now organised in buildings, rooms and devices. A circuit is described by its ID and can be associated to a single building or room.

The just introduced device control is implemented using web sockets, so that both the dashboard and the gateway can keep a TCP connection open to publish/subscribe for events. This seemed the lightest and easiest-to-develop alternative to XMPP, RabbitMQ (AMQP) and MQTT.

Below is a video providing a walk through the system.

As usual, the code of the gateway is available on source forge, along with the dashboard system.

  1. A. Monacchi, F. Versolatto, M. Herold, D. Egarter, A. M. Tonello, and W. Elmenreich. An open solution to provide personalized feedback for building energy management. ArXiv preprint arXiv:1505.01311, 2015.

lunedì 13 luglio 2015

Using web socket to manage remote devices

The advent of web sockets opened to full-duplex communication channels between browsers and web servers. Being based on the port 80. this has also the benefit to bypass firewall policies. Perfect application for this technology are real-time messaging, such as multiplayer games and chats. 

We discuss hereby an example of a client and server to remotely control IoT devices based on web sockets. The server uses jdbc to connect to an internal MySQL database for authentication/authorization, the simplejson java library to format messages, as well as the 1.3.0 library for its networking aspects.

class Server extends WebSocketServer{
private java.sql.Connection dbConn;
  private Map< WebSocket, String> authenticatedConnections;
  private Map<String, ArrayList<WebSocket>> subscriptionControllers;
  private Map<String, ArrayList<WebSocket>> subscriptionDashboards;
  public Server(int port) throws UnknownHostException {
        super(new InetSocketAddress(port));
  authenticatedConnections = new HashMap< WebSocket, String>();
  subscriptionControllers = new HashMap<String, ArrayList<WebSocket>>();
  subscriptionDashboards = new HashMap<String, ArrayList<WebSocket>>();
  public void setDBMSCredentials(String host, int port, String user, String passwd, String db){
  try {
  dbConn = DriverManager.getConnection("jdbc:" + "mysql" + "://" + host + "/" + db , user, passwd);
  System.out.println("Connected to database "+db+" at "+host);
  } catch (SQLException e) {
  public void onOpen(WebSocket conn, ClientHandshake arg1) {
  System.out.println( conn.getRemoteSocketAddress().getAddress().getHostAddress() + " connected!" );
  public void onClose(WebSocket conn, int arg1, String arg2, boolean arg3) {
  System.out.println( conn.getRemoteSocketAddress().getAddress().getHostAddress() + " has disconnected!" );
  // remove the connection from both the authenticated and the subscribed ones if available
  String authkey = authenticatedConnections.get(conn);
  // remove also from the subscriptions
  subscriptionDashboards.get(authkey).remove(conn); // remove the subscription if available
                subscriptionControllers.get(authkey).remove(conn); // remove the subscription if available

  public void onError(WebSocket conn, Exception arg1) {
  System.out.println("Error while interacting with "+conn.getRemoteSocketAddress().getAddress().getHostAddress()+":\n\t"+arg1.getMessage());

  public void onMessage(WebSocket conn, String s) {
  JSONParser parser = new JSONParser();
  try {
  JSONObject obj = (JSONObject) parser.parse(s);
  // check authentication key
  String authkey = (String) obj.get("authkey");
  String type = (String) obj.get("type");
  String action = (String) obj.get("action");
  // authenticate the user first
  if(authenticatedConnections.containsKey(conn) ||
  // check if the connection was already authenticated
                    authenticatedConnections.put(conn, authkey);
  // check the operation type
  case "device_status":
  System.out.println(authkey+": received "+action+" device_status from "
// check if an entry exists for the given key
                                subscriptionDashboards.put(authkey, new ArrayList<WebSocket>());
                                // avoid get on missing key
  // avoid multiple entries for the same connection
System.out.println("\tAdding "+authkey+" to subDash");
}else{ // publish
                                this.sendAll(s, subscriptionDashboards.get(authkey));
case "device_control":
System.out.println(authkey+": received "+action+" device_control from "
// check if an entry exists for the given key
                                subscriptionControllers.put(authkey, new ArrayList<WebSocket>());
  // avoid get on missing key
  // avoid multiple entries for the same connection
System.out.println("\tAdding "+authkey+" to subContr");
}else{ // publish
                                this.sendAll(s, subscriptionControllers.get(authkey));
} catch (ParseException e) {
// out of protocol format
                               +": out of protocol format!");
public void sendAll(String message, ArrayList<WebSocket> conns){
Iterator<WebSocket> i = conns.iterator();
WebSocket s =;
s.send( message );
System.out.println("\tRouting to "

public boolean authenticateUser(String authkey){
boolean auth = false;
if(dbConn != null){
try {
Statement stmt = dbConn.createStatement();
// prevent sql injection by using a prepared statement
java.sql.PreparedStatement ps = dbConn.prepareStatement("SELECT * FROM advisor.user WHERE user.authkey = ?");
                ps.setString(1, authkey);
ResultSet rs = ps.executeQuery();
auth = true;
} catch (SQLException e) {
return auth;

The server main function might look like this then:

public static void main(String[] args) throws IOException, TimeoutException,     
                    ShutdownSignalException, ConsumerCancelledException, InterruptedException  {
    DeviceManager dm = new DeviceManager();
    Server srv = Server(8887);

    srv.setDBMSCredentials("localhost", 3306, "root", "", "advisor");
    System.out.println("Websocket server started");

We can now test the server by subscribing for control events on one side and publishing control events on the other one. Let's do this in Python for simplicity.

import threading
import sys, os
from threading import Timer
from websocket import create_connection, WebSocketConnectionClosedException
import json
from plugwise import Stick, Circle, TimeoutException
import serial = create_connection(self._control_interface)
print "connection with remote server established!"
# subscribe to control events for the given authentication key{'authkey':'authkey-for-user',
                         'action': 'subscribe',
print "successfully subscribed to device control events!"
# start a listener for control events

This basically creates a connection to the server, sends a subscribe for 'device_control' events and spawns a thread listening for incoming events, which is implemented as follows:

def listen_device_control_events(self):
        while not self._exit: # listen for incoming commands until the script is voluntarily terminated
                received = json.loads( )
                if 'type' in received.keys() and received['type'] == 'device_control': # the 'is' is for object comparison
                    print 'received control event to ', received['payload']['state'], 'for', received['payload']['device_id']
                    self.set_circle(received['payload']['device_id'], int(received['payload']['state']))
            except ValueError as e:
                print e
            except WebSocketConnectionClosedException:
                print "the server closed the socket"

Upon reception of control events, the code simply calls the method set_circle (defined in the python plug wise library) to control the switch status of connected ZigBee wireless nodes.

Now a device_status event can be published each time a device becomes available in the network, by simply calling the following function:

def update_circle_status(self, circle_id, status):{'authkey':self._settings['apikey'],

For instance, an event can be sent when initiating all ZigBee circles as follows:

self.circles = {}
    for c in self._settings['circles']:
        self.circles[c] = Circle(c, self.stick)
        self.update_circle_status(c, 1) # send all connected circles as available

Here is a simple implementation for the controlling client, which simply sends control events based on user's input:

from websocket import create_connection, WebSocketConnectionClosedException
import json
import threading

ws = create_connection("ws://")

while True:
    input = raw_input("Insert status of device")      

'action': 'publish',

The code was integrated into the Mjölnir gateway to provide an event mechanism for remote device control.

Authenticating and authorising users in RabbitMQ

RabbitMQ is a messaging broker, basically an infrastructure providing message queues to which applications can push and pull data messages. Such time asynchrony provides a decoupling between data producers and consumers, as well as interoperability between clients running on different machines and technologies. In particular RabbitMQ implements the Advanced Message Queuing Protocol (AMQP), a lightweight application-level binary protocol based on TCP [1]. AMQP together with the MQTT, XMPP and CoAP, is emerging as the leading machine-to-machine application protocol for internet-of-things applications [2]. An important aspect of RabbitMQ is also the reliability of its message queues (e.g. persistence of messages, retransmission, etc.) as well as its scalability over clusters of computers [3]. Its plugin mechanism also allows for connecting the broker to others written in different protocols, such as MQTT. A relevant issue in M2M application protocols is, however, authentication and authorisation management. This is mainly due to the fact that those protocols originated from the WSN community, where the network was normally managed by the same corporate or individual. 

A basic way is to add users and define their privileges using the rabbitmqctl command line interface, or the rabbitmq http API [4]. Accordingly, access control is provided by RabbitMQ using an internal database, which is initialized to a guest user, entitled to login only from localhost. 

The main alternative we discuss here is the use of an external authentication server, based on a RESTful interface. In particular, we use the rabbitmq_auth_backend_http plugin available at [5]. 
As also shown in [6], the plugin can be simply downloaded with a WGET from the community plugins [7], placed directly in the folder rabbitmq/plugins, and enabled with ./sbin/rabbitmq-plugins enable rabbitmq_auth_backend_http. The next step is to configure the plugin to connect to our authentication server. To this end we need to modify the ./etc/rabbitmq.config file which should look like this:

  {rabbit, [{auth_backends, [rabbit_auth_backend_http]}]},
   [{user_path,     "http://www.domainname/rmq_auth.php"},
    {vhost_path,    "http://www.domainname/rmq_auth.php"},
    {resource_path, "http://www.domainname/rmq_auth.php"}]}

with the first line enabling the auth_http backend and the second one properly configuring the goal server. As shown in [5], the tool will mainly perform 3 calls, user_path to authenticate a user, vhost_path to grant access to a certain vhost and resource_path to grant access to a specific resource, such as a queue or an exchange. We provide below a very simple script, granting access to the default '/' vhost to all users registered in our MySQL db. In particular, we avoid sending username and password in favour of a hashed token (authentication_key). For simplicity, we omit here the connection detail and the "SELECT * from user WHERE authentication_key = %s" query implemented in the getUser function.


// Authentication script for RabbitMQ

 if(isset($_GET['username']) ){ 

$authkey = $con->escape($_GET["username"]);

// first of all check if the user exists
$user = $con->getUser($authkey);

  if($user == -1) {
echo "deny";
  // go ahead with checking
  if( isset($_GET['password']) ){
  // AUTHENTICATION mechanism
  // having the authkey is enough, allow the user
  // USER
  echo "allow management";
  }else if( isset($_GET['name']) ){ 
  // AUTHORIZATION mechanism
  // both vhost and resource are set (RESOURCE)
  // a certain user can only access it's specific resource
$resource_name = $con->escape($_GET["name"]);

$valid_names = array($user, $user."_device_control", $user."_device_status");

// a user can only access 2 specific queues
if( in_array($resource_name, $valid_names) ){
echo "allow";
} else{
echo "deny";

//file_put_contents('filename.php', $resource_name);
  }else if( isset($_GET['vhost']) ){ 
  // only vhost is set (VHOST)
  echo "allow";
// for simplicity we only have '/' as unique host
  echo "deny";

  echo "deny"; // deny access to the user


The example Python code shown on the RabbitMQ site would now look for the sender like:

#!/usr/bin/env python
import pika

credentials = pika.PlainCredentials('authenticationkey-of-administrator', 'test')
parameters = pika.ConnectionParameters(credentials=credentials, host="", virtual_host='/')
connection = pika.BlockingConnection(parameters)

channel =

channel.exchange_declare(exchange='Administrator', type='direct')

                      body='Hello World!')
print " [x] Sent 'Hello World!'"


and for the receiver:

#!/usr/bin/env python
import pika
from pika.exceptions import ProbableAccessDeniedError, ProbableAuthenticationError

def callback(ch, method, properties, body):
    print " [x] Received %r" % (body,)

credentials = pika.PlainCredentials('authenticationkey-of-administrator', '')
parameters = pika.ConnectionParameters(credentials=credentials, host="", virtual_host='/')
connection = pika.BlockingConnection(parameters)

channel =
channel.exchange_declare(exchange='Administrator', type='direct')

queue_name = 'Administrator_device_status'
result = channel.queue_declare(queue=queue_name)
channel.queue_bind(exchange='Administrator', queue=queue_name, routing_key='device_status')

print ' [*] Waiting for messages. To exit press CTRL+C'

print 'Terminate'

Which basically creates an exchange called Administrator, with a queue named Administrator_device_status accessible solely by the administrator user. Such resource is therefore replicated for each user. For instance, this could allow users to remotely control their IoT devices.