Comfort  Automation/ Security System Forums Home
Home Search search Menu menu Not logged in - Login | Register

Comfort to MQTT
 Moderated by: slychiu
 New Topic   Reply   Printer Friendly 
 Rating:  Rating
AuthorPost
 Posted: Monday Feb 8th, 2016 09:13 pm
   PM  Quote  Reply 
1st Post
cab123
Member
 

Joined: Saturday Oct 3rd, 2009
Location: Coimbra, Portugal
Posts: 25
Status: 
Offline

  back to top

Hi, i would like to share with the community my simple driver implementation of comfort to MQTT.

I have this running for a couple of months and never registered any failure with the communication. MQTT is amazingly light and mature. And my driver seems to be stable enough for the job, had some issues with strange USB resets, but now it gracefully handles every USB quirks i found, including reconnecting the UCM.

This approach provides a simple and flexible way to interact with comfort from/to multiple sources. Personally i have comfort ULTRA, individual power monitoring, solar inverter statistics all feeding data to a MQTT broker. Then I process some of that data in NodeRED and present/control everything in OpenHAB. It works really well.

I do not use the alarm section of comfort, only home automation with push buttons, relays, infrared and some KNX stuff.


Used hardware:
- UCM USB
- A x86 server with Debian Jessie installed (could be a raspberry Pi)



(1)
Connect the UCM/USB to the server and make the USB connection deterministic:

(1.1)
Find the Device / ID / Serial Number of the device:

root@debian:/# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

root@debian:/# lsusb -d 0403:6001 -v | grep iSerial
  iSerial                 3 A7007iyv
 
 

(1.2)
From the output above we get:
- idVendor: 0403
- idProduct: 6001
- Serial Number: A7007iyv

Your data will probabily be the same, except for the unique serial number of your UCM.
Next create a file like the one below and add the right info:

nano /etc/udev/rules.d/99-usb-serial.rules

SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="A7007iyv", SYMLINK+="comfort"


Now everytime the USB UCM is connected th the server it will be uniquely available at /dev/comfort, regardless of other USB peripherals in the system.

(2)
Install MQTT broker and clients


http://mosquitto.org/2013/01/mosquitto-debian-repository/

for the clients:
sudo apt-get install mosquitto-clients

anoher good reference with some testing:
https://pallavichaurasia94.wordpress.com/2014/10/14/mosquitto-in-debian/



(3)
the script to establish a bridge between Comfort and MQTT:

(3.1)
First install the folloing perl modules:

cpan Net::MQTT::Simple
cpan IO::Select
cpan Time::HiRes


(3.2)
Find a appropriate directory, for example /opt and copy the contents of the script to a new file, for example:

nano simple_mqtt_serial_select.pl

#!/usr/bin/perl

use Net::MQTT::Simple;
use IO::Select;
use Time::HiRes qw(time);
use strict;

#############################################################
###                                                by:APR ###
###                                                       ###
### Perform the following system commands:                ###
### lsusb                                                 ###
### lsusb -d 0403:6001 -v | grep iSerial                  ###
### nano /etc/udev/rules.d/99-usb-serial.rules            ###
### SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", ATTRS{serial}=="A7007iyv", SYMLINK+="comfort"
###                                                       ###
### Change the following lines to suit your configuration ###
###                                                       ###
#############################################################

my $comfort_port = '/dev/comfort';
my $mqtt_hostname = 'localhost';
my $mqtt_to_comfort = "comfort/to";
my $mqtt_from_comfort = "comfort/from";

#############################################################

my $serial_location = "off";
my $serie =  IO::Select->new();

my $mqtt = Net::MQTT::Simple->new($mqtt_hostname);
$mqtt->subscribe($mqtt_to_comfort,\&callback);

my $ts = time();

while(1) {

  if ( time() - $ts > 10 ) { # every 10 secs
    $ts = time();
    my $SL = `ls -al /dev/comfort 2>&1`;
    if( $SL =~ /(comfort -> .+)/ ) { # isconnected
      $SL = $1;
      if ( $serial_location eq "off" ) { # was plugged
        print localtime(time) . " $SL\n";
        $serial_location = $SL;
        open(SERIAL,  "+<$comfort_port") or die "Can't open serial port: $!\n";
        $serie->add(\*SERIAL);
      }
      elsif ( $SL ne $serial_location) { # was restarted
        print localtime(time) . " $SL\n";
        $serial_location = $SL;
        $serie->remove(\*SERIAL);
        close(SERIAL);

        open(SERIAL,  "+<$comfort_port") or die "Can't open serial port: $!\n";
        $serie->add(\*SERIAL);
      }
    }
    else {  # is disconnected
      if($serial_location ne "off") { # was removed
        print localtime(time) . "UCM disconnected\n";
        $serial_location = "off";
        $serie->remove(\*SERIAL);
        close(SERIAL);
      }
    }
  }
  $mqtt->tick(0.1);
  if($serial_location ne "off") {
    if ($serie->can_read(.01)) {
      my $recv = <SERIAL>;
      $recv =~ tr/\x03\x0D\x0A//d;
      $mqtt->publish($mqtt_from_comfort => $recv);
    }
  }
}

sub callback {
  my ($topic, $message) = @_;
  if($serial_location  ne "off") { print SERIAL chr(0x03) . "$message\r\n"; }
  else { $mqtt->publish($mqtt_from_comfort => "OFFLINE"); }
}


To test:
WINDOW_1: run the script

root@debian:/# perl simple_mqtt_serial_select.pl



WINDOW_2: listen for MQTT events

root@debian:/# mosquitto_sub -v -t comfort/#


WINDOW_3: post something to comfort topic MQTT

root@debian:/# mosquitto_pub -t comfort/to -m 'LI1234'


On the WINDOW_2 you should see something like:

comfort/to LI1234
comfort/from LU01


That´s it! Now you can subscribe (listen) or post commands to the MQTT broker and interact with comfort.

Make the script boot at startup by editing the file just before "exit 0"

root@debian:/# nano /etc/rc.local
/usr/bin/perl /opt/simple_mqtt_serial_select.pl &


And now comfort can easily be integrated with other bits of amazing software, like openHAB, NodeRED, etc.


If there is any interest in this i can share some of my NodeRED and OpenHAB integration configuration.


Have fun!



 Posted: Wednesday Feb 10th, 2016 07:23 am
   PM  Quote  Reply 
2nd Post
tech07
Administrator
 

Joined: Thursday Jan 29th, 2009
Location: Singapore
Posts: 106
Status: 
Offline

  back to top

yes pleasecan this work with UCM/ETh03?
How can this allow working with openhab  etc



 Posted: Wednesday Feb 10th, 2016 07:31 pm
   PM  Quote  Reply 
3rd Post
cab123
Member
 

Joined: Saturday Oct 3rd, 2009
Location: Coimbra, Portugal
Posts: 25
Status: 
Offline

  back to top

Hi, i'm not sure how the UCM/ETH03 works, since i don't have one, but i suppose everything is possible :).

I'll gather some info and post here later a working example of OpenHAB controlling comfort.



 Posted: Saturday Feb 13th, 2016 06:58 am
   PM  Quote  Reply 
4th Post
cab123
Member
 

Joined: Saturday Oct 3rd, 2009
Location: Coimbra, Portugal
Posts: 25
Status: 
Offline

  back to top

So the next step after having a MQTT broker interacting with comfort is to install NodeRED
http://nodered.org/

It is really easy to set up stuff in logical blocks, however some javascript is needed for the functions.
This is my processing of comfort stuff:


The first "flow" updates comfort's time periodically. Every 10 minutes a timestamp is generated, passed through the "Build Date Command" and sent to the MQTT topic comfort is listening to. The function is something like this:



At the bottom of the first image, every group of received messages from Comfort are handled. For instance, i have temperature sensors, one KNX and another from a SCS that are transformed to MQTT also and sent to the broker. This way i can later read the sensors and display them on openHab. 

The "split function is configured as shown:



and the Sensor function decodes comfort sensor messages like this:
var string = msg.payload;
var re = /^s[r\?](..)(..)(..)/; // regex to extract LSB, MSB
var sensor = msg.payload.replace(re, "$1"); //get sensor id
var hex = msg.payload.replace(re, "$3$2"); //Store MSB LSB
var dec = parseInt(hex,16); // convert to Decimal

if ((dec & 0x8000) > 0) { // handle negative numbers   
  dec = dec - 0x10000;
}
switch(sensor) {   
  case "00":       
    msg.topic = "house/floor2/livingroom/temperature";       
    break;   
  case "09":       
    msg.topic = "house/floor2/bedroom/temperature";       
    break;   
  default:       
    msg.topic = "unspecified";
}

msg.payload = dec;return msg;


All processing can be done here. Imagination is the limit, everything can be done with small bits of code and simple "flows".

Later i will try to show some of my OpenHab setup that takes advantage of all this data.

Fell free to ask any questions.




 Posted: Wednesday Aug 10th, 2016 07:34 pm
   PM  Quote  Reply 
5th Post
cab123
Member
 

Joined: Saturday Oct 3rd, 2009
Location: Coimbra, Portugal
Posts: 25
Status: 
Offline

  back to top

Hi guys sorry for the late update, i didn't want to show the openhab examples in portuguese so i kept delaying a update. Now i have changed my dashboards to show you what is the final result.

for setting up openhab to control comfort you just need to interact with the mqtt broker, still it is a bit slow learning curve but it is totally worth it!




and for example inside the TV control you can find the actual control:




The buttons listen for updates, so if i turn off my living room TV locally, i am actually pressing a KNX button that sends a telegram to the BUS that is captured by the UCM and passed on to comfort that updates a counter that is sent though serial connection that is captured by the perl driver and sent to MQTT, that broadcasts it to any listening device, where openhab is waiting, and then applies some rules to update the button status that is binded to the TV, so next time i toggle the TV i will perform the right action! Easy!  :D


Last edited on Wednesday Aug 10th, 2016 07:36 pm by cab123



 Posted: Friday Aug 12th, 2016 08:58 am
   PM  Quote  Reply 
6th Post
slychiu
Administrator


Joined: Saturday Apr 29th, 2006
Location: Singapore
Posts: 5650
Status: 
Offline

  back to top

Thats nice. Its good to see Comfort with Openhab.
We will try to test it



 Posted: Saturday Aug 13th, 2016 01:43 pm
   PM  Quote  Reply 
7th Post
slychiu
Administrator


Joined: Saturday Apr 29th, 2006
Location: Singapore
Posts: 5650
Status: 
Offline

  back to top

How does the Comfort program go into this setup?



 Posted: Friday Sep 9th, 2016 04:40 pm
   PM  Quote  Reply 
8th Post
cab123
Member
 

Joined: Saturday Oct 3rd, 2009
Location: Coimbra, Portugal
Posts: 25
Status: 
Offline

  back to top

Hi not sure i understand your question. The comfort software requires direct and exclusive access to the UCM, so they cannot exist in a system at the same time.

In order to use comfort apps or to upload new configuration, i normally shutdown this "driver" and use the socat approach described here:

http://www.comfortforums.com/view_topic.php?id=3249&forum_id=83&jump_to=13896#p13896



 Posted: Thursday Oct 6th, 2016 04:45 pm
   PM  Quote  Reply 
9th Post
meep
Member


Joined: Saturday Sep 6th, 2014
Location:  
Posts: 11
Status: 
Offline

  back to top

Hi

Is there any chance you could post more of your Node-Red javascript?

I need to do exactly this as I want to start using my old comfort as essentially an i/o block for openHab.

The more reference code I can find the better.

As an aside - any thoughts on how one could go about accessing Comfort voicemail recordings from openHab?

Many thanks

Peter



 Posted: Sunday Oct 9th, 2016 01:32 am
   PM  Quote  Reply 
10th Post
idheath
Member
 

Joined: Saturday Aug 11th, 2007
Location: United Kingdom
Posts: 87
Status: 
Offline

  back to top

I'm so pleased that I've seen this thread :-)

I have just started with openHAB and am just about to have a play with MQTT (for other reasons). I've been wondering for ages how I could connect Comfort to openHAB and wondered myself about MQTT as a sensible option.

I have a UCM/ETH02 which I intend to keep for Comfigurator.

I also have a UCM/232 which I plan to connect to my Raspberry Pi 3 (which is running openHAB and will also be running MQTT when I install it). I've got a tiny USB to serial converter so I should be able to link my UCM/232 to the Raspberry Pi that way.



 Posted: Sunday Oct 9th, 2016 02:01 am
   PM  Quote  Reply 
11th Post
meep
Member


Joined: Saturday Sep 6th, 2014
Location:  
Posts: 11
Status: 
Offline

  back to top

I managed to get comfort talking to Mqtt and node-red via ucm/232 and a usb/serial convertor. Works great both to and from.

It's on Windows though so I needed to completely re-write the perl driver to work. I'll share it here when I've got it refined.

Last edited on Sunday Oct 9th, 2016 02:02 am by meep



 Posted: Sunday Oct 9th, 2016 11:23 pm
   PM  Quote  Reply 
12th Post
idheath
Member
 

Joined: Saturday Aug 11th, 2007
Location: United Kingdom
Posts: 87
Status: 
Offline

  back to top

meep wrote:
I managed to get comfort talking to Mqtt and node-red via ucm/232 and a usb/serial convertor. Works great both to and from.

It's on Windows though so I needed to completely re-write the perl driver to work. I'll share it here when I've got it refined.

Thanks v.much



 Posted: Monday Oct 10th, 2016 03:39 am
   PM  Quote  Reply 
13th Post
meep
Member


Joined: Saturday Sep 6th, 2014
Location:  
Posts: 11
Status: 
Offline

  back to top

Well, I managed to get this working on windows with Win32::SerialPort in Perl. (just couldn't hack the above code to work so started afresh).

It's a long way from cab123's original driver in terms of code but owes a lot to his concept, structure and techniques. It's also got a LOT less error checking implemented so is really just a proof of concept at this stage and would require a few more additions to make it more robust.

While this does send and receive between comfort and MQTT, and you can check these messages in Node-Red, I added a Trace function to allow key messages be printed to the console - really helps with debugging.

!/usr/bin/perl

use Net::MQTT::Simple;
use Win32::SerialPort;
use strict;
use warnings;

#hello
print STDERR "\n".localtime()."\nOpening Comfort Communications... \n";

#Set up & open port
my $port = Win32::SerialPort->new("COM5") or die "Failed to open port: $!\n";
print STDERR "Port established: $port \n";

$port->baudrate(9600);
$port->databits(8);
$port->parity("none");
$port->stopbits(1);
$port->handshake("xoff");
$port->write_settings;


#set up mqtt & polling for inbound messages
my $mqtt_hostname = '192.168.1.199';
my $mqtt_to_comfort = "comfort/to";
my $mqtt_from_comfort = "comfort/from";

my $mqtt = Net::MQTT::Simple->new($mqtt_hostname);
$mqtt->subscribe($mqtt_to_comfort,&callback);

#additional control vars
my $serial_stream = ""; #container for serial input from comfort
my $ts = time(); #current time
my $defaultreadspeed = 1; #port read interval
my $trace = 1; #enable debug output to console

#setup complete
print STDERR "Starting to listen...\n";

while (1) { #main loop

if ( time() - $ts > $defaultreadspeed ) { #use $defaultreadspeed to throttle

$ts = time(); #set target time for next loop
$serial_stream = $port->read(200); #read up to 200 bytes from Comfort

if ($serial_stream) { #check if there's data read

doTrace ("received from Comfort: $serial_stream");
#split the data into an array based on STX or EOL
#remove those extraneous characters from the resulting array elements
my @reports = split /(?:\x03|\x0D|\x0A)/, $serial_stream;

for my $rep (@reports) { #loop through each of the returned strings
if (length($rep)){ #only process non-empty strings
doTrace ("sending report to MQTT: $rep");
$mqtt->publish($mqtt_from_comfort => $rep); #send to MQTT
}else{
doTrace("empty string, not sending to MQTT");
}
}

$port->lookclear; #needed to prevent blocking (?)
} #end check if data read

$mqtt->tick(1); #force query MQTT input with 1 second timeout

} #time check
}#while loop



################
##SUBROUTINES##
##############

sub callback { #process inputs from MQTT and send to Comfort
my ($topic, $message) = @_;
doTrace ("got message from MQTT: $message");

$port->lookclear; #needed to prevent blocking (?)
my $str = chr(0x03).$message."\r\n"; #prepend STX char and add CR
$port->write($str); #send to Comfort

doTrace ("sent to Comfort: $str");
}

sub doTrace { #write trace logs to console
my ($traceString) = @_;

if ($trace){ #only implement if enabled
my $timestamp = getLoggingTime(); #get timestamp
print STDERR "$timestamp $traceString \n"; #print to console
}
}

sub getLoggingTime { #compose a decent looking timestamp

my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
my $nice_timestamp = sprintf ( "%04d%02d%02d %02d:%02d:%02d",
$year+1900,$mon+1,$mday,$hour,$min,$sec);
return $nice_timestamp;
}


This can be tested with something like the command below at the command line. Obviously change your own Mosquito server and port details. (note, you must first issue the login (LI) command to Comfort, with a user code, or it will keep returning 'NA' as the response);

mosquito_pub -h 192.168.1.199 -p 1883 -t comfort/to -m "z?"


Attached is an example of this running. You can see the initial receipt of the "z?" query from MQTT and the response from comfort sent back. Then you can see Comfort reporting activity Input 05 (that's the result of me opening and closing a window with a contact sensor).

NB: For some reason, the forum is stripping backslashes from the code segments above. I've tried to add them back in but if you're trying to run this and it's failing, it mey be due to a missing '\' somewhere.


Attachment: Capture.PNG (Downloaded 117 times)

Last edited on Monday Oct 10th, 2016 01:09 pm by meep



 Posted: Tuesday Oct 11th, 2016 03:15 am
   PM  Quote  Reply 
14th Post
cab123
Member
 

Joined: Saturday Oct 3rd, 2009
Location: Coimbra, Portugal
Posts: 25
Status: 
Offline

  back to top

Hey! I'm very happy to see you are having fun with this!

I don't have much more javascript examples, aside from what i already shared, it's pretty basic stuff.

i would recommend you to switch to a linux box for these server-side protocols - they will run forever, will never complain, and be readily available to serve you anytime :-)
A small raspberry Pi is cheap to buy/run and has enough horsepower to do this and much more.



 Posted: Tuesday Oct 11th, 2016 03:49 am
   PM  Quote  Reply 
15th Post
meep
Member


Joined: Saturday Sep 6th, 2014
Location:  
Posts: 11
Status: 
Offline

  back to top

Cheers, it's more getting to grips with node-red and regex to be honest, getting there slowly.

I run all my stuff on a single unraid server through a combination of dockers and vms.

I went with a Windows vm as I needed to pass through the usb/serial adapter and as I'll likely need to run comfigurator for a while, thought I'd go with Windows as I wouldn't need to keep swapping connections etc.

Might switch back to a Linux vm as I get more comfortable with it all.

Thanks for the inspiration on this and taking the time to document it here, really helped me realise what I needed to do was possible.



 Posted: Thursday Jul 11th, 2019 02:35 pm
   PM  Quote  Reply 
16th Post
jelockwood
Guest
 

Joined: 
Location:  
Posts: 
Status: 
Offline

  back to top

@cab123

If you're still working on this any plans to implement an MQTT broker on the UCM/Pi module?

Obviously one could use the UCM/Eth03 and then link to a standard external Raspberry Pi but would seem to make sense to do the Comfort side of this using the dedicated Pi module.



 Posted: Friday Jul 12th, 2019 12:25 pm
   PM  Quote  Reply 
17th Post
mattbrain
UCM Pi Users


Joined: Tuesday Jun 12th, 2007
Location: United Kingdom
Posts: 185
Status: 
Offline

  back to top

The alphaWerk components uses MQTT as their backend & Mosquitto is installed on the CM3. It is locked to localhost only by default but can be opened up to external devices easily.

There is also nothing stopping you from installing different software, hardware notes are available to use your own tools to talk to comfort and manage the watchdog timer.



 Current time is 08:41 pm
Top




UltraBB 1.172 Copyright © 2007-2014 Data 1 Systems