The adventure to a remote Z-Wave USB - Part 1
Why?
First a little background information on why I want a remote Z-Wave USB. I am a fan of Kubernetes and a huge geek so a logical step was to run my home automation from a Kubernetes cluster (right?).
I set up a Kubernetes(k8s) cluster with three Rock64 nodes and I am running Home Assistant, Zwave2MQTT and Zigbee2MQTT on it, a post explaining this more in depth will follow soon!
But for now it is important to know that both Zwave2MQTT and Zigbee2MQTT are MQTT bridges for their protocols. You connect the Z-Wave/Zigbee coordinator to the bridge and from then on your protocol is exposed over MQTT. So if Home Assistant needs to turn on the light, it sends a MQTT message which gets picked up by either Zwave2MQTT or Zigbee2MQTT which then sends it to the light using either Z-Wave or Zigbee.
Since I am running Zwave2MQTT on Kubernetes I don’t want to depend on USB devices plugged in nodes, this undermines the whole high availability and fail over principle of Kubernetes.
Oh node01 with your Z-Wave USB is down? To sad, now I can’t schedule Zwave2MQTT anymore"
– the Kubernetes scheduler
I had this happen and it resulted in half the lights not working (the other half is Zigbee), so I wanted to get rid of the tight coupling of the Zwave2MQTT container to a specific node.
With Zigbee2MQTT you have the option to host the CC2530 Zigbee coordinator over remote serial and let Zigbee2MQTT connect to the socket (explained here in the docs).
serial:
port: 'tcp://192.168.2.13:20108'
I use an ESP8266 based microcontroller with jeelabs/esp-link as serial bridge for the controller. Instructions how to setup and flash esp-link can be found in FLASHING.md in the root of the esp-link repository. This is working flawlessly so I would like a similar setup for Z-Wave as well.
The Z-Wave module
I started looking for suitable Z-Wave modules to hookup to an ESP8266 to create a similar setup as with Zigbee2MQTT. This was proven quite hard since most modules have an USB connector which is difficult to connect to an ESP8266. Something with a pin header for serial access would be the goal.
I started searching for random terms since I had no idea where to start but ‘zwave serial’ in DuckDuckGo images gave a promising image which turned out to be the Z-Wave.Me RaZberry module. This module is exactly what I need, it provides a pin header for communication and is supported on a Raspberry Pi. A good change it will work with an ESP8266 as well since it is just serial communication (hopefully). So I started looking where I could buy one and found a second-hand one for a good price, nice!
Now how do I hook this up to the ESP8266? I found the manual for the module online and luckily it stated something about the connections used:
The RaZberry board use only four pins of this header connector:
• Gnd
• VCC (3.3V)
• Serial TX
So I lookup up the pin-out of a Raspberry Pi and reverse engineered that this is the pinout of the RaZberry Z-Wave module. I used the same colors as the Raspberry Pi schema over at pinout.xyz.
So I connected it to the Wemos D1 mini with esp-link (RX->TX, TX->RX, GND->GND and 3.3V->3.3V) and had some output on the console!
Z-Wave 3.99
^�
^�2!D-�/2!D-�
Hooking it up to Zwave2MQTT
Now, how to hook this up to Zwave2MQTT? First I checked with the developer on Github if a TCP socket connection as device is possible to support. But sadly it can’t be supported since the library Zwave2MQTT uses (open-zwave or OZW in short) requires a serial device on the host.
It wont work. OZW expects a Serial Device…
You could try configuring USB/IP on your kernel
Hmm, USB/IP as suggested isn’t working for what I had in mind since it uses their own protocol which means that the client and server both need to run USB/IP which is not possible on an ESP8266. As mentioned before I have Zigbee2MQTT successfully working with jeelabs/esp-link as serial bridge. A setup for Z-Wave with esp-link would be the goal so I started googling again and went from looking for ‘remote serial client linux’ to ‘ser2net client’ to ‘rfc2117 client’ which turned up two possible clients:
I tried with cyclades (LANsible/docker-cyclades) but I couldn’t get it
to work and the documentation is referring to Cyclades Terminal Servers all the time, so I moved on.
Ttynvt was the next candidate which I got to work on my localhost! Ttynvt uses CUSE
(character in user space) to emulate a new serial device which forwards all the calls to the remote client.
Before running the container there is no /dev/ttyNVT* device.
# ls -l /dev/ttyNVT*
zsh: no matches found: /dev/ttyNVT*
When running the container this /dev/ttyNVT0 device is created and it the forward to 192.168.1.23:23 in my case.
# docker run -it --device /dev/cuse lansible/ttynvt:master -n ttyNVT0 -S 192.168.1.23:23
# ls -l /dev/ttyNVT*
crw-rw---- 1 root dialout 242, 0 mrt 22 14:05 /dev/ttyNVT0
So that should work for the open-zwave library since it is a serial device on the host, as expected. I wrote a daemonset so the port will be available on each of my Kubernetes hosts, the daemonset can be found in the repository as an example: LANsible/docker-ttynvt.
Important to note is the requirement of a kernel with CUSE support.
I ran into this since the ayufan-rock64 image I was running wasn’t
compiled with CONFIG_CUSE enabled so the daemonset was not working. So I had to reinstall my nodes with
Armbian which has the needed kernel flag.
The moment of truth! I changed Zwave2MQTT to connect to /dev/ttyNVT0 but sadly no avail. Zwave2MQTT kept complaining
about the connection to the controller (sadly didn’t save the logs).
Updating the RaZberry
So I started looking for a solution and found that there are updates available for my RaZberry module, that might help, right? At this point I am throwing everything at it and seeing what sticks.
First I tried updating with the provided ZMESerialUpdater, I connected the RaZberry to a CP210 serial to USB and gave it a go. It starts the flashing process fine but it keeps reporting the old version so it didn’t seem to work. Then I came across this post at the OpenHAB forum where Ben explains in detail how to update the RaZberry with Z-Way server (massive thanks @Ben!).
I tried to install the Z-Way server in a Docker container but ran into all kinds of issues and decided that it wasn’t worth my time so I grabbed my trusty Raspberry Pi 1 model B, flashed Raspbian and went to work:
wget -q -O - razberry.z-wave.me/install | sudo bash
sudo /etc/init.d/z-way-server start
Then went to <ip>:8083/expert/#/network/controller to check the current version (5.00) and deducted my upgrade path from:
https://service.z-wave.me/expertui/uzb-stats/versions-graph.html?hw=327
At ip:8083/expert/#/uzb I uploaded the upgrade files one by one and kept checking if the version kept increasing.
This all worked and I ended up on the latest version, 5.36!
Hooking it up to Zwave2MQTT, second try
Time to try again with Zwave2MQTT! This time it came further and recognized the controller!
So I started pairing my Neo Coolcam Power Plugs and they paired successfully and I was getting measured energy values so far so good.
2020-03-19T12:53:13.449Z z2m:Zwave zwave node 2: changed: 50-1-0:Electric - kWh:0.05 -> 0.05
2020-03-19T12:53:13.454Z z2m:Zwave zwave node 2: changed: 50-1-256:Exporting:false -> false
2020-03-19T12:53:14.217Z z2m:Zwave zwave node 2: changed: 50-1-2:Electric - W:0 -> 0
2020-03-19T12:53:14.223Z z2m:Zwave zwave node 2: changed: 50-1-256:Exporting:false -> false
2020-03-19T12:53:15.129Z z2m:Zwave zwave node 2: changed: 50-1-4:Electric - V:224.23 -> 224.29
2020-03-19T12:53:15.134Z z2m:Zwave zwave node 2: changed: 50-1-256:Exporting:false -> false
2020-03-19T12:53:16.135Z z2m:Zwave zwave node 2: changed: 50-1-5:Electric - A:0 -> 0
2020-03-19T12:53:16.140Z z2m:Zwave zwave node 2: changed: 50-1-256:Exporting:false -> false
But when I tried turning of one of the power plugs the log didn’t look happy and showed some kind of timeout.
2020-03-19T12:53:08.319Z z2m:Mqtt Message received on homeassistant/2/37/1/0/set
2020-03-19 12:53:09.066 Warning, WARNING: 500ms passed without reading the rest of the frame...aborting frame read
2020-03-19 12:53:09.066 Warning, WARNING: Out of frame flow! (0x00). Sending NAK.
2020-03-19 12:53:09.682 Warning, WARNING: 500ms passed without reading the rest of the frame...aborting frame read
2020-03-19 12:53:09.682 Warning, WARNING: Out of frame flow! (0x00). Sending NAK.
2020-03-19 12:53:10.302 Warning, WARNING: 500ms passed without reading the rest of the frame...aborting frame read
2020-03-19 12:53:10.302 Warning, WARNING: Out of frame flow! (0x00). Sending NAK.
2020-03-19 12:53:10.911 Warning, WARNING: 500ms passed without reading the rest of the frame...aborting frame read
2020-03-19 12:53:10.911 Warning, WARNING: Out of frame flow! (0x00). Sending NAK.
2020-03-19 12:53:18.427 Error, Node002, ERROR: Dropping command, expected response not received after 1 attempt(s). Command: "SwitchBinaryCmd_Get (Node=2): 0x01, 0x09, 0x00, 0x13, 0x02, 0x02, 0x25, 0x02, 0x25, 0x2f, 0xc8"
So I went looking in the source code and patched the timeout to 1000ms, this got rid of the Out of frame flow errors.
But the command was still dropped and the power plug isn’t turning on or off.
2020-03-19T12:49:30.913Z z2m:Mqtt Message received on homeassistant/2/37/1/0/set
2020-03-19 12:49:32.352 Warning, Node002, WARNING: Checksum incorrect - sending NAK
2020-03-19 12:49:33.353 Warning, Node002, WARNING: Checksum incorrect - sending NAK
2020-03-19 12:49:41.276 Error, Node002, ERROR: Dropping command, expected response not received after 1 attempt(s). Command: "SwitchBinaryCmd_Get (Node=2): 0x01, 0x09, 0x00, 0x13, 0x02, 0x02, 0x25, 0x02, 0x25, 0x33, 0xd4"
2020-03-19T12:49:41.280Z z2m:Zwave Notification from node 2: Notification - TimeOut (1)
At this point I am out of ideas and seeking help of the OpenZWave developers. This one of the reasons for getting this post online, it was way to much information for in a Github issue :')
To be continued…