Author Archives: tobymurray

Unify Signal, WhatsApp and SMS in a personal Matrix server: Part 1 (Matrix)

Goal: Use a single app to satisfy all my messaging needs. Bonus points if I can get it working on a PinePhone

Good overview from https://docs.mau.fi/bridges/go/whatsapp/index.html by http://matrix.to/#/@azata:gazizova.net

Prerequisites

  • a reasonably recent version of Docker
  • a reasonably recent version of Docker-Compose
  • somewhere to deploy this and a corresponding domain name (preferable) or IP address
    • See naming considerations here
  • a Matrix client of some sort
    • I have not found any Android clients which work with a Matrix server deployed locally on a LAN
  • an example docker-compose.yaml at the bottom of this post, needs a tiny bit of modification
  • a rudimentary understanding of Matrix and Matrix bridges

Summary

Just to be upfront, this is all a bit of a pain in the ass. The configuration is stateful, so there’s no docker-compose up that I’m aware of that will “just work”. The pieces need to be done one at a time.

  1. We’ll configure a Matrix server – Synapse, start it up, and add a user
  2. We’ll start a Signal daemon (signald), configure a Matrix-Signal bridge (mautrix-signal) then integrate it with our Matrix server
  3. We’ll configure a Matrix-WhatsApp bridge, then integrate it with our Matrix server

What’s not included:

  • SSL configuration
  • Matrix user definition/roles (beyond the bare minimum)
  • double puppeting
  • considerations for sharing this server with others

Create volumes for shared persistence

  1. Create Docker volumes that we’ll use throughout the rest of this
    • note these must match the names in the docker-compose.yaml, so either don’t change them or change them both places
    • docker volume create synapse-data
    • docker volume create whatsapp-bridge-data
    • docker volume create signal-bridge-data

Matrix server

  1. make sure you update the docker-compose.yaml to contain an actual password for synapse-db
  2. use an ephemeral Docker container to generate the Synapse configuration file
    1. docker run --rm -e SYNAPSE_SERVER_NAME=nuc.localdomain -e SYNAPSE_REPORT_STATS=yes -v synapse-data:/data matrixdotorg/synapse:latest generate
    2. note that it’s extremely unlikely you’ll have the same Synapse server name, so ensure you’re customizing it. This value should be what you decided in the prerequisites above.
    3. additionally note that the volume must be the same as step 1 above (and the docker-compose.yaml file)
$ docker run --rm -e SYNAPSE_SERVER_NAME=nuc.localdomain -e SYNAPSE_REPORT_STATS=yes -v synapse-data:/data matrixdotorg/synapse:latest generate
/usr/local/lib/python3.8/site-packages/twisted/conch/ssh/common.py:14: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes, int_to_bytes
Generating config file /data/homeserver.yaml
Generating signing key file /data/nuc.localdomain.signing.key
A config file has been generated in '/data/homeserver.yaml' for server name 'nuc.localdomain'. Please review this file and customise it to your needs.
  1. now we have a Docker volume (a Docker managed directory) that is populated with the most recent template homeserver.yaml by a Docker container that attached to the volume, generated it, then removed itself after generating. This next step is configuring this homeserver.yaml so you can ultimately run Synapse.
    1. as we’re using a Docker volume, the first step is to actually find the file we want to change:
      • docker inspect --format='{{.Mountpoint}}' synapse-data
      • this should show you something like: /var/lib/docker/volumes/synapse-data/_data
    2. you can look in that directory (probably going to need sudo) or we can go straight to modifying it (use whatever editor you’re comfortable with
      • sudo nano $(docker inspect --format='{{.Mountpoint}}' synapse-data)/homeserver.yaml
    3. Assuming you used the right server_name, there’s only one thing we have to update right now
      • search for database:
      • comment out the existing configuration (sqlite3)
      • uncomment the Postgres configuration (psycopg2)
      • update the user to the synapse-db username in the docker-compose.yaml
      • update the password to the synapse-db password in the docker-compose.yaml
      • update the database to the synapse-db database name in the docker-compose.yaml
      • update the host to synapse-db (the host Docker-Compose introduces)
      • Save and exit, assuming this section of the homeserver.yaml looks like:
database:
  name: psycopg2
  args:
    user: synapse     
    password: reallyStrongPassword!1
    database: synapse_db
    host: synapse-db
    cp_min: 5
    cp_max: 10

# For more information on using Synapse with Postgres, see `docs/postgres.md`.
#
#database:
#  name: sqlite3
#  args:
#    database: /data/homeserver.db
  1. get that sweet endorphin hit of progress by starting up Synapse!
    1. docker-compose up synapse
    2. navigate to your host (don’t forget the port!), for me: http://nuc.localdomain:8008
    3. if you have anything but sweet joy, see the troubleshooting section towards the end of this post
      • keep an eye on the logs for anything suspicious
  2. you can ctrl+c to escape
Successful Synapse start up page!
  1. Unfortunately, our Matrix server has no users, so it’s not all that useful. With the container running, lets add one to get this moving
    • docker-compose exec synapse register_new_matrix_user http://nuc.localdomain:8008 -c /data/homeserver.yaml
$ docker-compose exec synapse register_new_matrix_user http://nuc.localdomain:8008 -c /data/homeserver.yaml
New user localpart [root]: toby
Password: 
Confirm password: 
Make admin [no]: yes
Sending registration request...
Success!

Done this piece!

Synapse the Matrix server is running, it has a user associated with it. Now is a good time to get used to what it looks like running, maybe try out clients to see which ones can actually connect to your set up, generally troubleshoot things.

Troubleshooting

  • during set up, the containers are completely stateless – all the state is in the volumes. This means you’re free to delete and recreate them at your leisure
  • generally speaking, if you need to update a configuration file (e.g. homeserver.yaml), restart the relevant service afterwards
  • The “nuclear” option is to stop and remove all containers as well as removing all volumes. This shouldn’t be done lightly, you’ll have to start over. The command (executed in the directory where your docker-compose.yaml is: docker-compose down --volumes to remove everything your docker-compose file manages, then you’ll also have to remove the external named volumes with docker volume rm synapse-data whatsapp-bridge-data signal-bridge-data.

Example docker-compose.yaml

version: "3.7"

volumes:
  # We need to do some initial set up outside of Docker-Compose, so make these volumes external
  synapse-data:
    external: true
  whatsapp-bridge-data:
    external: true
  signal-bridge-data:
    external: true

  # The Signal daemon (Signald) and the Signal bridge (Mautrix-Signal) need to share a volume
  signald-data:

  # Use these named volumes just for convenience
  signal-bridge-db:
  synapse-db:

services:
  synapse-db:
    container_name: synapse-db
    image: postgres:13-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
      POSTGRES_DB: synapse_db
      # ensure the database gets created correctly
      # https://github.com/matrix-org/synapse/blob/master/docs/postgres.md#set-up-database
      POSTGRES_INITDB_ARGS: --encoding=UTF-8 --lc-collate=C --lc-ctype=C
    volumes:
      - synapse-db:/var/lib/postgresql/data
    ports:
      - 5435:5432/tcp

  synapse:
    container_name: synapse
    image: docker.io/matrixdotorg/synapse:latest
    # Since synapse does not retry to connect to the database, restart upon failure
    restart: unless-stopped
    environment:
      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
    volumes:
      - synapse-data:/data
      - signal-bridge-data:/signal-bridge
      - whatsapp-bridge-data:/whatsapp-bridge
    depends_on:
      - synapse-db
    ports:
      - 8008:8008/tcp
      - 8448:8448/tcp

  whatsapp-bridge:
    container_name: whatsapp-bridge
    image: dock.mau.dev/tulir/mautrix-whatsapp:latest
    restart: unless-stopped
    volumes:
    - whatsapp-bridge-data:/data
    ports:
      - 29318:29318/tcp

  # Signal and bridge
  signald:
    container_name: signald
    image: finn/signald:latest
    restart: unless-stopped
    volumes:
    - signald-data:/signald
  signal-bridge-db:
    container_name: signal-bridge-db
    image: postgres:13-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: mautrix_signal
      POSTGRES_DB: signal_bridge_db
      POSTGRES_PASSWORD: <GENERATE A PASSWORD HERE> # Make sure to update this!
    volumes:
    - signal-bridge-db:/var/lib/postgresql/data
    ports:
        - 5434:5432/tcp
  signal-bridge:
    container_name: signal-bridge
    image: dock.mau.dev/tulir/mautrix-signal
    restart: unless-stopped
    volumes:
    - signal-bridge-data:/data
    - signald-data:/signald
    ports:
      - 29328:29328/tcp
    depends_on:
      - signal-bridge-db
      - signald

Carbon fiber heating coil power calculator

A bunch of projects I try to do end up essentially being “put a heater somewhere”. Most of the time I don’t really know what I’m going to do or what size heater I’ll ultimately want. At the the same time, I want the lowest power heater possible to decrease the chance I’ll burn anything down.

I’ve purchased a variety of small heaters, and they all work. They also cost 10-20$ and only come in the sizes they’re sold in. This makes everything a pain in the butt AND I have to wait for them to show up. As with everything, the less you want to pay the lower the chance you’ll get what you want when you want it.

After looking at a bunch of different options, I’ve recently purchased multiple rolls of different carbon fiber heating cable. When ordered from China, it’s ~25$ USD for 100m. I got 4 different rolls, identified by the number of strands of carbon fiber – 1k, 3k, 6k, and 12k. These, respectively, have resistances of 400 Ω/m, 130 Ω/m, 61 Ω/m, 33 Ω/m. That means I can now create a heater of all sorts of different shapes and powers!

The one big thing to be aware of is the heating cable can only get so hot before it burns down the house. One of the listings I purchased from mentioned 25 watts per meter as the max power. Now to do math to figure out which specific coil at which length for a given geometry.

After trying (and failing) to keep the numbers straight I decided to make a quick calculator:

Put in the resistance per meter and the length, get out the total power and the power per meter. Very niche but very helpful! As an example, one of the heaters pictured above is ~18 W. To make it out of the 3k cable I need 2 meters, meaning I am paying an equivalent of ~0.60$ CAD (+ a few cents for 3D printed holder). Hard to beat that price!

Site is available here: https://tobymurray.github.io/Heating-Coil-Power-Calculator/

Thermal mass in an incubator

Visualizing things is always dramatically cooler than just “knowing” them, and the conditions within an incubator are no exception. Graphing the temperature and humidity over time turned out really cool to see. This is about the only real life example I’ve personally ever come across that looks like a math class. Specifically, both the humidity and the temperature are pretty damn consistently periodic, and they’re pretty close to sinusoidal. Because relative humidity depends on temperature, the graph of relative humidity is very obviously the composition of two functions with different frequencies (the “absolute” humidity and the temperature).

This is mostly a function of a really naive heating and humidifying implementations: heat until a threshold, turn heater off, passively cool until a threshold, turn the heater on, repeat. Same for the humidifier.

humidity

That’s all I wanted to show about the humidity, just that it looks like math class.

temperature
Detail view of the temperature over time (roughly 9 hours represented)

Taking some liberties with the specific values, the heat can roughly be graphed by:

original-function
initial stab at heat

Breaking it down, this means it’s averaging about 38.2°C, varies by ±1.45°C, and has a heat/cool cycle every 5.6 seconds. In terms of an incubator, that’s too hot compared to the target temperature, varies by more than is ideal, and… has a heat/cool cycle every 5.6 seconds. I don’t really know if there are pros/cons to longer or shorter heating cycles, I imagine some middle ground is likely ideal so 5.6s doesn’t seem unreasonable.

In particular, I’m way overshooting the upper heat threshold as the heater remains hotter than ambient for a bunch of seconds. Rather than solve this the “right” way, e.g. with PID algorithm, I continued to mess around. Aside from changing the heating/cooling thresholds, one of the more interesting things to do is add some thermal mass. I had some preconceived notions that thermal mass was like a battery but for heat.

after-adding-water-bucket-detail
Detail view of the temperature over time after adding a jug of water (roughly 9 hours represented)

May be a bit hard to see close up, but it’s quite a bit more clear when zoomed out (ignoring the outliers from me mucking with things). No water jug is on the left, water jug is on the right.

after-adding-water-bucket
The blue line in the center is where I added the bucket of water

With some more liberties taken with the numbers, the graph of the temperature after adding a jug of water is roughly:

second-function

In practical terms:

  • now averaging 37.8°C (previously 38.2°C)
    • I’m guessing that the thermal mass is “resisting” heating caused by overshooting the threshold, dropping the max temperature reached in the heating cycle
  • now varying ±1.05°C (previously 1.45°C)
    • this is a great improvement! The incubator spends more time closer to the target temperature
  • now has a period of 6.7s (previously 5.6)
    • the heating/cooling cycle takes a full extra second now that it’s heating/cooling the water.
    • I’m not sure if this is better or worse, seems better that it’s more gradual I guess?

Takeaways

  • if you have time series data, graph it! It’s almost always rewarding to see
  • all else held equal, a little bit of thermal mass has a dramatic impact
  • thermal mass is barely a “battery for heat”

After seeing the impact of the thermal mass, I went back and revisited the concept of “like a battery, but for heat”. While that’s kind of true, it’s pretty misleading. A battery is a battery because it stores a significant amount of energy in a usable form. In terms of the amount of heat stored in thermal mass (how much energy it takes to heat water from the low temperature threshold to the high temperature threshold), one could ask “how long would it stay within acceptable temperature if the power went out” or similar, and the answer is: barely any time at all, on the order of seconds.

By my current understanding, “thermal mass” isn’t really about being a thermal battery so much as it’s about resisting temperature change. Relative to a battery, a jug of warm water has low energy density. Relative to the same amount of air though, it’s incredibly high energy density, and that’s the key.

Signal is a bit of a disappointment

Recently I was looking to try and use Signal on my PinePhone, and the more I got into it the more my love of Signal diminished.

One of my biggest uses for my cellphone is Signal Messenger. It’s been eye opening (in a disappointing way) trying to get it on my PinePhone. I’ve started thinking that a messaging app is like the “hello world” of deployment – send information cross-platform, sync it, handle failures etc. Whether that message is literally user text or driving an application seems less important. The baseline deployment aspects seem pretty similar. Anyways, a few disappointing things about Signal:

Signal will not federate (actively come out against it)

This one is I think the biggest disappointment. It feels a bit like you’re damned if you do, damned if you don’t, but a non-federated messaging platform doesn’t feel like “the future”. It’s depressing to think 20 years from now we’re still going to struggling with the same fragmentation and mega-corp “lure-and-lock” style of closed ecosystem shenanigans (Blackberry was bad for that, Apple is perhaps the most egregious in modern times). The number of high profile chat applications built off the Signal technology that cannot interact with one another is already sad.

It’s worth reading the blog post about it, as it’s definitely a tough decision to make: Reflections: The ecosystem is moving

That said, there are various projects that provide unofficial clients for Signal. As these are using the official Signal servers, I’m curious what the organization thinks of them – maybe custom clients can fill most of the gaps?

There appears to be no intention for a web client, due to security concerns

There are some good reasons for this, so it’s a bit of “reality is disappointing”

The fundamental problem with web interfaces is: there’s no way to version, sign and securely distribute a web page. Instead, you’re re-requesting the code you’ll run every single time you visit the site (making audits practically impossible).

This effectively reduces the security of your end-to-end encrypted communication to that of your SSL connection to the server, i.e. you’re only as secure as the CA system. Anyone able to intercept the client-server SSL connection (and the server itself) can silently change the code you receive and execute, with a very low risk of getting caught. This is why products which offer end-to-end encrypted communication through in-browser crypto are often considered snake oil, unless they use some form of a packaged & signed browser extension.

https://community.signalusers.org/t/google-to-retire-chrome-apps-what-will-be-with-signal-desktop/469/6

Limited device access

First of all, everything is linked to a mobile number, so there’s no desktop only client. The desktop client is linked to a cell phone that has a phone number associated with it. Increasingly, I don’t think I have any reason to have a cell number – it’s mostly an avenue for spam/scams to reach me at this point. Due to the terrible quality of cell reception around me, I make most of my calls via data (Signal calls and Hangouts Dialer). This implies there’s effectively a subscription price for Signal if you don’t have/don’t want a cell phone – pay your local telecom for a number, or lose access. If anyone knows how to get a free/extremely cheap cell phone number in Canada I’m all ears.

Adding on to that, each account has a max limit of 1 mobile device and 5 non-mobile devices. In a very real situation I’ve been in, where I have one phone with my SIM in it and another one on Wi-Fi, they can’t both use Signal. This feels like a really arbitrary and silly restriction that is a pain in the ass to work around. I’m really curious if there’s a technical reason for this – e.g. why not make it 6 cumulative devices?

What are the alternatives?

Presently, I think the most compelling “alternative” to Signal is Matrix. Alternative is in quotes because it’s dramatically different in a number of ways. The reason I think it’s viable is the number of high quality bridges that exist and work today. This breaks down the walls a bit when it comes to moving to Matrix without leaving contacts behind.

Noting that I’m extremely biased by working in IT, but Matrix feels like it could be a modern libpurple. I’ll be making a sincere effort to give Matrix a shot and see how it goes. That said, a worrying possibility is that Matrix steals libpurple users and doesn’t significantly grow the ecosystem of people interested in messengers that are compatible with one another. I guess we’ll see…

Incubator break: 3D printed detour

This stage is where everything went sideways. I was not making significant progress on the incubator for 2 very distinct reasons:

  • On the electrical front, I continued to not know anything and was losing motivation. I couldn’t figure out the right direction to go, I was spending money on components, waiting ~6 weeks for them to arrive, then figuring out I had made some mistake or misunderstood what I wanted or not realized there was a better way.
  • On the mechanical front, making stuff is HARD. Designing mechanisms really does take time and experience. Making things with wood is imprecise unless you have a lot of knowledge and skill, which extends to both how to work with wood and the wood itself. Making things with metal is expensive, daunting and messy. 3D printing is pretty much the holy grail for things like this – clean, cheap, easy, and sharable.

That brought me to a point where I finally decided to buy a “real” 3D printer.

3D printer!

I had been interested in 3D printing for ages. I had been peripherally following it since it became “mainstream” with the RepRap project (I started paying attention ~2007). It was always a bit out of reach because I had none of the skills or money to dive into it. On the other hand, it always felt like in some alternate universe I would have been right in the midst of it. Regardless, I had wanted a 3D printer for ages. I had personal experience with one in the past, but it wasn’t a good experience.

The M3D Micro came along ~2014, and it was SO cheap that it seemed like a good opportunity to figure out if I wanted to save up and buy a nice one or abandon the whole idea. My wife and I bought one, and while it was interesting it was also a complete piece of garbage. Now that I know more about 3D printing, I wouldn’t even give it to someone for free – there are virtually no quality components in the whole thing. It put me off so much that I didn’t look at 3D printing again until I was staring down the barrel of this incubator project.

Do not buy this. If you see one on the ground, consider not even stopping to pick it up.

With an actual use case (mounting things, building cases/enclosures, gears, connectors), I was ready to reconsider 3D printers. The “go-to” printer at the time was the Prusa MK3. It was a little expensive, but had thousands of great reviews. I went for it, and proceeded to spend ~a year getting completely distracted by 3D printing.

The progression of 3D printing I found was:

  1. print out all sorts of random desk baubles and tchotchkes (low poly Pokemon , succulent pots, etc) while you’re getting an understanding of printing.
    • this helps build out the very practical “how does something go from a file to an object” as well as what polymers exist for printing, what their trade-offs are, how things can fail etc.
  2. print out some useful things
    • all the sudden you find out you can print a hinge, or a clip or a vice or something does something. For me, the big eye opener was cases for Raspberry Pi models and custom gears
  3. design something custom
    • the printing itself is no longer the barrier, it’s CAD skills. The world opens up as you can now attach anything to anything with a custom, perfect fit mount. You can build gear boxes with a coupler that fits whatever random cheap motor you’re able to find, powered by a PSU that’s safely enclosed within a custom printed case and mounted robustly without having to drill or tap or cut.

With that said, I picked up a Prusa MK3:

https://makezine.com/wp-content/uploads/2017/09/Prusa_i3_mk3_White-1.jpg
Buy this!

It’s roughly the #1 most enjoyable thing I’ve ever spent money on. Upon spending ~1100$ CAD (after shipping/taxes/exchange rate), the only thing I regretted was not buying it sooner.

Inky wHAT with Raspberry Pi 4

If you have a Raspberry Pi 4 and an Inky wHAT ePaper/eInk/EPD, you’ll likely want to use one with the other at some point. Here’s everything I did from opening the box of the Raspberry Pi and Inky wHAT to running the example code:

  • Assemble the required ingredients:
    • Raspberry Pi 4
    • MicroSD card + SD card adapter + SD card reader (laptop maybe?)
    • USB-C cable and 5V power supply
  • Download a SD card flashing tool, e.g. from https://www.raspberrypi.org/downloads/
  • Run the flashing tool, and flash an OS, I used Raspberry Pi OS Lite (Raspbian?)
  • Before ejecting the SD card, set up your Wi-Fi connection and SSH
  • For SSH:
    • Create an empty file named “ssh” (no extension) in the SD card’s boot partition
  • For Wi-Fi:
    • Create a file “wpa_supplicant.conf” in the SD card’s boot partition with your configuration
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=<Insert 2 letter ISO 3166-1 country code here, e.g. CA for Canada>

network={
    ssid="<Name of your wireless LAN>"
    psk="<Password for your wireless LAN>"
}
  • Eject the SD card from your computer
  • Put the SD card in the Pi
  • Connect the Pi to power
  • Wait a minute or two
  • Determine the IP address of the Pi as it joins your Wi-Fi
  • ssh into your Pi
    • e.g. ssh pi@192.168.1.215 with password ‘raspberry
  • Generally I update the PI first thing, just so I know where I stand
    • sudo apt update
    • sudo apt upgrade -y
    • sudo apt autoremove
    • sudo reboot
  • Now, enable SPI and also I²C (I’m not sure why it needs both)
Select ‘Interfacing Options’
Select ‘SPI’
Enable it! Repeat for I2C
  • Now on to the Pimoroni aspect!
  • They recommend the ol’ magic script, which is a great way to start:
    • Execute: curl https://get.pimoroni.com/inky | bash
    • Opt to include the documentation and examples
    • Wait a bit, restart the Raspberry Pi again
  • Now, the moment of truth, run one of the included examples
    • change directory into ~/Pimoroni/inky/examples
    • sudo python name-badge.py --type "what" --colour "red" --name "Inigo Montoya"
    • Note that “sudo” is required because you’re accessing the GPIOs
    • Also note the British spelling of “colour” – at this time “color” does not work

It looks absolutely fantastic in person. I love eInk displays, so I’m jacked up about this. Now time to create my own images!

Incubator build: Part 3 – build something

With the particularly sketchy parts of both the hardware and software “figured out”, I was now ready to actually build something that might some day hold eggs. Mostly this consisted of leaning together some spare materials I found, with just enough tape and fasteners to keep it from falling over. The BOM for this is:

  • an old single pane window
  • a 2×10 cut into 6 pieces such that it makes a rectangle (4 sides) and a bottom (2 pieces)
  • a piece of scrap vapor barrier
  • a NodeMCU ESP32 dev board
  • a 12V computer fan
  • a breadboard
  • two DHT22 sensors
  • a lightbulb (with fixture)
  • a loaf pan (for water)
  • an unused computer power supply (never to be used again after this…)
  • a smattering of Wago connectors and wires
  • a transistor
  • a bunch of screws and some tape
  • some literal garbage, to “mount” things (an empty cardboard box for some staples or something, and a broken off piece of wood?)
  • an extension cord with the female end cut off

I wish I had taken more pictures, because I was super proud at the time, but this is the only one I could find.

20180408_205442
It’s not pretty, but it sure is a box!

As you can see, this is an electrical hazard, it’s filthy, and barely held together. It was also among the coolest things I’d ever built at that point. It seemed to work half decently – as long as the water was kept topped up, the humidity seemed pretty stable. With no real thermal mass except for the air and the small amount of water, the temperature fluctuated quite a bit. By virtue of the temperature fluctuating, the relative humidity also fluctuated significantly. Either way, it was a bit hard to tell exactly how it was performing because I was just dumping the readings out into the terminal and publishing them as MQTT messages (nothing was subscribed).

Automatic egg turner

At the same time as I reached the limits of my very basic understanding of electrical components, I was struggling to come up with a way to make an egg turner. I knew I wanted to automatic egg turner to be a part of this build, but I couldn’t figure it out. There are a bunch of recommendations on the internet (e.g. use the old style car antenna motor), but they were not particularly accessible (in terms of availability or price point). Sure, it’s a great tip to use an old antenna motor from the scrap heap, but it’s not that great a tip if you don’t have old antenna motors sitting around. Intentionally ordering something that’s only recommended because it’s cheap/accessible would be like buying used pallets to make woodworking product. It just misses the point.

Embarrassingly, this is about as close as I got. The idea was to mount a little wooden sled on the drawer pull, and have that push back and forth. It “worked” with some very generous interpretations of the word…

20180505_140918
It’s not pretty, but it… also doesn’t really work that well

Assembling robust mechanisms that do even simple things is also hard! This is well outside my domain, so I don’t even know where you go on the Internet to look up “DIY convert rotational movement into oscillating linear movement”. Ironically, searching that exact phrase leads to some pretty great resources. At the time I didn’t realize that’s what I wanted though. Even if you DO find something, you have to build it in a way that doesn’t suck.

And this is where the whole project truly became hideously sidetracked…

Digi-Key/Mouser vs. Adafruit/SparkFun

While trying to build an incubator, I very quickly ran into the limits of my very basic electrical component understanding. As I tried to tackle new problems, I continually had to do new things. Specifically, one of the issues I had was DHT22 sensors locking up. The easiest/best thing I could think of was to use a transistor to toggle the power supplied to the sensors, so my micro controller could reboot them when necessary.

Getting that transistor set up was the most complicated electronics thing I had done to date. When you don’t understand the theory, everything is hard. It’s easy to forget once you know everything, but electronic components are bruuuuutally complicated when you don’t know anything. As soon as anyone mentions PNP vs NPN, you’re off the deep end.

I wanted a switch I could control with a GPIO, I didn’t want a crash course in electrical theory. When you know, you can quickly make a wide variety of components work (or at least rule them out). When you don’t know, you can’t even figure out what area of component you should be looking up to buy. Add in the lead time for getting those new components, only to find out when you go to use them that they’re not compatible with something else you ordered a month ago, and it’s a rough go. I think there’s something missing between the SparkFun/Adafruits of the world and the Mouser and Digi-Keys of the world. One end of the spectrum is basically premium kit builds with a curated component selection, the other end is a ridiculously overwhelming parts list of ultra specific items.

As an example, here’s an example of the search results for “transistor” on Mouser (it actually leads to the “All Products | Semiconductors | Discrete Semiconductors | Transistors” section):

Capture
are these even words?!

And here’s most of the first page of results for “transistor” on Adafruit:

Capture
yes, the first 8 results were for t-shirts

So, do I start poking through the 40 000 things that are “transistors” and try and figure out what I actually want? It’ll take me multiple evenings to have a basic knowledge of the dozen words used to categorize them. Or do I start trying to find an Adafruit project that happens to line up with what I’m doing and therefore suggests a specific part? Neither is particularly enticing.

To be clear: I love all these sites. If you want a specific thing, Mouser/Digi-Key are absolutely invaluable. If you want to get jacked up about technology as hobby and get project ideas, Adafruit and SparkFun are fantastic resources. Adafruit and SparkFun also produce a ton of amazing “tutorial” style content as well. If you’re somewhere in the middle though and you don’t already have the knowledge, you’re a bit boned. I had a theoretical idea what I wanted, but it wasn’t an existing project, so now what?!

Maybe there’s an opportunity for something in the middle – a “choose your own adventure” style catalog that asks some rudimentary questions and guides you to an appropriate selection of components. Or maybe the answer is to just learn enough to be able to make the decision myself…

Xplornet modem default credentials

Default username: admin
Default password: xplornet4g
Default IP: 192.168.209.1

Occasionally it seems appropriate to reboot the Xplornet modem. Whether it’s because it’s actually required or the Internet connection is so poor that you’re flailing for something to do to try and fix it, it’s marginally more convenient to restart remotely than going and unplugging it.

Xplornet tends to install their equipment with a default configuration, including default credentials. Note that if you muck with the settings, you could easily cause yourself to lose your Internet connection and need a service call to repair it.

Incubator build: Part 2 – hardware AND software issues

I had successfully (maybe with an asterisk or two) kept a plastic tub roughly at a stable temperature. On to plastic tub v2! This is getting into mid-2018, and I’m starting to have a rough sense of things but for the most part I don’t know anything. Still using an ESP8266 and still using the Arduino wrapper. By this point, my Arduino program is getting pretty unwieldy, but it’s still less than 1000 lines. I had no idea how Arduino programs generally look, so it was pretty stream-of-consciousness with plenty of direction changes.

The original plastic tub had been scrapped (something about a giant hole melted in the side?), but luckily I had another. It was identical, but I’d learned my lesson. Ensuring the heater doesn’t burn anything down is priority number one. In the longer term, it was clear an incandescent light bulb wasn’t going to be the best heater for a number of reasons:

  • they get too hot (over 100°C easily)
  • they’re a weird shape, as heat is a byproduct and not the main use
  • they’re just not easily available anymore (that’s weird to say…)
  • they put off a ton of light, which doesn’t necessarily have an incubator-related downside, but it dissuades me from looking at the thing

I ordered a bunch of different styles of heaters, but they wouldn’t be showing up for a long time (almost everything I get is the cheapest version of itself and comes from China in 1 – 2 months). While a light bulb wasn’t ultimately going to be the goal, it’s what I had available to me. To make it safer I picked up one of these from the local home improvement store:

light-bulb-fixture

This was dramatically safer than my old light bulb holder, as by its very nature things are less likely to be close to it. I still just rested it on the bottom of the tub, but now it could be centered to maximize the distance from the sides.

When things go pear shaped – software side

Note that at this point while the project is keeping a relatively “fixed” heat, that’s pretty secondary. So much other stuff is going wrong that the fact it’s intended to be an incubator is virtually irrelevant. At this stage, I ran into two significant problems. First was a nefarious software problem. I had blown past my previous record run of ~16 continuous hours of operation and was closing in on 30 hours before catastrophic malfunction. For some reason, after just under 30 hours my microcontroller would reboot.

I’ve been working in Java for years, and I’ve honestly never really thought about memory. Even when manually allocating memory in school, it wasn’t for anything real – if you got it wrong you lost marks or something. You might be able to guess where this is headed – I had a very slow memory leak. I was pointed towards the possibility of a memory leak with the overly kind help of Internet stranger Rob Tillaart (really, an absolute gem who just so happened to be the author of a library I was using).

I started logging the free memory on the device. After 5 or 10 minutes, it had stayed rock steady, so I was quickly dismissing that as the culprit. Then the unthinkable happened. I checked back a couple hours later and the free memory had dropped. Not by a lot, but by some. That meant that something somewhere was allocating memory and not freeing it, even though I didn’t intentionally do so anywhere.

Reviewing my logs, it looked like roughly every 10 minutes it leaked 240 bytes. Always 240 bytes. Coming from web application development, who even cares about 240 bytes?! Turns out I do, in fact I very much do. With ~41000 bytes free on the ESP8266, that means every ~28.8 hours it would hit an out of memory error and restart. With some data, I was able to quickly narrow in on the issue. Every time I send the NTP packet (once every 10 minutes) it leaked 240 bytes. I never actually looked at the root cause, but it seemed like sending UDP data but not reading the responses (I was ignoring some content I didn’t care about) would keep the received memory allocated.

Fixing that code up, the memory leak was resolved. Lesson learned: always track memory consumption in development!

When things go pear shaped – hardware side

Clearing the ~29 hour memory leak induced hard limit for execution time let me run into my next failure. I was using DHT22 temperature and humidity sensors. If you don’t know what you’re doing and you’re in the Arduino world, these are ubiquitous. They’re also pretty much trash, in my experience. They work a little but there’s just no reason for them to exist when other sensors exist. The sensor can be had for ~3$ a piece, has a ±0.5°C temperature accuracy, and a 2 second sensing period. They’re not objectively terrible, and I’m sure someone more competent could make them work more reliably, but there are just so many alternatives that either have better performance, better price, better reliability, or all of the above.

dht22

My scheme was this: I didn’t want to actually go to any effort to calibrate the sensors, so if I put a bunch of them in, discarded obvious outliers and averaged the remaining values I was hoping I could get some reasonable consistency. I was reading each sensor every 2 seconds, sending the reading off to a server, then repeating. That’s when the sensor failures started to kick in…

Following another recommendation from Rob Tillaart, I tried swapping the original 10k resistors for 4.7k and started tracking specifically what the failures were. These represent the interactions with a given sensor. Here’s a snapshot:

Total			OK	Checksum error	Timeout		Unknown	
( 10k) 1: 48950		48929	21		0		0
( 10k) 2: 48950		48354	596		0		0
(4.7k) 3: 48950		46612	2336		0		0	
(4.7k) 4: 48950		47447	1503		0		0	

As you can see, it’s just shy of 50000 reads across 4 sensors – 2 with 10k resistors and 2 with 4.7k. The best sensor had a ~0.04% error rate while the worst performing sensor had a whopping 4.77% error rate! The obvious conclusion here is this is a hardware implementation/user issue and not a sensor issue, but it’s still a pain.

Ignoring the checksum errors (which just manifest as a failed read), the real issue is that 100% of the time the sensors would lock up and timeout. That is to say, at an arbitrary seeming time, every DHT22 sensor in every configuration I tried eventually locked up. I had purchased a few different form factors and manufacturers as well, so this seemed like a fundamental issue somewhere.

While I’d absolutely shattered the 28 hour continuous run record, sensors consistently started to drop off one-by-one and I couldn’t get any reading much into the second day. The program still ran fine, just not temperature or humidity data coming in. After floundering on this issue for a week or so, I ended up just cutting the power to the sensors when a timeout was detected. This gave them a chance to reboot, and seemed to work robustly enough to move forward. Again, I didn’t discover the root cause of this and enough changed over time that I’m not sure specifically what the issues were, but in my opinion in decreasing likelihood:

  • using a breadboard to hook things up (these are the cause of SO MANY of my problems)
  • using 3.3V logic – I got a logic level shifter eventually
  • timing interference from WiFi or other ESP8266 shenanigans
  • the ESP8266 itself – I got a bunch of ESP32 modules and have been using those since

Lessons learned:

  • always track your memory usage during development, memory leaks can be slow and come from unexpected places
  • just… don’t use breadboards. If anything at all is going wrong and you’re using a breadboard, assume the breadboard is the issue until proven otherwise
  • even sensors can fail! Robust software/electronics design has to accept that failure (e.g. by not burning anything down) and deal with it (e.g. by rebooting the sensor)