Incubator build: Part 3 – build something

With the particularly sketchy parts of both the hardware and software “figured out”, I was now ready to actually build something that might some day hold eggs. Mostly this consisted of leaning together some spare materials I found, with just enough tape and fasteners to keep it from falling over. The BOM for this is:

  • an old single pane window
  • a 2×10 cut into 6 pieces such that it makes a rectangle (4 sides) and a bottom (2 pieces)
  • a piece of scrap vapor barrier
  • a NodeMCU ESP32 dev board
  • a 12V computer fan
  • a breadboard
  • two DHT22 sensors
  • a lightbulb (with fixture)
  • a loaf pan (for water)
  • an unused computer power supply (never to be used again after this…)
  • a smattering of Wago connectors and wires
  • a transistor
  • a bunch of screws and some tape
  • some literal garbage, to “mount” things (an empty cardboard box for some staples or something, and a broken off piece of wood?)
  • an extension cord with the female end cut off

I wish I had taken more pictures, because I was super proud at the time, but this is the only one I could find.

20180408_205442
It’s not pretty, but it sure is a box!

As you can see, this is an electrical hazard, it’s filthy, and barely held together. It was also among the coolest things I’d ever built at that point. It seemed to work half decently – as long as the water was kept topped up, the humidity seemed pretty stable. With no real thermal mass except for the air and the small amount of water, the temperature fluctuated quite a bit. By virtue of the temperature fluctuating, the relative humidity also fluctuated significantly. Either way, it was a bit hard to tell exactly how it was performing because I was just dumping the readings out into the terminal and publishing them as MQTT messages (nothing was subscribed).

Automatic egg turner

At the same time as I reached the limits of my very basic understanding of electrical components, I was struggling to come up with a way to make an egg turner. I knew I wanted to automatic egg turner to be a part of this build, but I couldn’t figure it out. There are a bunch of recommendations on the internet (e.g. use the old style car antenna motor), but they were not particularly accessible (in terms of availability or price point). Sure, it’s a great tip to use an old antenna motor from the scrap heap, but it’s not that great a tip if you don’t have old antenna motors sitting around. Intentionally ordering something that’s only recommended because it’s cheap/accessible would be like buying used pallets to make woodworking product. It just misses the point.

Embarrassingly, this is about as close as I got. The idea was to mount a little wooden sled on the drawer pull, and have that push back and forth. It “worked” with some very generous interpretations of the word…

20180505_140918
It’s not pretty, but it… also doesn’t really work that well

Assembling robust mechanisms that do even simple things is also hard! This is well outside my domain, so I don’t even know where you go on the Internet to look up “DIY convert rotational movement into oscillating linear movement”. Ironically, searching that exact phrase leads to some pretty great resources. At the time I didn’t realize that’s what I wanted though. Even if you DO find something, you have to build it in a way that doesn’t suck.

And this is where the whole project truly became hideously sidetracked…

Digi-Key/Mouser vs. Adafruit/SparkFun

While trying to build an incubator, I very quickly ran into the limits of my very basic electrical component understanding. As I tried to tackle new problems, I continually had to do new things. Specifically, one of the issues I had was DHT22 sensors locking up. The easiest/best thing I could think of was to use a transistor to toggle the power supplied to the sensors, so my micro controller could reboot them when necessary.

Getting that transistor set up was the most complicated electronics thing I had done to date. When you don’t understand the theory, everything is hard. It’s easy to forget once you know everything, but electronic components are bruuuuutally complicated when you don’t know anything. As soon as anyone mentions PNP vs NPN, you’re off the deep end.

I wanted a switch I could control with a GPIO, I didn’t want a crash course in electrical theory. When you know, you can quickly make a wide variety of components work (or at least rule them out). When you don’t know, you can’t even figure out what area of component you should be looking up to buy. Add in the lead time for getting those new components, only to find out when you go to use them that they’re not compatible with something else you ordered a month ago, and it’s a rough go. I think there’s something missing between the SparkFun/Adafruits of the world and the Mouser and Digi-Keys of the world. One end of the spectrum is basically premium kit builds with a curated component selection, the other end is a ridiculously overwhelming parts list of ultra specific items.

As an example, here’s an example of the search results for “transistor” on Mouser (it actually leads to the “All Products | Semiconductors | Discrete Semiconductors | Transistors” section):

Capture
are these even words?!

And here’s most of the first page of results for “transistor” on Adafruit:

Capture
yes, the first 8 results were for t-shirts

So, do I start poking through the 40 000 things that are “transistors” and try and figure out what I actually want? It’ll take me multiple evenings to have a basic knowledge of the dozen words used to categorize them. Or do I start trying to find an Adafruit project that happens to line up with what I’m doing and therefore suggests a specific part? Neither is particularly enticing.

To be clear: I love all these sites. If you want a specific thing, Mouser/Digi-Key are absolutely invaluable. If you want to get jacked up about technology as hobby and get project ideas, Adafruit and SparkFun are fantastic resources. Adafruit and SparkFun also produce a ton of amazing “tutorial” style content as well. If you’re somewhere in the middle though and you don’t already have the knowledge, you’re a bit boned. I had a theoretical idea what I wanted, but it wasn’t an existing project, so now what?!

Maybe there’s an opportunity for something in the middle – a “choose your own adventure” style catalog that asks some rudimentary questions and guides you to an appropriate selection of components. Or maybe the answer is to just learn enough to be able to make the decision myself…

Xplornet modem default credentials

Default username: admin
Default password: xplornet4g
Default IP: 192.168.209.1

Occasionally it seems appropriate to reboot the Xplornet modem. Whether it’s because it’s actually required or the Internet connection is so poor that you’re flailing for something to do to try and fix it, it’s marginally more convenient to restart remotely than going and unplugging it.

Xplornet tends to install their equipment with a default configuration, including default credentials. Note that if you muck with the settings, you could easily cause yourself to lose your Internet connection and need a service call to repair it.

Incubator build: Part 2 – hardware AND software issues

I had successfully (maybe with an asterisk or two) kept a plastic tub roughly at a stable temperature. On to plastic tub v2! This is getting into mid-2018, and I’m starting to have a rough sense of things but for the most part I don’t know anything. Still using an ESP8266 and still using the Arduino wrapper. By this point, my Arduino program is getting pretty unwieldy, but it’s still less than 1000 lines. I had no idea how Arduino programs generally look, so it was pretty stream-of-consciousness with plenty of direction changes.

The original plastic tub had been scrapped (something about a giant hole melted in the side?), but luckily I had another. It was identical, but I’d learned my lesson. Ensuring the heater doesn’t burn anything down is priority number one. In the longer term, it was clear an incandescent light bulb wasn’t going to be the best heater for a number of reasons:

  • they get too hot (over 100°C easily)
  • they’re a weird shape, as heat is a byproduct and not the main use
  • they’re just not easily available anymore (that’s weird to say…)
  • they put off a ton of light, which doesn’t necessarily have an incubator-related downside, but it dissuades me from looking at the thing

I ordered a bunch of different styles of heaters, but they wouldn’t be showing up for a long time (almost everything I get is the cheapest version of itself and comes from China in 1 – 2 months). While a light bulb wasn’t ultimately going to be the goal, it’s what I had available to me. To make it safer I picked up one of these from the local home improvement store:

light-bulb-fixture

This was dramatically safer than my old light bulb holder, as by its very nature things are less likely to be close to it. I still just rested it on the bottom of the tub, but now it could be centered to maximize the distance from the sides.

When things go pear shaped – software side

Note that at this point while the project is keeping a relatively “fixed” heat, that’s pretty secondary. So much other stuff is going wrong that the fact it’s intended to be an incubator is virtually irrelevant. At this stage, I ran into two significant problems. First was a nefarious software problem. I had blown past my previous record run of ~16 continuous hours of operation and was closing in on 30 hours before catastrophic malfunction. For some reason, after just under 30 hours my microcontroller would reboot.

I’ve been working in Java for years, and I’ve honestly never really thought about memory. Even when manually allocating memory in school, it wasn’t for anything real – if you got it wrong you lost marks or something. You might be able to guess where this is headed – I had a very slow memory leak. I was pointed towards the possibility of a memory leak with the overly kind help of Internet stranger Rob Tillaart (really, an absolute gem who just so happened to be the author of a library I was using).

I started logging the free memory on the device. After 5 or 10 minutes, it had stayed rock steady, so I was quickly dismissing that as the culprit. Then the unthinkable happened. I checked back a couple hours later and the free memory had dropped. Not by a lot, but by some. That meant that something somewhere was allocating memory and not freeing it, even though I didn’t intentionally do so anywhere.

Reviewing my logs, it looked like roughly every 10 minutes it leaked 240 bytes. Always 240 bytes. Coming from web application development, who even cares about 240 bytes?! Turns out I do, in fact I very much do. With ~41000 bytes free on the ESP8266, that means every ~28.8 hours it would hit an out of memory error and restart. With some data, I was able to quickly narrow in on the issue. Every time I send the NTP packet (once every 10 minutes) it leaked 240 bytes. I never actually looked at the root cause, but it seemed like sending UDP data but not reading the responses (I was ignoring some content I didn’t care about) would keep the received memory allocated.

Fixing that code up, the memory leak was resolved. Lesson learned: always track memory consumption in development!

When things go pear shaped – hardware side

Clearing the ~29 hour memory leak induced hard limit for execution time let me run into my next failure. I was using DHT22 temperature and humidity sensors. If you don’t know what you’re doing and you’re in the Arduino world, these are ubiquitous. They’re also pretty much trash, in my experience. They work a little but there’s just no reason for them to exist when other sensors exist. The sensor can be had for ~3$ a piece, has a ±0.5°C temperature accuracy, and a 2 second sensing period. They’re not objectively terrible, and I’m sure someone more competent could make them work more reliably, but there are just so many alternatives that either have better performance, better price, better reliability, or all of the above.

dht22

My scheme was this: I didn’t want to actually go to any effort to calibrate the sensors, so if I put a bunch of them in, discarded obvious outliers and averaged the remaining values I was hoping I could get some reasonable consistency. I was reading each sensor every 2 seconds, sending the reading off to a server, then repeating. That’s when the sensor failures started to kick in…

Following another recommendation from Rob Tillaart, I tried swapping the original 10k resistors for 4.7k and started tracking specifically what the failures were. These represent the interactions with a given sensor. Here’s a snapshot:

Total			OK	Checksum error	Timeout		Unknown	
( 10k) 1: 48950		48929	21		0		0
( 10k) 2: 48950		48354	596		0		0
(4.7k) 3: 48950		46612	2336		0		0	
(4.7k) 4: 48950		47447	1503		0		0	

As you can see, it’s just shy of 50000 reads across 4 sensors – 2 with 10k resistors and 2 with 4.7k. The best sensor had a ~0.04% error rate while the worst performing sensor had a whopping 4.77% error rate! The obvious conclusion here is this is a hardware implementation/user issue and not a sensor issue, but it’s still a pain.

Ignoring the checksum errors (which just manifest as a failed read), the real issue is that 100% of the time the sensors would lock up and timeout. That is to say, at an arbitrary seeming time, every DHT22 sensor in every configuration I tried eventually locked up. I had purchased a few different form factors and manufacturers as well, so this seemed like a fundamental issue somewhere.

While I’d absolutely shattered the 28 hour continuous run record, sensors consistently started to drop off one-by-one and I couldn’t get any reading much into the second day. The program still ran fine, just not temperature or humidity data coming in. After floundering on this issue for a week or so, I ended up just cutting the power to the sensors when a timeout was detected. This gave them a chance to reboot, and seemed to work robustly enough to move forward. Again, I didn’t discover the root cause of this and enough changed over time that I’m not sure specifically what the issues were, but in my opinion in decreasing likelihood:

  • using a breadboard to hook things up (these are the cause of SO MANY of my problems)
  • using 3.3V logic – I got a logic level shifter eventually
  • timing interference from WiFi or other ESP8266 shenanigans
  • the ESP8266 itself – I got a bunch of ESP32 modules and have been using those since

Lessons learned:

  • always track your memory usage during development, memory leaks can be slow and come from unexpected places
  • just… don’t use breadboards. If anything at all is going wrong and you’re using a breadboard, assume the breadboard is the issue until proven otherwise
  • even sensors can fail! Robust software/electronics design has to accept that failure (e.g. by not burning anything down) and deal with it (e.g. by rebooting the sensor)

 

 

Convert UTC datetime to timezone in Grafana

I imagine this is one of those super commonsense things that anyone who works with databases knows, but I didn’t. In my database I have a bunch of timestamps – specifically in Postgres I have columns with type timestamp with time zone. In Grafana, the time column is selected appropriately, and ends up in UTC. I couldn’t figure out a way for Grafana to convert the data (I’d love to hear it if there is). Turns out it’s trivial to edit the query to convert the timestamp. For example:

SELECT
  created_At AT TIME ZONE 'America/Toronto' AS "time",
  temperature
FROM temperature
WHERE
  $__timeFilter(client_time)
ORDER BY 1

No more mental gymnastics while looking at my dashboards! Small quality of life improvement for me, plus learned a little tiny bit.

Incubator build: Part 1 – Warm a tub

Circa early 2018, I was able to stitch together enough Arduino example code to get an ESP32 program that did the following:

  • connected to Wi-Fi
  • set the time via NTP
  • read two DHT22 temperature/humidity sensor
  • publish temperature, humidity, and time of the reading over MQTT
  • toggle a pin when heat got hotter than a threshold, and again when it got cooler than a threshold

That sounds like it’s in the realm of an incubator. It can’t adjust the humidity itself, but it can at least relay that information. By varying the size of container of water inside the incubator, an operator could theoretically get pretty close to a controlled humidity level experimentally.

This was profoundly exciting for me. I was checking off all sorts of stuff: I started completely confounded, figured out some tiny piece, and repeated:

  • I had figured out how to flash a microcontroller
  • I’d learned very high level concepts around embedded program design
  • I figured out how to get the CURRENT time on a microcontroller
    • it’s so much harder than it sounds, and it’s nothing I’ve ever had to do before
  • I’d learned about MQTT, Eclipse Mosquitto
  • some GPIO stuff was starting to make a little bit of sense, but not really
  • I was getting a very rough idea about voltage and current
    • is 50mA a lot? Of course not, that’s a tiny amount of current! Oh, from a GPIO pin? Yeah, that’s WAY too much
  • I had learned the difference between 3.3V logic and 5V logic, and practically speaking what that means
    • a common ground is obvious in retrospect, but not if you don’t know anything!
  • I’d also learned how to control a transistor to act as a digital switch

Sensing that I was close to accomplishing my dream of building an incubator, I starting the hardware build. Build 0 was a “Tuff Store” tub. It was < 5$ at Walmart, and I already had it in the house, so it fit the bill.

zoomed image

I literally dangled two DHT22 sensors off their leads and added a light bulb in a socket harvested from a broken flood light. This was attached to an ESP8266 board sitting on top, with a transistor to control a mechanical relay to turn the light bulb on and off.

20200501_125900

As you can notice, there’s no real mount for that light bulb socket. Also it looks as though there’s some discoloration on the one side of the socket. This will become relevant shortly.

As far as boxes that are warmer than room temperature goes, this one was absolutely crushing it. The occasional “tick” of the mechanical relay became reassuring background noise that everything was working as intended. Actually… that may be a bit boastful – it was alright. There were a few areas for improvement:

  • incandescent light bulbs (the ones that get hot) are hard to come by these days!
  • mounting anything to smooth plastic is tough. The options are generally drill permanent holes, use an adhesive that barely works, or use an adhesive that is permanent
  • the thin walls provided very little insulation, so heat transferred out of the box pretty readily

That said, it worked. The Arduino program was:

  • connect to Wi-Fi
  • if it had been long enough, sync the time with NTP
  • read the temperature and humidity from the DHT22
  • post temperature and humidity to an MQTT broker
  • if above a threshold temperature, turn lightbulb off. If below a threshold temperature, turn lightbulb on
  • repeat

It worked pretty much as intended. I didn’t have any way of persisting the readings, so it was hard to profile how it performed, but watching the temperature come in on an MQTT subscriber things looked pretty solid. I gradually let it sit for longer and longer. Being keenly aware of how little I know about electronics, and how shoddily my build was put together, I was cautious to not let it sit completely unattended. This ended up being prudent.

My very rudimentary Arduino program blocked while connecting to Wi-Fi. What I didn’t anticipate was that Wi-Fi wasn’t 100% guaranteed to be accessible all the time. One day, our Wi-Fi went out. The “incubator” had been in a heating state, and the program was effectively stalled because it was waiting for Wi-Fi. This meant the heater (light bulb) stayed on way longer than intended, deformed the really sub-par mount I had rested the light bulb socket on (to keep it away from the side of the container), and melted clear through the side of the plastic tub. This resulted in a bunch of burned plastic, and a light bulb shaped hole in the container.

With that, build 0 ended in catastrophic disaster. The maximum consecutive run time was around 16 hours – far short of the required 21 days. Roughly 6$ down the drain (5$ tub and 1$ light bulb). Lessons learned:

  • hobby electronics can turn into a house fire horrifyingly easily
  • even though the focus of the project was on the electronics, I couldn’t take for granted that the software would still be significant
  • “bad” decisions are okay, as long as they’re deliberate and not made recklessly
    • it’s valuable to be honest with yourself about how bad you are at things, so you can mitigate the impact of those things going horrifically wrong

Incubator build: Part 0 – Introduction

I want to build an incubator for chicken/duck/general fowl eggs. More than that, I want to actually finish the project (hatch out birds) AND I want the incubator to not suck. All 3 of these goals are particularly challenging for me, as I tend to lose interest in a project once it’s ~80% complete and never actually get to the part where it works, or it will kinda work but be fragile and miss the target. Additionally, the difference between the MVP of something and a version that is “nice” is absolutely massive. Part of that is generally re-starting some aspect, as now you actually understand the problem you’re trying to solve. The last few years are littered with the corpses of projects I have abandoned at various stages, most of them depressingly expensive (in both time and money) and depressingly incomplete.

I started trying to build an incubator over 2 years ago. The very first incubator-looking code I produced was from March of 2018. For people who know anything about electronics or hardware, that’s a bit preposterous. It’s really not that hard – incubators are pretty simple. Unless, of course, you know nothing about electronics, or soldering, or embedded development or incubating or… That’s where I started.

Through a whole bunch of trial and error, the first ever version kind of worked. I was starting with the ESP32 microcontroller. Knowing nothing about anything, I started with the Arduino wrapper for ESP32. This got me up and running pretty quickly, and there’s some kind of library for virtually every sensor you’re likely to come across. That said, the quality of some of the libraries can be pretty suspect, and the interface exposed by them often is not quite what you want. Of course, good luck determining the wheat from the chaff when you have no idea what you’re doing.

Overall, the theme of the project seems to have been: even if you’re peripherally familiar with something, having to actually implement it and make it robust from a hardware/electronics perspective means you have to actually UNDERSTAND it. Not necessarily all the way down to the physics of it, but way more intimately than what is required for web application development (which I’m familiar with). Some of the stuff I either encountered for the first time, or actually had to start trying to understand at a practical level:

  • voltage, resistance, current, power, grounding, and (much, much later on) capacitance
  • soldering, crimping, terminals, solid vs. stranded wire, wire gauge, breadboards, prototoboards
  • power supplies, stepper motors, logic level shifting, transistors, regulators, mechanical and solid state relays
  • analog sensors, digital sensors, serial communication, SPI, I²C
  • volatile vs. non-volatile memory, microcontroller power management
  • Arduino environment, ESP-IDF, cmake
  • MQTT, Eclispe Mosquitto, Postgres (TimescaleDB), Grafana, CAD, 3D printing

I fully realize people successfully hatch eggs with a cooler, an incandescent light bulb, and a container of water. As I stated at the beginning; I wanted to do this, I wanted to do it well, and I wanted to finish it. That meant a significant portion of this was straight up learning.