N Ca Mayflys through the winter storms

Following on from the phased deployment  envirodiy.org/mayflys-in-coastal-redwoods-northern-california I’m pleased the way my fork of ModularSensors has been working with the winter storms, in incised  riparian locations with lots of vegetation around and challenging cell phone connections. The real-world in streams on the West Coast USA.

Though when I dig deeper into some of the nodes data – they aren’t behaving as well as I would like – still some bugs hiding somewhere – is it software or hardware?. Computers are supposed to be reliable and predictable, but turns out they can have “character” – or bugs ~ and technically the end-2-end systems need to be characterized for when and how the bugs  show up.

To recap, I’ve modified ModularSensors to be more reliable. West Coast Winter time in the Redwoods is one of the great challenges for solar powered sensors – as the storms roll in, the solar harvesting dips for a number of days to weeks. For some systems, like WiFi in somebodies rural building, when they loose power, the network disappears for a week on end.

I’m happy that when the temporary disruptive environmental conditions subside, the stored readings are chugged up to MonitorMyWatershed.org – and the only trace of it is looking at the low battery voltages.

Sierra Magazine did a nice description of the context sierraclub.org/sierra/coho-salmon-warm-springs-dam-dry-creek , and there are many groups supporting the recovery of salmon and improved ecological integrity of the upper watersheds of rivers redwoodempire-tu.org/russian-river

One aspect is the TU California hydrologists monitoring flow in the smaller tribs of the main stem of the rivers. That is taking stage readings every 15minutes, and periodically taking flow readings to derive a  flow curve. The stage gauge used is the Insitu LT500 and has been reliable and repeatable over a number years. The LT500 builds in a logger so can be downloaded by “bootnet” – walking up to it and plugging a computer to slurp the readings.  Then there is the benefit to have the stage reported on  monitormywatershed.org/browse  , so nice!

I’m a great fan of MonitoMyWatershed. I also have a lot of experience in the field of engineering small microcomputers – so I guess, I’m also a technical critic. Providing technical insights! Sometimes making observations that I really don’t want to make, doing it from a technical honesty. 

Computer systems components from hardware to “software” are typically categorized into layers. So, there is the hardware layer and then software “layers” (also sometimes discussed as C++ Classes).

MMW is an instance of ODM2DataSharingPortal , and has a communication channel – tcp/ip opened from a field node, and data POSTed. The firmware running on the field node, a Mayfly or other node, needs to create this POST and manage the readings taken on the device. Modular Sensor  sensor equipment interface is fantastic for the range of sensors and protocols. I modified the Modbus/RS485 layer to work with the LT500 & Keller Nanolevel, files InsituTrollModbus.h and KellerNanolevel.h that works with modbus layer and pushed it back via a PR into the main repository for everyone. Mostly the subsystems “Classes”  have just worked or been easy to tweak!!. Classes like TinyGsm, interface to different hardware Xbee modems, and allowed flexibility of deployment. Though I made some tweaks to it, and its time consuming to debug.

Computer systems need testing, and the testing needs to emulate real world field conditions. The testing shows errors that need fixing. My experience leads me to generate one image – for the Mayfly a .HEX, test it, and fix bugs. Then deploy that tested, characterized image. This means I have one image that is used for all the systems with the same sensors. envirodiy.org/geographical-scaling-modularsensors

A real-world subsystem that is largely exercised in the field is powering. Sufficient power (milliWatts)  is required by the hardware to operate, and then a layer of software to manage when there is enough power for an activity, like turning on the modem.  For a specific technology to work, under specific circumstances software engineers refer to a “Use case” or “Edge condition”.   When I first started checking ModularSensors I identified that it must work for “Winter time low solar or no communications” and gracefully recover when the storm has passed. It must also be able to be configurable key site-specific parameters to optimize for specific well documented wireless field conditions. Analyzing the open source (yeah!!)  ModularSensors and Mayfly I was able to get a good grasp of it – and my heart sank with what I saw and tested. There was a lot of aspirational descriptions – implying it just worked – but not a lot of dealing with nitty gritty edge conditions.  It uses the Arduino programming framework, but IHMO the design doesn’t make it easy for the end user – as a system, I thought, it isn’t in the spirit of Arduino .  

Just taking powering, the real basics, and the Mayfly 1.x was a great improvement – it works with a nice solar aspect, but once the solar disappeared, and the battery has a low voltage – Mayfly 0.5 with 2Ahr would go into reset oscillations. During the pandemic of course, electronics hardware for devices like the Mayfly became stretched, issues came up with the Digi LTE modem, and another source was found – so generally it was amazing for the hardware technology, (and a lot of behind the scenes work) that Mayfly logging technology kept being available. I characterized for my field nodes I needed Mayfly 1.1 with 4Ahr battery and 3W solar panel.  I traded out my Mayfly 0.5b for Mayfly 1.1 and standardized on 4Ahr battery.

What did I want it to do that was so hard – my Winter Edge Condition, well I want all the deployed devices to gracefully cope with the West Coast winter storms – or any normal disappearance of the sun, or transmission networks, for two weeks. Seems like that is a reasonable expectation of what weather does and of course there are low solar winter days that need to be designed for. Then when conditions are normal, all the readings should be chugged reliably to MMW. The technical language to describe this potential effect on server I documented here  github.com/ODM2/ODM2DataSharingPortal/issues/485

An artistic view of asking gemin.google.com to “draw cloud server chugging data”

Of course, there are a lot of locations, where water freezes, sensors must be removed from water systems to prevent them being damaged. Not so on the west coast – the sensors are protected from frosts in the water – even though the battery and electronics might dip to minus 10C.  The sensors once deployed could run for years.

Each manufactured system has devices/Mayfly, LTE modem and enclosure that are unique, and the software (ModularSensors) expects the reading of hardware to be standardized.  For low solar times, winter storms and  cold battery conditions, it would be nice to pause the power-hungry LTE modem, and continue operating at a lower power, taking periodic readings (15 minutes) patiently waiting for more power to become available in the battery, that is  the sun comes back. Then gracefully deliver all the waiting readings.  I described that layer of ModularSensos software here github.com/EnviroDIY/ModularSensors/issues/194

Now it turns out this also solves another problem – of weak transmission systems and potentially short term server overloads.  When the POST doesn’t get a, thankyou ACK HTTP response “201 Created” for a reading, it queues it on the uSD. Pretty standard “network Layer” protocol.

For power management it is desirable, ModularSensors algorithms detect low battery power.

The LiIon battery has a stated range of 4.2V to 2.8V – the Mayfly Vbat doesn’t specify what range it measures accurately, that I have found of. The upper threshold is above 4.2V and should be OK. But the lower threshold needs to be accurately determined. Doing a quick circuit analysis Vbat lower accuracy threshold is likely 3.5V (mega1284 AVCC/Vref  depends on Vcc which is depends on actual LiIon voltage+~0.15V). As well if the battery is discharged, cold and weekly being charged, the Li-Ion battery voltage will be higher, but still unlikely to deliver enough power for a modem connection. Through my own work I calculated the lowest value from Vbat that can be relied on to deliver power once the load is turned on, is 3.6V with a light load, and 3.8V for a heavy modem load.  github.com/EnviroDIY/EnviroDIY_Mayfly_Logger/issues/32

One of my deployed systems really performed well under some very adverse conditions, low cellular and heavy LTE modem demands

monitormywatershed.org/sites/TUCA_GV01  It was initially set up to read every 15minutes, but send the readings ever hour. However the cellular conditions were so bad, that very few POSTs got through, and they were all queued on the uSD. So, on a visit last November, I installed an external antenna to improve the signal sensitivity, and manually through an uSD configuration ms_cfg.ini, I tweaked the POST rate to once every 15minutes. 

Boom!!  it started sending all the queued readings. Hurrah!.

Though it was still challenging to get the readings through to the server due to weak cellular signal strength and it quickly ran down the power to 3.8V. Due to my “reliability design” it was still taking and queuing readings. When the sun came back it started sending readings and eventually it caught up. Now I get an email from MMW to say sometimes it doesn’t get any readings for 12hours  – 48 attempts. Then it seems wind conditions improve, signal strength goes up, and it delivers a bunch of messages.  However it looks reliable – the readings are delivered – though sometime 12+ hours to get them all through.

Looking at the record of the battery voltage from Oct 1- the trail can be decoded

And the depth readings over the period are all reliably delivered. Even the “5 days of data loss” that other nodes experienced, where delivered from GV01 to MMW after the software upgrade, so are now present.

Some of the other systems also needed to be upgraded for the solar panels – as they are in the Redwoods, they had a shifting canopy pattern and I upgrade the solar panel to 10W. All the Mayfly 1.1 boards I use I configure to charge at 1A – though practically it’s only going to be an hour a day if lucky.

Even as an experienced programmer, it took me over 2000hrs to make this work and test it in my fork with ModularSensors. That’s the benefit of open source – being able to retrofit more features.

I can only manage my software – and very appreciative there is a cloud system to catch it and display it.

Knowing that there can be errors “bugs” in software subsystem, is part of the art of programming. I keep track of every POST made by the C++ Logger subsystem, and on visiting can retrieve a historical view. For active systems this is called “Characterization”. Early 2023 was a battle to make visible some slowdowns on the server, and then in October there was 5 days of data lost. Gone. I detected the loss 2months after it was actually lost.  It has shown that there is a learning curve still going on with ODM2 code and characterizing the responses. I did a heads up on potential issues I saw in 2020 and they didn’t get solved until it seemed like the system was close to collapsing in early 2023.

Practically real systems that can be tested, are deployed in the real-world that can’t be easily checked out till field deployment. Out of 9 systems that I built and then deployed by TU field hydrologists, only one was deployed and worked perfectly without me having to visit. This one system was installed 11-Sep-2023   monitormywatershed.org/sites/TUCA_MW12

A system running over WiFi – survived a 1 week power outage of the WiFi in February 2024, and now just looks perfect monitormywatershed.org/sites/TUCA_Sa01

One system was in a Verizon radio shadow – and I changed it to ATT and it started working – all without an upgrade of software – it required editing the ms_cfg.ini file and replacing the Digi Modem.

I should point out I’m an independent engineer, no connection with the Stroud Research Center and though I can suggest improvements to the open source software, the reality is I don’t have any input (or budget) in the process. 3 years after making a suggestion, the reality became visible in the MMW slowdowns in 2023.  There is a suggested fix github.com/ODM2/ODM2DataSharingPortal/issues/688. The fix involves MMW returning an http status response “202 Accepted” instead of the “201 Created”– which means I now have to upgrade all the systems in the field. A really simple solution in my ModularSensors sfotware, and painful to visit each system to upgrade. If the MMW had been tested and planned for scaling two years ago it wouldn’t be a problem. So still a learning curve for the MMW software.

My software for reliable delivery was started to be integrated into Modular Sensors by request mid-2023 github.com/EnviroDIY/ModularSensors/issues/194#issuecomment-1629331122 then aborted.  Reliability is hard to grok.