[Anthony Aufdenkampe]

Hi @brianjastram,

We noticed that same issue back in Sep with issue #455 Disappeared github.com/MartinL , and @srgdamiano posted a fix to the master branch with commit <tt>db7bfb0.</tt>

If you point your platformio.ini file to the repo’s master branch, it should work. Unfortunately, that hasn’t been rolled into a new release yet, so it’s not yet part of anything on PlatformIO’s library registry.

BTW, it’s good to know about https://github.com/adafruit/Adafruit-BMP3xx-PCB. Are you using the sensor?

[Anthony Aufdenkampe]

@neilh20, We just looked at the database and the server logs and found the following.
Only 5 sampling features stopped reporting new data on April 12.

• TUCA_PO03
• nh_LCC45
• TIMU01
• TUCA_Sa01
• DEADFALL_ELC-6

These stations have not attempted to post anything in the last 30 minutes.  We’ll keep looking.

Our guess is that the issue is originating from these stations, not from a bug on our servers.

We are indeed confused why they stopped reporting the same morning that we issued the release. That said, many of these stopped reporting a couple of hours BEFORE we switched the production servers, which according to our logging happened from 10:34 to 10:38 ET  ( = 7:34 to 7:38 PT).

[Anthony Aufdenkampe]

@ldecicco, last autumn I started working on a somewhat similar task using the same radio module, but for posting to Azure EventHub.

As @srgdamiano mentioned, the key is to add HTTPS capabilities using the TinyGsmClientSecure client(modem) . Doing that worked for me and I was able to Post data.

Although I haven’t worked on this for a few months, I need to get back to it and I just created this Pull Request from my feature branch to the develop branch, so that you can compare my code changes: https://github.com/EnviroDIY/ModularSensors/pull/432

[Anthony Aufdenkampe]

@jamesbailey, I just noticed this post and have a quick answer to your first problem regarding the EnviroDIY/TippingBucketRainCounter library.

In April 2021, I issued the v0.2.0: Add Anemometer Counting and Bug Fix release, which not only fixed a byte-order bug in the original release, but also expanded event counter from 2-bytes to 4-bytes.

Shortly afterwards I updated the EnviroDIY/ModularSensors library, in a feature branch, to work with the RainCounter release, and these updates were merged into the develop branch in October 2021 with the Update RainCounterIC2 to support reed-switch anemometer #384 pull request.

Unfortunately, that code is only now working its way into a new official release of ModularSensors, see https://github.com/EnviroDIY/ModularSensors/pull/407. When that release happens, you will no longer need to “hack” the libraries to get things to work.

Sorry for that delay in getting these two libraries synced up, and for the extra troubles that it caused you!

[Anthony Aufdenkampe]

@ayalavinay, Monitor My Watershed has two APIs:

• An HTTP POST API designed for devices to submit data to MonitorMW, as documented here: https://github.com/ODM2/ODM2DataSharingPortal/blob/main/doc/example_rest_requests.md
• A Water One Flow (WOF) / WaterML API for scripts or apps to get data out of the MonitorMW database, as documented here: https://monitormywatershed.org/wofpy/

Which one are you trying to use, and how? We need more information in order to help you.

[Anthony Aufdenkampe]

I’ll second Neil, and say that Keller sensor are very high quality and work really well with Mayfly RS485 integrations. I’ve helped deploy many AccuLevel sensors.

There’s a small chance that Keller 36XiW-CTD uses the same Modbus map as the Acculevel, but I think it is unlikely. You’ll need to get the Modbus documentation from Keller for that. I found that Keller’s tech support staff was super helpful.

The place to define the Modbus map for this sensor andstart testing is here: https://github.com/EnviroDIY/KellerModbus.   That’s the library that ModularSensors uses.

SDI-12 has the benefit that it uses universal commands, doesn’t require a special “map”, and can generally be implemented on the Mayfly without a special adapter board. The drawback of SDI-12 is that it has a much less stable communication protocol (both logical and physical), and often the sensors are more expensive to purchase that Modbus options.

 

[Anthony Aufdenkampe]

The first explanation that pops to mind is that the DIY Modbus wing has power bleed from the digital pins when the power is turned off.

Modbus stop bits are high, which leaves the AltSoftSerial transmit pin (5 or 6) at 3.3V when the sensor power shuts down. This then bleeds through RS485 converter (when it is powered off) over to the switched power rails, which then interferes with the startup of some other sensors, such as the MaxSonar. For a solution, see ModularSensors Issue “Add to Modbus an AltSoftSerial “flush” or “end” function, to set pins low #140″ for the coding solution.

The coding solution is now provided in the complex_loop option shown in the menu_a_la_carte.ino example. Specifically, at the end of every loop, the AltSoftSerial pins need to be set to low before the system is put to sleep, as shown here: https://github.com/EnviroDIY/ModularSensors/blob/292371055ab9d9b884f8fedb7c9587181cf7789d/examples/menu_a_la_carte/menu_a_la_carte.ino#L2928-L2932

This in turn required an altSoftSerial.begin(9600); statement at the top of every measure loop, as shown here:  https://github.com/EnviroDIY/ModularSensors/blob/292371055ab9d9b884f8fedb7c9587181cf7789d/examples/menu_a_la_carte/menu_a_la_carte.ino#L2850-L2854

This means that you must use the complex loop every time you use these Modbus Wings.

Let us know if that solves it.

[Anthony Aufdenkampe]

@neilh, we’re glad to have you doing the intensive testing, reporting what you find, and having patience with us over the years before we could dive in to this issues, and over these recent weeks and future months as we work out all the tech debt and also adjust to a growing data system (we have 400 million records!).

[Anthony Aufdenkampe]

@neilh, we’ve been tracking your many issues in the last few days, including #542, and have been working on solutions. I just responded in detail here:  https://github.com/ODM2/ODM2DataSharingPortal/issues/542#issuecomment-999785715

The short story is that we’re working hard to improve error handling (i.e. making it more accurate, rather than just passing a 201 immediately), but doing so has had some unintended consequences, especially for your specific code that resends all data that do not receive a 201.

The old system would queue up all the POST requests every 5 to 10 minutes, sometimes taking a minute to complete them all. However, ModularSensors times out after 7 seconds, so the radio and logger can go back to sleep. Sending a 201 immediately, before the POST completes, allowed that to happen.

The unintended consequence of our more accurate error handling is that if the POST doesn’t fully complete in <7 seconds, then you get a 504, even if it the data do get inserted into the database. With your specific “reliable delivery” code, you then send that data all over again at the next logging time along with one more data row. This has lead to a steady increase in our server load that was making the problem worse for everyone. This is why we recently switched back to sending 201 responses immediately.

Right now we are fast-tracking some hot fixes to the Gunicorn app server to allow for more concurrent posts, along with other fixes. For the next release, we’ll be upgrading Django 2.2 to 3.2, which enables us to use an ASGI (Asynchronous Server Gateway Interface) that will perform much better at routing the ever increasing traffic.

[Anthony Aufdenkampe]

@jlemontu-org,  we won’t be releasing that code until February with our planned v0.13 release, so I would encourage users to clear their cache before then. You can track that progress at: https://github.com/ODM2/ODM2DataSharingPortal/issues/529

That said, you don’t have to clear the cache for all websites, but just do it for MonitorMyWatershed. That’s not that painful. Clearing it for all websites is somewhat painful, so would be clear in how you make the recommendation.

[Author]

Posts