Welcome to EnviroDIY, a community for do-it-yourself environmental science and monitoring. EnviroDIY is part of 
@srgdamiano
Active 1 months, 1 weeks agoForum Replies Created
-
AuthorPosts
-
Have you found the ulmo library: https://github.com/ulmo-dev/ulmo It can parse the WaterML.
I believe it’s what Model My Watersehd and other tools we have are using. I haven’t actually used it myself, so I can’t help with it more than giving you the link.
There are some bugs with WOFpy right now; I’d thought some had been fixed, but it looks like the only-returning-one-point issue is known and un-fixed: https://github.com/ODM2/ODM2DataSharingPortal/issues/432
I very rarely look at the WOFpy endpoint, so I’m not always up-to-date on it’s working status. Sorry!
Just a note on sending to ThingSpeak: there’s no free tier for anything but “personal use.” If you set your MathWorks/MatLab/ThingSpeak account up as anything else (even non-profit/educational) you only get 30 days free and have to pay beyond that, no matter how low your usage is.
If a new csv is being created, it’s almost certainly because the logger is restarting. It could be some irregularity in the power that’s causing the board to restart directly or it’s the watchdog. If you’re not specifically setting the file name, whenever the board attempts to talk to the SD card, it puts together the logger ID and the date and checks if a file already exists with that name. If that file exists, it adds to it; if not, it creates it. So multiple restarts in the same day wouldn’t show up as multiple files, but you would see a new file for each new day there’s a restart. If the gap is 15 or 20 minutes, chances are really high the watchdog is firing.
It’s also possible that there’s something up with the connections between the SD card and the Mayfly, but since you’re seeing the issue on multiple loggers I doubt that’s it.
The WoFpy endpoint is here: http://data.wikiwatershed.org/wofpy/
Right now there’s no ability to do units conversions in either ModularSensors or on Monitor My Watershed. Your only option in ModularSensors would be to create a calculated variable that did the units conversion.
On Monitor My Watershed, I think almost all of the temperature variables have the option of being in either C or F, but nearly everything else will only give you the option right now of the “default” unit any given sensor spits out. Those lists are curated, but it’s not difficult for us to add any new output variables/units you need. Let me know what you’d like to have added and I can do it. If the only “calculation” you’re doing is a unit conversion, then the output should still be tied to a specific sensor make/model. Once I add the extra unit, you’d be able to select that unit from the drop down in the same way you see the Celsius/Fahrenheit option now.
The “all/calculated” was really put in place for values that are calculated using the outputs of more than one sensor – like that barometric correction that uses atmospheric pressure from one sensor, water pressure from another, and temperature from a third. It feels really “hacky” because the Monitor My Watershed portal is primarily intended for “level 0” (totally raw) data that hasn’t had any calculations or QC applied. But, if you want to have a new calculated option, just let me know what you need to have added and, within reason, I can put it in. Just be aware that inside the database, no matter how polished that calculation is, the data on MonitorMW is still tagged as “raw”.
To turn on the “deep” debugging for the modem, add build flags “-DMS_DIGIXBEECELLULARTRANSPARENT_DEBUG” and “-DMS_DIGIXBEECELLULARTRANSPARENT_DEBUG_DEEP” for cellular or “-DMS_DIGIXBEEWIFI_DEBUG” and “-DMS_DIGIXBEEWIFI_DEBUG_DEEP”. Keep the publisher build flags set. You’d also have to change your serial monitor line endings to just a carriage return instead of CR/LF (quirk of the XBee). In PlatformIO/Atom you can do that by clicking for more options in the pop-up to open the serial monitor. In VSCode, you need to add this to your platformio.ini in the env section:
INI123monitor_flags =--eolCRIf you want to save the output, I think you’ll need to use a different terminal monitor. I often use TeraTerm, but there are a lot of free terminals out there.
The only thing coming to mind right now is possible issues with the json or the content length header. If the length of the json doesn’t match up with the actual size of the json, the send would probably be rejected. (You’d get a 5xx I think.) Unfortunately the only way you’d see that 5xx or details on why the send failed would be to turn on the modem’s “deep” debugging, log the full output to a text file, and then scroll through the thousands and thousands of lines of output looking for a time when a send failed to see what the response was. Or you could get *really* lucky and catch it in the act. I’ve been trying to catch the LTE XBee3’s in bypass mode in the act of crashing to see if I can figure out why it sometimes locks up but I haven’t managed.
You could also download the csv from MonitorMW and go line-by-line against a logger csv until you find a gap. See if the data in the gap is in any way different from the surrounding lines. Does one of the values have more or fewer digits when there is a gap? (As in, something like 4.65, 4.66, 4.62, *4.6*, 4.61 where the 4.6 is suddenly “shorter” because of the missing trailing 0.) I’m kind-of grasping at straws though.
I’d completely forgotten about the filename issue, too. It’s definitely a “gotcha” that’s hard to catch.
Hm. Well, I’m stumped. I have issues all the time with the cellular, but not the Wifi.
I’m sorry; I’m going to throw a zillion questions at you again.
You’re using 4 separate Mayfly’s, right? (ie, you don’t have both the wifi and cellular XBee’s hooked up to the same Mayfly). You’re wifi is connected to “real” wifi, right? (ie, not a cellular hot-spot). Are the loggers all near each other? Does the timing of the holes in the loggers match up or are they different for each? Have you ever manually changed any settings on the XBee’s (ie, with XCTU)?
What sensors do you have attached? I don’t think you said, but you’re using ModularSensors, right? Have you set/modified MS_SEND_BUFFER_SIZE? Can you try running with the build flags “-DMS_DATAPUBLISHERBASE_DEBUG” and “-DMS_ENVIRODIYPUBLISHER_DEBUG” to look at the outgoing json? Is your json longer than the “default” buffer of 750 (ie, is it broken up into more than one chunk)? Do you see anything strange going on like variable numbers of significant figures that sometimes get truncated leading to ill-formed json? What response codes are you getting back (ie, 201/403/5xx)?
Are you using LTE carrier boards for the LTE XBee’s? Do you have “useCTSforStatus” set appropriately (ie, false with a carrier board, true without)? If you’re really lucky and manage to catch the board “in the act” of a failed send, do the LED’s on the carrier board turn on as expected (ie, “on” is solid, “status” is solid and then begins to blink)?
-
AuthorPosts
