Debug flags for modem comms

What?? You haven’t read all the docs??  Actually, I don’t remember how well documented the build flags are.  Writing good documentation is hard.

Are you using transparent or bypass?  You can remove the build flags for whatever you’re not using.  You also don’t need the MQTT flag, it only applies to ThingSpeak (or anything else MQTT).  You also don’t need the NeoSWSerial flag unless you’re using that library.  Having those extra defines shouldn’t be a problem though.  Any build flag that starts with a ‘-D’ is the same as typing “#define” in your code.  When you hit “build” one of the first steps is for the pre-processor to essentially find-and-replace everything you’ve “defined” with the value you’ve defined it as.

Jake must have just updated TinyGSM.  I’d added a line to the debugging just a few days ago but screwed up and tried to print the value of a pointer.  I just fixed that and pushed.  You can both update again (“pio lib update”).

If you have an Arduino mega or other board with more than two serial ports available, I find debugging the modem can be a bit easier if you can “snoop” the modem Rx and Tx lines with a separate board.

@ensign and others (Rachel Johnson, Matt Gisondi, David Bressler, @shicks, @heather and a few more) have put a lot of work into that manual!

But we still have a 1-star review on Amazon for insufficient documentation. :*(

You are exactly 1 commit behind for TinyGSM.  Try again.  Have you maybe installed TinyGSM to your “global” library registry instead of your local one?  You might need to do a “pio lib -g update” (-g flag for global).  Otherwise you can navigate directly to into the folder and use git to update:  cd .pio\libdeps\mayfly\TinyGSM followed by git pull.

You might try using a new SIM card or a new LTEBee module.  We’ve had several of them stop working recently for no reason at all.  Sometimes using a brand new SIM card will fix the problem, but other times it requires using a new LTEbee module.  We haven’t determined the cause of the failure yet, nor a solution to get them working again.

What is the battery voltage that your Mayfly is recording in your data file?  We’ve had a few random failures of a mosfet on the board that will cause the Mayfly to read the incoming Lipo or USB voltage as being much lower than it really is (see this post: https://www.envirodiy.org/topic/innacurate-lipo-battery-voltage-data/).   Your logger code probably has a line in it that prohibits the bee from transmitting when the battery gets low, so it’s possible that your board has a bad mosfet which is tricking the Mayfly into not transmitting.

I’m sorry.  No, I don’t think it’s an issue with the battery or the SIM.

When you were running in Transparent, the XBee3 just didn’t register an internet connection.  They just do that sometimes – refuse to acknowledge they are registered on the network.  Sometimes switching from transparent to bypass and back to transparent “magically” fixes the connection issue on the transparent side.  Do you know what version of the Digi firmware you have?  The newest is a little better than older, but it’s still not perfect.

When you were running in Bypass, you were hitting the same issues that you hit with the UBee, except this time it connected, opened the channel, got data, and then somehow didn’t realize there was a connection and re-opened it several times.

I finally actually did some testing with my own board and managed to get most of the same errors you found in Bypass mode and pushed some corrections to TinyGSM.  It’s all about the issues with synchronous and asynchronous opening.  I figured out how to make it crash every time and (hopefully) how to make that not happen.  And the good news is that since both the XBee3 in Bypass and the Sodaq UBee are using the same code in TinyGSM, it should help both of them.

As for the issues with not connecting in transparent.. I’ve got nothing.  Try some prayers to Isidore of Seville, patron of the internet.  Sometimes switching the same board back and forth between transparent and bypass mode a couple of times works.  More often I’ve been able to “fix” a board that wouldn’t connect in transparent by using bypass, but today I had three different boards that I couldn’t get to connect in bypass and suddenly happily connected in transparent.  I have no idea why it happens.

Doh!  I’m sorry, I was confusing you with Dan who was troubleshooting issues with the UBee last week.  They’re even worse than the LTE XBee3’s.

The power isn’t your problem testing in the lab.  That seems a hair low for being connected to USB, but not concerning.  On the last bypass output that you posted from over the weekend, you did connect and even got the time from NIST, so your SIM and XBee and Mayfly are all OK.  It’s just making them play nice.

Yes, I think your next step is to update TinyGSM and re-test.  You can try switching between the modes to see if it helps the connection.  Don’t bother waiting more than 1 cycle and you can drop your interval down to as low as 1 minute while your watching.  If NIST connection doesn’t work in any of the 5 attempts, you can almost give up right there and post the output.

If the XBee3 refuses to submit, try switching to the “bleeding edge” version of ModularSensors on the ModemLast branch.  To use that run pio lib uninstall EnviroDIY_ModularSensors or pio lib -g uninstall EnviroDIY_ModularSensors (case matters).  Then after the uninstall, modify your platformio.ini and remove EnviroDIY_ModularSensors from the lib_deps section and add https://github.com/EnviroDIY/ModularSensors.git#modemLast in its place.  The modem last branch (as you might guess) restructures the way the internet connection happens.  Instead of starting the modem first and waiting for it to connect as if it’s a sensor detecting the signal strength, the library keeps the modem off until all the other sensors are complete.  Once the other sensor data is saved, the modem comes on, sends data to the world, and makes note of the signal strength just before going back to sleep.  On the next sensor interval, the prior modem strength will be recorded.  On the surface, the changes to your code should be pretty trivial, but the library will be doing different things underneath.

LOL!

@shicks came over to my office yesterday with a pile of LTE XBee3’s that will no longer connect to the internet.  We’re going to buy some alternative types of modems to see if any of them are any better.

I’m curious, were those LTE XBee3’s deployed in the field, worked properly then at some point stopped connecting to the internet? Or did you receive them and realize they wouldn’t connect upon initial testing? Just wanting to understand if this is something I need to watch out for with our LTE stations we’ve already deployed.

Thanks

@lemon – most of them were deployed in the field working and stopped for no apparent reason.  I didn’t realize we’d had so many failures until Shannon dumped them on my desk the other day.

@mbarney – we’ve gone back and forth over and over about the transparent-vs-bypass issue.  My experience has generally been that the modem connects more quickly and more consistently and is more frequently successful in sending data when in bypass mode.  Despite that, there is some bug that I have not been able to track down and only seems to happen with an XBee3 in bypass that causes the whole system to occasionally completely lock up until the watchdog finally forces a reset.  So we’ve been deploying with transparent.  The watchdog reset does work meaning you should only lose 15-20 minutes of data when the board freezes, so if you’re ok with that it should be fine to deploy in bypass mode.

As for the logs, I’m not sure about that one error.  It’s still working, so it’s probably not a big deal, but it should be fixed.  The permissions error in installing or compiling sometimes happens when somehow the computer thinks a file is stuck open.  Whenever you install a library or ‘build’ anything the PlatformIO and the compiler create and modify a whole bunch of files in the hidden .pio directory.  If it thinks one of those files is open by another process it can’t do its thing and give that error.  Usually completely closing all open copies of VSCode or Atom will ‘free’ whatever file got locked and you can re-open and it will be fine.  Sometimes you have to reboot.

I’m sorry about the issue with the “modemLast” branch.  Bleeding edge .. I changed something and was sloppy and didn’t make it backwards compatible.  I’ll push a fix.

Fix pushed.