Coding to send data from a non-Mayfly logger

Tagged: connectivity, rssi

This topic has 5 replies, 3 voices, and was last updated 2023-12-11 at 2:08 PM by neilh20.

Viewing 5 reply threads

Author

Posts
- 2023-12-01 at 1:37 PM #18205
  NormO
  Participant
  <p style=”font-weight: 400;”>Hi there,</p>
  <p style=”font-weight: 400;”>I’m trying to figure out if it’s possible to route our data to your website. We have a couple water quality monitoring buoys that currently send data to ThingSpeak (using HTTP POST I believe) and I’m wondering if your Monitoring My Watershed data sharing portal is an option for us. But, I can’t really see from the variety of resources on the website whether and exactly how you we would route our data there. We don’t have the Mayfly logger nor do we use the codebase developed for it. We have our own home-grown DIY logger and codebase that routes data via the BluesWireless (cellular) network. I’ve looked through a bunch of your help topics and done some searches of the forum, but I’ve not found anything that tells me if and how I can route our data once I create an account and a site on the Monitoring My Watershed. Can you tell me whether it’s possible (or not) and point me to the doc that would tell me how to do it? Thanks!!!
  cheers,
  …Norm…</p>
- 2023-12-01 at 2:32 PM #18206
  Sara Damiano
  Moderator
  If you can create a POST request, you can get data to Monitor My Watershed. There’s information about the POST format here:
  
  https://github.com/ODM2/ODM2DataSharingPortal/blob/main/doc/example_rest_requests.md
- 2023-12-01 at 5:46 PM #18207
  NormO
  Participant
  Thank you!
  
  I will see if I can setup a route and test it out.
- 2023-12-06 at 1:59 PM #18212
  neilh20
  Participant
  @oakleynode something to note is that the MMW/ODM2 packet design with UUIDs make an extra heavy packet. The packet design is not optimally designed for wireless conditions, the thingspeak packets are much lighter. If your site is on the edge of the wireless range, which varies a lot of with weather conditions, its going to have more trouble with the MMW UUID overheads. A site that I have on the very edge of wireless connectivity is https://monitormywatershed.org/sites/TUCA_GV01/. I implement a sequence number in all my packets to be able to easily characterize reliability. I have been upgrading the antenna on the site to be able to improve the wireless signal strength.
  
  I’ve found the server is pretty exacting, and the error responses are wip. In addition due to some undocumented compression routines on the server, if the POST timestamp is too old, and that region of the database its going to insert into is compressed, it then initiates an uncompres for that region and that can take an undefined looong, time . Which can the client POSTer to define a times-out.
  
  I’m an advocate of reliability for ModularSensors, – and that also has meant characterizing the server, and in the process of testing my devices I’m finding server issues, one example https://github.com/ODM2/ODM2DataSharingPortal/issues/685
  
  My reliability enhancements can do multiples POSTs per connection . Over a celluar connection (with 9600bard to the modem) the first POST has had responses of ~ 2.5sec, and subsequent ones 10seconds. However the server is not set up to take too many serial POSTs. Discussion https://github.com/ODM2/ODM2DataSharingPortal/issues/673#issuecomment-1692514992
  
  For the ODM example, seems to me it is missing the “Content-Length: ”
  
  The best documentation is the code.
  
  https://github.com/EnviroDIY/ModularSensors/blob/master/src/publishers/EnviroDIYPublisher.cpp
  
  I got the POST working on a Wio Terminal device (WiFi), but I needed to get wire-shark out to be able to sniff the packet to see exactly what the WioT device driver was outputting.
  
  Here is an example of a trace from the transmitting device that succeeded (with a couple of changes to UUIDs so this won’t work)
  
  POST /api/data-stream/ HTTP/1.1
  Host: data.envirodiy.org
  TOKEN: d9e65186-6cf6-4803-xxxx-4d9b1142bd48
  Content-Length: 508
  Content-Type: application/json
  
  {“sampling_feature”:”851d0f00-3d8b-41c7-xxxx-40cb86b9ad2c”,”timestamp”:”2023-10-27T16:04:49-08:00″,”740cc4f8-6bc5-471a-811c-e0977f286f51″:3,”1f239d16-e6e4-4f76-ae98-0b274c75c79e”:48.04,”5ab3af01-bf49-4187-893d-276d7d256cd7″:17.93,”fd4e9448-8f67-4b5b-bd73-630b0659af8b”:17.1875,”586f9006-261b-4c5a-8abe-c93d5ec799fe”:17.8750,”87e166b4-0810-424f-bcd1-bbfaf9bda24d”:17.5625,”f10be729-b2d0-48ba-9c30-912274985b83″:17.7500,”27d7e570-7ef1-4ab5-af94-dfef2eeb19c1″:4.101,”66b0642a-d8c9-4e25-9a8f-39e081c188fc”:6.557}
  
  and in this case over WiFi the response is faster
  
  — Response Code — 201 waited 632 mS Timeout 10000
  
  regards
- 2023-12-10 at 12:58 PM #18223
  NormO
  Participant
  @neilh20 – thanks for your very detailed post and information!!
  
  I hadn’t really considered the relationship between connectivity and packet size. I do have some issues at some of my sites with connectivity but always thought about it in terms of local interference. About half of my sites are buoys and someone has suggested to me that the ground plane of the water can interfere with the cell signal reception, but I’ve not had time to test that out. I’m also saving the data to an onboard SD card so, theoretically, nothing is being lost when I drop the cell signal.
  
  I did look at the site you described as being the worst for connectivity TUCA_GV01, downloaded and graphed the RSSI data just to get an idea about what the worst might look like and notice that it seems to drop out beyond about -80 and seems to have improved over time, although I’d really have to put a little more work into the graph to prove it.
  
  I’m not capturing regular RSSI data from my sites but it looks like that would be a good piece of info to trend, so I’ll have to look into modifying my code to get it. Assuming RSSI is a significant parameter, are there particular ranges for what you consider excellent, good, and poor?
  
  cheers!
  Attachments:
  TUCA_GV01_RSSI_2023.png
- 2023-12-11 at 2:08 PM #18227
  neilh20
  Participant
  @oakleynode wow interesting you are looking at RSSI and it has a lot of variation. I haven’t characterized the RSSI result for a high percentage of failing calls.
  
  Another system that is also on the edge, and I added a new antenna that improved it, the reading is -81dBm https://monitormywatershed.org/sites/TUCA_Mi06/
  
  The modem is a Digi CAT-M1 XB3-C-A2 with Verizon service. I haven’t got a real figure of ranges that should work
  
  A test system is giving a crazy -1dBm – https://monitormywatershed.org/tsv/tu_rc_test07/4904/ – maybe its too strong to characterize!
  
  The retrieved RSSI is the measurement of the last call, not the current one. Not sure what it indicates if the call fails completely – doesn’t go through. for GV01 its attempting to make calls every 15minutes, and then subsequently 15minutes later would be retrieving the RSSI.
  
  For Mi06 it only makes a call every hour, though it retrieves the RSS reading on every 15minutes, to its likely the RSSI is the same value between calls.
  
  For cell reflection off the ocean – you could look it up in a text book – but with waves maybe too chaotic. Not sure what you do with any answer, as it works or it doesn’t. It might be an interesting answer if you are wanting to compare against different frequencies – say would LoRa 900MHz be more reliable, but requires data packing. I haven’t figured out to easily do LoRa with MMW.
  
  I keep an internal log on the uSD of connection attempts, and time to make the connection. Mostly that’s been useful for validating that connections where happening, that is verifying the device software. It also allowed me to show that the server was loosing data and in early 2023 the server response time was going out .
Author

Posts