[neilh20]

Hi Erik, Sounds like we ended up working on the same issue at about the same time.
Great to hear you have a basic path forwards. I am very much a fan of the open source ModularSensors and EnviroDIY which covers a lot of functionality.

However, it is a matrix of south bound (away from internet) instrument sensor interfaces ~~which are fantastic~~ and north bound (towards internet) communication interfaces.
I personally have just a few configurations that I’m trying to build for the future, very much with RS485/12V (and SDI12/5V as instruments support it).
From the software engineering aspect of the open source, there is no regression defined. So I’m attempting a definition of that.
SO interested how it works for you.

[neilh20]

Sara sorry about the stomach bug. Life is challenging when the rulers of the internal ecology get distracted with nasty bug invasions.

thanks for the fix ~ I just noticed this conversation today.

I think it was working with the AVR compiler as I had it all working for many months on a Mayfly (except that specific riparian monitoring Mayfly got stolen a couple of months ago). I’ve seen this problem before, that the AVR compiler has some defensive won’t execute a NULL pointer. Which is great. The NULL pointer came up in the ARM context, but I hadn’t regression tested it with the AVR.

I haven’t looked at this AVR Mayfly RS485 DIR management as its a streams based environment. That is where the Auto Direction SOM works very nicely. I had no problems with it.

Apologies if this is repeating something that everyone knows;
Modbus is a packet of information. At the begin of the packet the RS485 driver half duplex goes to transmit, and then only when the complete Modbus msg is sent does the RS485 driver revert to receive.
The reverting to receive is critical hard real time, as the modbus slave starts to respond shortly afterwards and I believe its specified at 3.5Character timings or about 3.5mS. Modbus also specifies for a packet that there should never be a gap between characters of more than 1.5Character timings or about 1.5mS.
The RS485 SOM does this with a simple hardware timing. Resistor, Cap, Diode and buffer(schmitt trigger) I believe.
For a streams based environment I believe its challenging.
I did investigate this for an Ubuntu Linux build and compiled in a different set of drivers for an RS485 mode.
For the streams based environment, turning the driver to transmit is easy and can be done before starting writing to the stream.
However it requires an accurate time to detect, not just when the last byte in the packet is written to the hardware (and how does a stream know its the last byte especially if there is a multibyte FIFO for the Tx UART) but when that byte has completed transmission through the UART. So for a stream based environment, the driver needs to start a timer whenever the UART end of character interrupt is received, and if another character isn’t written into the UART in 1.5character timings (a 1.5mS) then it needs to switch the direction to receive.
In the world of industrial designed systems for Modbus there are a number of potential solutions a) design a packet based interface for the Mega1280 UART- the UART supports end-of-byte transmission, but the driver guarenttees the Rx switch. b) use a MAX1348 driver which also has auto direction, but it requires +5V c) using a processor with UART that supports RS485 like the SAM51

Well Solstice Salutations to everyone, I hope you have a good holiday – and that there is harmony with the anthropogenic masters, the gastric microbial ecology!.

[neilh20]

Thanks for the front door 🙂 eTree was going down a whirly pooll!

This worked for me, though maybe there is a nicer way.
The output is value_num – which can be used for the threshold check.

import ulmo #post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:EnviroDIY_Mayfly_Batt' #post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:All_ExternalVoltage_Battery' wsdl_url = 'https://monitormywatershed.org/wofpy/soap/cuahsi_1_1/.wsdl' site_code = 'TU-RC-01' variable_code ='All_ExternalVoltage_Battery' site_values_dict = ulmo.cuahsi.wof.get_values(wsdl_url, site_code, variable_code, start=None, end=None, suds_cache=('default', )) for key_svd in site_values_dict : elem_svd = site_values_dict[key_svd] if key_svd == 'values' : print('**elem_svd',elem_svd) # Fut may have multiple 'value' first_or_default = next((x for x in elem_svd if 'value'), None) if first_or_default : print('**full_value',first_or_default) value_txt = first_or_default['value'] if value_txt : value_num = float(value_txt) print('value (v)', value_num) else : print('err2:value not found') else: print('err1:value not found')

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

import ulmo   #post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:EnviroDIY_Mayfly_Batt' #post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:All_ExternalVoltage_Battery'   wsdl_url = 'https://monitormywatershed.org/wofpy/soap/cuahsi_1_1/.wsdl' site_code = 'TU-RC-01' variable_code ='All_ExternalVoltage_Battery' site_values_dict = ulmo.cuahsi.wof.get_values(wsdl_url, site_code, variable_code, start=None, end=None, suds_cache=('default', ))   for key_svd in site_values_dict :     elem_svd = site_values_dict[key_svd]     if key_svd == 'values' :         print('**elem_svd',elem_svd)         # Fut may have multiple 'value'         first_or_default = next((x for x in elem_svd if 'value'), None)         if first_or_default :             print('**full_value',first_or_default)             value_txt = first_or_default['value']             if value_txt :                 value_num = float(value_txt)                 print('value (v)', value_num)             else :                 print('err2:value not found')         else:             print('err1:value not found')

The retrieved ‘sites_values_dict’ is

([('site', { 'code': 'TU-RC-01', 'name': 'TU-RC-01 LDSR PitTag Monitor', 'network': 'envirodiy', 'location': { 'latitude': '38.40731', 'longitude': '-122.81813', 'srs': 'EPSG:4236' } }), ('variable', { 'value_type': 'Instrument deployment', 'data_type': 'Average', 'general_category': 'Instrumentation', 'sample_medium': 'Surface water', 'no_data_value': '-9999.0000000000', 'code': 'All_ExternalVoltage_Battery', 'id': '215', 'name': 'Battery voltage', 'vocabulary': 'envirodiy', 'time': {}, 'units': { 'abbreviation': 'V', 'code': '369', 'name': 'Volt', 'type': 'Electromotive force' }, 'description': None }), ('values', [{ 'value': '16.0', 'method_code': '2', 'quality_control_level_code': '1', 'censor_code': 'nc', 'date_time_utc': '2019-11-23T03:51:02', 'source_code': '611', 'time_offset': '-08:00', 'datetime': '2019-11-22T19:51:02' }]), ('censor_codes', { 'nc': { 'censor_code': 'nc', 'description': 'nc' } }), ('methods', { '2': { 'id': '2', 'code': '2' } }), ('quality_control_levels', { '1': { 'id': '1', 'code': '1', 'definition': 'Raw Data' } }), ('sources', { '611': { 'id': '611', 'code': '611', 'organization': 'Trout Unlimited', 'description': 'Today TU is a national organization with about 300,000 members and supporters organized into over 400 chapters and councils from Maine to Montana to Alaska. This dedicated grassroots army is matched by a respected staff of lawyers, policy experts and scientists, who work out of more than 30 offices nationwide. These conservation professionals ensure that TU is at the forefront of fisheries restoration work at the local, state and national levels.', 'contact_name': 'Neil Hancock', 'email': 'neilh20+turc1910@wllw.net', 'link': 'https://www.tu.org/' } })])

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

([('site', {   'code': 'TU-RC-01',   'name': 'TU-RC-01 LDSR PitTag Monitor',   'network': 'envirodiy',   'location': {     'latitude': '38.40731',     'longitude': '-122.81813',     'srs': 'EPSG:4236'   } }), ('variable', {   'value_type': 'Instrument deployment',   'data_type': 'Average',   'general_category': 'Instrumentation',   'sample_medium': 'Surface water',   'no_data_value': '-9999.0000000000',   'code': 'All_ExternalVoltage_Battery',   'id': '215',   'name': 'Battery voltage',   'vocabulary': 'envirodiy',   'time': {},   'units': {     'abbreviation': 'V',     'code': '369',     'name': 'Volt',     'type': 'Electromotive force'   },   'description': None }), ('values', [{   'value': '16.0',   'method_code': '2',   'quality_control_level_code': '1',   'censor_code': 'nc',   'date_time_utc': '2019-11-23T03:51:02',   'source_code': '611',   'time_offset': '-08:00',   'datetime': '2019-11-22T19:51:02' }]), ('censor_codes', {   'nc': {     'censor_code': 'nc',     'description': 'nc'   } }), ('methods', {   '2': {     'id': '2',     'code': '2'   } }), ('quality_control_levels', {   '1': {     'id': '1',     'code': '1',     'definition': 'Raw Data'   } }), ('sources', {   '611': {     'id': '611',     'code': '611',     'organization': 'Trout Unlimited',     'description': 'Today TU is a national organization with about 300,000 members and supporters organized into over 400 chapters and councils from Maine to Montana to Alaska. This dedicated grassroots army is matched by a respected staff of lawyers, policy experts and scientists, who work out of more than 30 offices nationwide. These conservation professionals ensure that TU is at the forefront of fisheries restoration work at the local, state and national levels.',     'contact_name': 'Neil Hancock',     'email': 'neilh20+turc1910@wllw.net',     'link': 'https://www.tu.org/'   } })])

The debug out looks like this

** elem_svd[{ 'value': '16.0', 'method_code': '2', 'quality_control_level_code': '1', 'censor_code': 'nc', 'date_time_utc': '2019-11-23T03:51:02', 'source_code': '611', 'time_offset': '-08:00', 'datetime': '2019-11-22T19:51:02' }] ** full_value { 'value': '16.0', 'method_code': '2', 'quality_control_level_code': '1', 'censor_code': 'nc', 'date_time_utc': '2019-11-23T03:51:02', 'source_code': '611', 'time_offset': '-08:00', 'datetime': '2019-11-22T19:51:02' } value(v) 16.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

** elem_svd[{     'value': '16.0',     'method_code': '2',     'quality_control_level_code': '1',     'censor_code': 'nc',     'date_time_utc': '2019-11-23T03:51:02',     'source_code': '611',     'time_offset': '-08:00',     'datetime': '2019-11-22T19:51:02'   }]    ** full_value {     'value': '16.0',     'method_code': '2',     'quality_control_level_code': '1',     'censor_code': 'nc',     'date_time_utc': '2019-11-23T03:51:02',     'source_code': '611',     'time_offset': '-08:00',     'datetime': '2019-11-22T19:51:02'   }    value(v) 16.0

[neilh20]

Thanks for ulmo reference – I can sort of see something there, but its got a lot of layers and I can see it understands the response from a http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?
of the form
<timeSeriesResponse xmlns=”http://www.cuahsi.org/waterML/1.1/&#8221; xmlns:ns0=”http://www.cuahsi.org/his/1.1/ws/”&gt;

Well in the spirit of try many things, be prepared to fail, I’m putting aside trying to understand the namespaces built in here defined by
xmlns=”http://www.cuahsi.org/waterML/1.1/&#8221; xmlns:ns0=”http://www.cuahsi.org/his/1.1/ws/&#8221;

So just an FYI if there are an XML gurus, I’m trying to retrieve the simple value, and whenever I try to search down using xml.etree.ElementTree I can’t null out the effects of xmlns and xmlns:ns0.

A sensor value is encoded as “timeSeries” and shows

--->dump node <ns0:timeSeries xmlns:ns0="http://www.cuahsi.org/waterML/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <ns0:sourceInfo xsi:type="SiteInfoType"> <ns0:siteName>TU-RC-01 LDSR PitTag Monitor</ns0:siteName> <ns0:siteCode network="envirodiy" siteID="558">TU-RC-01</ns0:siteCode> <ns0:geoLocation> <ns0:geogLocation srs="EPSG:4236" xsi:type="LatLonPointType"> <ns0:latitude>38.40731</ns0:latitude> <ns0:longitude>-122.81813</ns0:longitude> </ns0:geogLocation> </ns0:geoLocation> </ns0:sourceInfo> <ns0:variable> <ns0:variableCode default="true" variableID="250" vocabulary="envirodiy">EnviroDIY_Mayfly_SampleNum</ns0:variableCode> <ns0:variableName>Sequence number</ns0:variableName> <ns0:variableDescription>The sample number returned by ModularSensors</ns0:variableDescription> <ns0:valueType>Instrument deployment</ns0:valueType> <ns0:dataType>Average</ns0:dataType> <ns0:generalCategory>Instrumentation</ns0:generalCategory> <ns0:sampleMedium>Surface water</ns0:sampleMedium> <ns0:unit unitID="409"> <ns0:unitName>Count</ns0:unitName> <ns0:unitType>Count</ns0:unitType> <ns0:unitAbbreviation>Count</ns0:unitAbbreviation> <ns0:unitCode>409</ns0:unitCode> </ns0:unit> <ns0:noDataValue>-9999.0000000000</ns0:noDataValue> <ns0:timeScale /> </ns0:variable> <ns0:values> <ns0:value censorCode="nc" dateTime="2019-11-21T15:44:02" dateTimeUTC="2019-11-21T23:44:02" methodCode="2" qualityControlLevelCode="1" timeOffset="-08:00">173.0</ns0:value> <ns0:qualityControlLevel qualityControlLevelID="1"> <ns0:qualityControlLevelCode>1</ns0:qualityControlLevelCode> <ns0:definition>Raw Data</ns0:definition> </ns0:qualityControlLevel> <ns0:method methodID="2"> <ns0:methodCode>2</ns0:methodCode> <ns0:methodLink /> </ns0:method> <ns0:censorCode> <ns0:censorCode>nc</ns0:censorCode> <ns0:censorCodeDescription>nc</ns0:censorCodeDescription> </ns0:censorCode> </ns0:values> </ns0:timeSeries>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

--->dump node <ns0:timeSeries xmlns:ns0="http://www.cuahsi.org/waterML/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">         <ns0:sourceInfo xsi:type="SiteInfoType">             <ns0:siteName>TU-RC-01 LDSR PitTag Monitor</ns0:siteName>             <ns0:siteCode network="envirodiy" siteID="558">TU-RC-01</ns0:siteCode>             <ns0:geoLocation>                 <ns0:geogLocation srs="EPSG:4236" xsi:type="LatLonPointType">                     <ns0:latitude>38.40731</ns0:latitude>                     <ns0:longitude>-122.81813</ns0:longitude>                 </ns0:geogLocation>             </ns0:geoLocation>         </ns0:sourceInfo>         <ns0:variable>             <ns0:variableCode default="true" variableID="250" vocabulary="envirodiy">EnviroDIY_Mayfly_SampleNum</ns0:variableCode>             <ns0:variableName>Sequence number</ns0:variableName>             <ns0:variableDescription>The sample number returned by ModularSensors</ns0:variableDescription>             <ns0:valueType>Instrument deployment</ns0:valueType>             <ns0:dataType>Average</ns0:dataType>             <ns0:generalCategory>Instrumentation</ns0:generalCategory>             <ns0:sampleMedium>Surface water</ns0:sampleMedium>             <ns0:unit unitID="409">                 <ns0:unitName>Count</ns0:unitName>                 <ns0:unitType>Count</ns0:unitType>                 <ns0:unitAbbreviation>Count</ns0:unitAbbreviation>                 <ns0:unitCode>409</ns0:unitCode>             </ns0:unit>             <ns0:noDataValue>-9999.0000000000</ns0:noDataValue>             <ns0:timeScale />         </ns0:variable>         <ns0:values>             <ns0:value censorCode="nc" dateTime="2019-11-21T15:44:02" dateTimeUTC="2019-11-21T23:44:02" methodCode="2" qualityControlLevelCode="1" timeOffset="-08:00">173.0</ns0:value>             <ns0:qualityControlLevel qualityControlLevelID="1">                 <ns0:qualityControlLevelCode>1</ns0:qualityControlLevelCode>                 <ns0:definition>Raw Data</ns0:definition>             </ns0:qualityControlLevel>             <ns0:method methodID="2">                 <ns0:methodCode>2</ns0:methodCode>                 <ns0:methodLink />             </ns0:method>             <ns0:censorCode>                 <ns0:censorCode>nc</ns0:censorCode>                 <ns0:censorCodeDescription>nc</ns0:censorCodeDescription>             </ns0:censorCode>         </ns0:values>     </ns0:timeSeries>

[neilh20]

On the ThingSpeak thanks for pointing it out – they have become painful. I have a number of legacy connections I’m setup up for.

Not knowing the architecture of the ODM2/WOF and WaterML 1.1 it was more of a reference.
Thingspeak used to be a lot more open, but Mathwork$ acquired them, and I guess they are making money out of it.
Which is to be expected, but I so appreciate enviroDIY coming along for environmental monitoring (and thanks to the NSF and William Penn Foundation funders)

I am digging into Python libs in parsing the WaterML – but if you have any examples on getting to the data that would be great.

[neilh20]

OK I think I’ve figured out something. I can only get the last variable at the moment (432) .. which is what I need for my real analysis of looking at the last value of the voltage returned and and checking it it has dropped below a threshold.

http://data.envirodiy.org/sites/TU-RC-01/ visually its 16.0V the first Battery Voltage “All_ExternalVoltage_Battery” and I can also see with a download of series=1906

So then using
http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:All_ExternalVoltage_Battery
comes back with something that includes

<values>
<value methodCode=”2″ qualityControlLevelCode=”1″ censorCode=”nc” dateTime=”2019-11-18T19:27:02″ dateTimeUTC=”2019-11-19T03:27:02″ sourceCode=”611″ timeOffset=”-08:00″>16.0</value>

which seems like I read as

<values>
<value …>16.0</value>

and its the 16.0 Volts that I’m after. So I need to parse to find that and compare it to my threshold.

[neilh20]

Gosh @srgdamiano wonderful!. ..

As every another question … (thankyou thankyou in advance)

So following the examples and trying for some GetValues on EnviroDIY_Mayfly_Temp from
https://data.envirodiy.org/sites/LCC45a3/
with this
http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:LCC45a3&variable=envirodiy:EnviroDIY_Mayfly_Temp&startDate=2019-09-01T00:00:00&endDate=2019-11-01T02:30:00

I get a response but no <values> that I would think I should get.
Any pointers – many thanks.

[neilh20]

To piggy back on this discussion – I’m wondering if anybody (@srgdamiano?) knows if there is a way of doing a POST/GET to request data from the ODM database for the latest row, or be able to access latest readings via WaterML.
The readings POST structure is described here
https://github.com/ODM2/ODM2DataSharingPortal/blob/master/doc/example_rest_requests.md

https://www.envirodiy.org/midstream-making-real-time-data-publicly-available/ from early architecture suggest access through WOFpy

What I’m looking for is to be able setup elsewhere a periodic request on the latest reading to be able to do a threshold check.

I know I can request data from ThingSpeak in this fashion, just wondering if there is a spec for it for data.enviroDIY.org
many thanks

[neilh20]

I had a similar problem – see my comment in https://www.envirodiy.org/topic/trouble-registering-with-lte-m-network-on-hologram/
The fix that seemed to work was if it didn’t connect after a 100 AI status polls it issues
gsmModem.sendAT(GF(“+CREG”));
Search for +CREG in
https://github.com/neilh10/ModularSensors/blob/rel1_dvlp1m/src/modems/DigiXBeeCellularTransparent.cpp

My Hologram with data connection every 4hrs looks like this – pink is no session , blue is with data

Attachments:

1. 

   191110HologramTimeLine.jpg
   

[neilh20]

Yup – the network is challenging when making it work, and adding to the biggest and most diverse machine in the world.
I’ve been more the old wireline, POTS to ISDN (now ADSL) – moving the analog to the CircuitSwitched Digital, and now its packet switched.

Though when I was working with one customer, when our product wasn’t connecting to the bigger machines, he referred to us being on the “Bleeding Edge” (the other side of cutting edge).
It actually turned out our product was working, but we interworking with a German device that wasn’t coping with US standards.
That’s why its nice to know about the 7Layer OSI model to have some common language about what the scope of the discussion is.
A lot of the 2G v 3G v 4G is industry fashion, bands of specifications, the reality is related to very specific specifications which evolve slower than 2G->3G->4G. The cellular industry is coping with high bandwidth expectations (5G), as well as low bandwidth IoT.
Whatever the technology, what does it take for my packets to be delivered successfully at what cost of power and devices.
I have heard of new silicon that can have problems with memory deep inside the chip – debugging that is hard.
For manuals I’ve been using mostly the
“Digi XBee3® Cellular LTE-M_NB-IoT 90002258revLsept2019.pdf”
https://www.digi.com/resources/documentation/digidocs/PDFs/90002258.pdf
though I’ve seen the questions and advice in various support forums about the Sara-R4.
Again thanks for the details on CREG – I’m mostly a niche usage for some environmental monitoring I’m supporting.
When my bit is stable I’ll put it in a wiki.

[Author]

Posts