Parameter Threshold Notifications

Tagged: ulmo

This topic has 13 replies, 4 voices, and was last updated 2019-11-22 at 6:16 PM by neilh20.

Viewing 13 reply threads

Author

Posts

2019-05-21 at 2:06 PM #12908
Jake Lemon
Participant
Has anyone set up notifications when Mayfly logger stations data exceeds a prescribed threshold? For example, if conductivity were to exceed 500 us/cm an email would be sent to a designated email addr

Has anyone set up notifications when Mayfly logger stations data exceeds a prescribed threshold? For example, if conductivity were to exceed 500 us/cm an email would be sent to a designated email address.

More...

Less...
2019-05-25 at 3:32 PM #12910
Adam Gold
Participant
You can do this remotely if you send data to ThingSpeak (already supported by ModularSensors). There are caps on how many ThingSpeak channels you can have and still have it be free (4 per account, I

You can do this remotely if you send data to ThingSpeak (already supported by ModularSensors). There are caps on how many ThingSpeak channels you can have and still have it be free (4 per account, I think), but you can make a ThingSpeak channel email or tweet at you when it meets certain conditions. Here’s a similar question about alarms from the Mayfly: https://www.envirodiy.org/topic/programming-mayfly-to-send-alerts/

Emailing from ThingSpeak involves using an IFTTT applet and some built-in ThingSpeak apps: https://www.mathworks.com/help/thingspeak/analyze-channel-data-to-send-email-notification-from-ifttt.html

Tweeting functionality is built in to ThingSpeak: https://thingspeak.com/apps

More...

Less...
2019-05-28 at 9:34 AM #12911
Jake Lemon
Participant
Thanks Adam!
2019-11-18 at 12:47 PM #13358
neilh20
Participant
To piggy back on this discussion – I’m wondering if anybody (@srgdamiano?) knows if there is a way of doing a POST/GET to request data from the ODM database for the latest row, or be able

To piggy back on this discussion – I’m wondering if anybody (@srgdamiano?) knows if there is a way of doing a POST/GET to request data from the ODM database for the latest row, or be able to access latest readings via WaterML.
The readings POST structure is described here
https://github.com/ODM2/ODM2DataSharingPortal/blob/master/doc/example_rest_requests.md

https://www.envirodiy.org/midstream-making-real-time-data-publicly-available/ from early architecture suggest access through WOFpy

What I’m looking for is to be able setup elsewhere a periodic request on the latest reading to be able to do a threshold check.

I know I can request data from ThingSpeak in this fashion, just wondering if there is a spec for it for data.enviroDIY.org
many thanks

More...

Less...
2019-11-18 at 1:38 PM #13361
Sara Damiano
Moderator
The WoFpy endpoint is here: http://data.wikiwatershed.org/wofpy/
2019-11-18 at 1:51 PM #13363
Sara Damiano
Moderator
Just a note on sending to ThingSpeak: there’s no free tier for anything but “personal use.” If you set your MathWorks/MatLab/ThingSpeak account up as anything else (even non-profit

Just a note on sending to ThingSpeak: there’s no free tier for anything but “personal use.” If you set your MathWorks/MatLab/ThingSpeak account up as anything else (even non-profit/educational) you only get 30 days free and have to pay beyond that, no matter how low your usage is.

More...

Less...
2019-11-18 at 3:53 PM #13365
neilh20
Participant
Gosh @srgdamiano wonderful!. ..

As every another question … (thankyou thankyou in advance)

So following the examples and trying for some GetValues on EnviroDIY_Mayfly_Temp from
https://data

Gosh @srgdamiano wonderful!. ..

As every another question … (thankyou thankyou in advance)

So following the examples and trying for some GetValues on EnviroDIY_Mayfly_Temp from
https://data.envirodiy.org/sites/LCC45a3/
with this
http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:LCC45a3&variable=envirodiy:EnviroDIY_Mayfly_Temp&startDate=2019-09-01T00:00:00&endDate=2019-11-01T02:30:00

I get a response but no <values> that I would think I should get.
Any pointers – many thanks.

More...

Less...
2019-11-18 at 4:00 PM #13366
Sara Damiano
Moderator
There are some bugs with WOFpy right now; I’d thought some had been fixed, but it looks like the only-returning-one-point issue is known and un-fixed: https://github.com/ODM2/ODM2DataSharingPo

There are some bugs with WOFpy right now; I’d thought some had been fixed, but it looks like the only-returning-one-point issue is known and un-fixed: https://github.com/ODM2/ODM2DataSharingPortal/issues/432

I very rarely look at the WOFpy endpoint, so I’m not always up-to-date on it’s working status. Sorry!

More...

Less...
2019-11-18 at 4:22 PM #13367
neilh20
Participant
OK I think I’ve figured out something. I can only get the last variable at the moment (432) .. which is what I need for my real analysis of looking at the last value of the voltage returned and

OK I think I’ve figured out something. I can only get the last variable at the moment (432) .. which is what I need for my real analysis of looking at the last value of the voltage returned and and checking it it has dropped below a threshold.

http://data.envirodiy.org/sites/TU-RC-01/ visually its 16.0V the first Battery Voltage “All_ExternalVoltage_Battery” and I can also see with a download of series=1906

So then using
http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:All_ExternalVoltage_Battery
comes back with something that includes

<values>
<value methodCode=”2″ qualityControlLevelCode=”1″ censorCode=”nc” dateTime=”2019-11-18T19:27:02″ dateTimeUTC=”2019-11-19T03:27:02″ sourceCode=”611″ timeOffset=”-08:00″>16.0</value>

which seems like I read as

<values>
<value …>16.0</value>

and its the 16.0 Volts that I’m after. So I need to parse to find that and compare it to my threshold.

More...

Less...
2019-11-19 at 3:40 PM #13376
neilh20
Participant
On the ThingSpeak thanks for pointing it out – they have become painful. I have a number of legacy connections I’m setup up for.

Not knowing the architecture of the ODM2/WOF and WaterML

On the ThingSpeak thanks for pointing it out – they have become painful. I have a number of legacy connections I’m setup up for.

Not knowing the architecture of the ODM2/WOF and WaterML 1.1 it was more of a reference.
Thingspeak used to be a lot more open, but Mathwork$ acquired them, and I guess they are making money out of it.
Which is to be expected, but I so appreciate enviroDIY coming along for environmental monitoring (and thanks to the NSF and William Penn Foundation funders)

I am digging into Python libs in parsing the WaterML – but if you have any examples on getting to the data that would be great.

More...

Less...
2019-11-20 at 1:34 PM #13382
Sara Damiano
Moderator
Have you found the ulmo library: https://github.com/ulmo-dev/ulmo It can parse the WaterML.

I believe it’s what Model My Watersehd and other tools we have are using. I haven’t actually

Have you found the ulmo library: https://github.com/ulmo-dev/ulmo It can parse the WaterML.

I believe it’s what Model My Watersehd and other tools we have are using. I haven’t actually used it myself, so I can’t help with it more than giving you the link.

More...

Less...

2019-11-21 at 2:51 PM #13386

neilh20

Participant

Thanks for ulmo reference – I can sort of see something there, but its got a lot of layers and I can see it understands the response from a http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues

Thanks for ulmo reference – I can sort of see something there, but its got a lot of layers and I can see it understands the response from a http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?
of the form
<timeSeriesResponse xmlns=”http://www.cuahsi.org/waterML/1.1/” xmlns:ns0=”http://www.cuahsi.org/his/1.1/ws/”>

Well in the spirit of try many things, be prepared to fail, I’m putting aside trying to understand the namespaces built in here defined by
xmlns=”http://www.cuahsi.org/waterML/1.1/” xmlns:ns0=”http://www.cuahsi.org/his/1.1/ws/”

So just an FYI if there are an XML gurus, I’m trying to retrieve the simple value, and whenever I try to search down using xml.etree.ElementTree I can’t null out the effects of xmlns and xmlns:ns0.

A sensor value is encoded as “timeSeries” and shows

--->dump node
<ns0:timeSeries xmlns:ns0="http://www.cuahsi.org/waterML/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <ns0:sourceInfo xsi:type="SiteInfoType">
            <ns0:siteName>TU-RC-01 LDSR PitTag Monitor</ns0:siteName>
            <ns0:siteCode network="envirodiy" siteID="558">TU-RC-01</ns0:siteCode>
            <ns0:geoLocation>
                <ns0:geogLocation srs="EPSG:4236" xsi:type="LatLonPointType">
                    <ns0:latitude>38.40731</ns0:latitude>
                    <ns0:longitude>-122.81813</ns0:longitude>
                </ns0:geogLocation>
            </ns0:geoLocation>
        </ns0:sourceInfo>
        <ns0:variable>
            <ns0:variableCode default="true" variableID="250" vocabulary="envirodiy">EnviroDIY_Mayfly_SampleNum</ns0:variableCode>
            <ns0:variableName>Sequence number</ns0:variableName>
            <ns0:variableDescription>The sample number returned by ModularSensors</ns0:variableDescription>
            <ns0:valueType>Instrument deployment</ns0:valueType>
            <ns0:dataType>Average</ns0:dataType>
            <ns0:generalCategory>Instrumentation</ns0:generalCategory>
            <ns0:sampleMedium>Surface water</ns0:sampleMedium>
            <ns0:unit unitID="409">
                <ns0:unitName>Count</ns0:unitName>
                <ns0:unitType>Count</ns0:unitType>
                <ns0:unitAbbreviation>Count</ns0:unitAbbreviation>
                <ns0:unitCode>409</ns0:unitCode>
            </ns0:unit>
            <ns0:noDataValue>-9999.0000000000</ns0:noDataValue>
            <ns0:timeScale />
        </ns0:variable>
        <ns0:values>
            <ns0:value censorCode="nc" dateTime="2019-11-21T15:44:02" dateTimeUTC="2019-11-21T23:44:02" methodCode="2" qualityControlLevelCode="1" timeOffset="-08:00">173.0</ns0:value>
            <ns0:qualityControlLevel qualityControlLevelID="1">
                <ns0:qualityControlLevelCode>1</ns0:qualityControlLevelCode>
                <ns0:definition>Raw Data</ns0:definition>
            </ns0:qualityControlLevel>
            <ns0:method methodID="2">
                <ns0:methodCode>2</ns0:methodCode>
                <ns0:methodLink />
            </ns0:method>
            <ns0:censorCode>
                <ns0:censorCode>nc</ns0:censorCode>
                <ns0:censorCodeDescription>nc</ns0:censorCodeDescription>
            </ns0:censorCode>
        </ns0:values>
    </ns0:timeSeries>

--->dump node

<ns0:timeSeries xmlns:ns0="http://www.cuahsi.org/waterML/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<ns0:sourceInfo xsi:type="SiteInfoType">

<ns0:siteName>TU-RC-01 LDSR PitTag Monitor</ns0:siteName>

<ns0:siteCode network="envirodiy" siteID="558">TU-RC-01</ns0:siteCode>

<ns0:geoLocation>

<ns0:geogLocation srs="EPSG:4236" xsi:type="LatLonPointType">

<ns0:latitude>38.40731</ns0:latitude>

<ns0:longitude>-122.81813</ns0:longitude>

</ns0:geogLocation>

</ns0:geoLocation>

</ns0:sourceInfo>

<ns0:variable>

<ns0:variableCode default="true" variableID="250" vocabulary="envirodiy">EnviroDIY_Mayfly_SampleNum</ns0:variableCode>

<ns0:variableName>Sequence number</ns0:variableName>

<ns0:variableDescription>The sample number returned by ModularSensors</ns0:variableDescription>

<ns0:valueType>Instrument deployment</ns0:valueType>

<ns0:dataType>Average</ns0:dataType>

<ns0:generalCategory>Instrumentation</ns0:generalCategory>

<ns0:sampleMedium>Surface water</ns0:sampleMedium>

<ns0:unit unitID="409">

<ns0:unitName>Count</ns0:unitName>

<ns0:unitType>Count</ns0:unitType>

<ns0:unitAbbreviation>Count</ns0:unitAbbreviation>

<ns0:unitCode>409</ns0:unitCode>

</ns0:unit>

<ns0:noDataValue>-9999.0000000000</ns0:noDataValue>

<ns0:timeScale />

</ns0:variable>

<ns0:values>

<ns0:value censorCode="nc" dateTime="2019-11-21T15:44:02" dateTimeUTC="2019-11-21T23:44:02" methodCode="2" qualityControlLevelCode="1" timeOffset="-08:00">173.0</ns0:value>

<ns0:qualityControlLevel qualityControlLevelID="1">

<ns0:qualityControlLevelCode>1</ns0:qualityControlLevelCode>

<ns0:definition>Raw Data</ns0:definition>

</ns0:qualityControlLevel>

<ns0:method methodID="2">

<ns0:methodCode>2</ns0:methodCode>

<ns0:methodLink />

</ns0:method>

<ns0:censorCode>

<ns0:censorCode>nc</ns0:censorCode>

<ns0:censorCodeDescription>nc</ns0:censorCodeDescription>

</ns0:censorCode>

</ns0:values>

</ns0:timeSeries>

More...

2019-11-21 at 3:04 PM #13387
Sara Damiano
Moderator
I don’t think you should have to be messing with eTree if you’re using ulmo. There’s a get_values function that returns a simple python dictionary: https://ulmo.readthedocs.io/en/l

I don’t think you should have to be messing with eTree if you’re using ulmo. There’s a get_values function that returns a simple python dictionary: https://ulmo.readthedocs.io/en/latest/api.html#ulmo.cuahsi.wof.get_values The WSDL URL is https://monitormywatershed.org/wofpy/soap/cuahsi_1_1/.wsdl.

I hate having to use the eTree library. I find trying to navigate through layers of XML to be approximately 0% fun.

More...

Less...

2019-11-22 at 6:16 PM #13390

neilh20

Participant

Thanks for the front door 🙂 eTree was going down a whirly pooll!

This worked for me, though maybe there is a nicer way.
The output is value_num – which can be used for the threshold check.

Thanks for the front door 🙂 eTree was going down a whirly pooll!

This worked for me, though maybe there is a nicer way.
The output is value_num – which can be used for the threshold check.

import ulmo

#post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:EnviroDIY_Mayfly_Batt'
#post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:All_ExternalVoltage_Battery'

wsdl_url = 'https://monitormywatershed.org/wofpy/soap/cuahsi_1_1/.wsdl'
site_code = 'TU-RC-01'
variable_code ='All_ExternalVoltage_Battery'
site_values_dict = ulmo.cuahsi.wof.get_values(wsdl_url, site_code, variable_code, start=None, end=None, suds_cache=('default', ))

for key_svd in site_values_dict :
    elem_svd = site_values_dict[key_svd]
    if key_svd == 'values' :
        print('**elem_svd',elem_svd)
        # Fut may have multiple 'value'
        first_or_default = next((x for x in elem_svd if 'value'), None)
        if first_or_default :
            print('**full_value',first_or_default)
            value_txt = first_or_default['value']
            if value_txt :
                value_num = float(value_txt)
                print('value (v)', value_num)
            else :
                print('err2:value not found')
        else:
            print('err1:value not found')

import ulmo

#post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:EnviroDIY_Mayfly_Batt'

#post_req='http://data.wikiwatershed.org/wofpy/rest/1_1/GetValues?location=envirodiy:TU-RC-01&variable=envirodiy:All_ExternalVoltage_Battery'

wsdl_url = 'https://monitormywatershed.org/wofpy/soap/cuahsi_1_1/.wsdl'

site_code = 'TU-RC-01'

variable_code ='All_ExternalVoltage_Battery'

site_values_dict = ulmo.cuahsi.wof.get_values(wsdl_url, site_code, variable_code, start=None, end=None, suds_cache=('default', ))

for key_svd in site_values_dict :

elem_svd = site_values_dict[key_svd]

if key_svd == 'values' :

print('**elem_svd',elem_svd)

# Fut may have multiple 'value'

first_or_default = next((x for x in elem_svd if 'value'), None)

if first_or_default :

print('**full_value',first_or_default)

value_txt = first_or_default['value']

if value_txt :

value_num = float(value_txt)

print('value (v)', value_num)

else :

print('err2:value not found')

else:

print('err1:value not found')

The retrieved ‘sites_values_dict’ is

([('site', {
  'code': 'TU-RC-01',
  'name': 'TU-RC-01 LDSR PitTag Monitor',
  'network': 'envirodiy',
  'location': {
    'latitude': '38.40731',
    'longitude': '-122.81813',
    'srs': 'EPSG:4236'
  }
}), ('variable', {
  'value_type': 'Instrument deployment',
  'data_type': 'Average',
  'general_category': 'Instrumentation',
  'sample_medium': 'Surface water',
  'no_data_value': '-9999.0000000000',
  'code': 'All_ExternalVoltage_Battery',
  'id': '215',
  'name': 'Battery voltage',
  'vocabulary': 'envirodiy',
  'time': {},
  'units': {
    'abbreviation': 'V',
    'code': '369',
    'name': 'Volt',
    'type': 'Electromotive force'
  },
  'description': None
}), ('values', [{
  'value': '16.0',
  'method_code': '2',
  'quality_control_level_code': '1',
  'censor_code': 'nc',
  'date_time_utc': '2019-11-23T03:51:02',
  'source_code': '611',
  'time_offset': '-08:00',
  'datetime': '2019-11-22T19:51:02'
}]), ('censor_codes', {
  'nc': {
    'censor_code': 'nc',
    'description': 'nc'
  }
}), ('methods', {
  '2': {
    'id': '2',
    'code': '2'
  }
}), ('quality_control_levels', {
  '1': {
    'id': '1',
    'code': '1',
    'definition': 'Raw Data'
  }
}), ('sources', {
  '611': {
    'id': '611',
    'code': '611',
    'organization': 'Trout Unlimited',
    'description': 'Today TU is a national organization with about 300,000 members and supporters organized into over 400 chapters and councils from Maine to Montana to Alaska. This dedicated grassroots army is matched by a respected staff of lawyers, policy experts and scientists, who work out of more than 30 offices nationwide. These conservation professionals ensure that TU is at the forefront of fisheries restoration work at the local, state and national levels.',
    'contact_name': 'Neil Hancock',
    'email': 'neilh20+turc1910@wllw.net',
    'link': 'https://www.tu.org/'
  }
})])

([('site', {

'code': 'TU-RC-01',

'name': 'TU-RC-01 LDSR PitTag Monitor',

'network': 'envirodiy',

'location': {

'latitude': '38.40731',

'longitude': '-122.81813',

'srs': 'EPSG:4236'

}

}), ('variable', {

'value_type': 'Instrument deployment',

'data_type': 'Average',

'general_category': 'Instrumentation',

'sample_medium': 'Surface water',

'no_data_value': '-9999.0000000000',

'code': 'All_ExternalVoltage_Battery',

'id': '215',

'name': 'Battery voltage',

'vocabulary': 'envirodiy',

'time': {},

'units': {

'abbreviation': 'V',

'code': '369',

'name': 'Volt',

'type': 'Electromotive force'

'description': None

}), ('values', [{

'value': '16.0',

'method_code': '2',

'quality_control_level_code': '1',

'censor_code': 'nc',

'date_time_utc': '2019-11-23T03:51:02',

'source_code': '611',

'time_offset': '-08:00',

'datetime': '2019-11-22T19:51:02'

}]), ('censor_codes', {

'nc': {

'censor_code': 'nc',

'description': 'nc'

}

}), ('methods', {

'2': {

'id': '2',

'code': '2'

}

}), ('quality_control_levels', {

'1': {

'id': '1',

'code': '1',

'definition': 'Raw Data'

}

}), ('sources', {

'611': {

'id': '611',

'code': '611',

'organization': 'Trout Unlimited',

'description': 'Today TU is a national organization with about 300,000 members and supporters organized into over 400 chapters and councils from Maine to Montana to Alaska. This dedicated grassroots army is matched by a respected staff of lawyers, policy experts and scientists, who work out of more than 30 offices nationwide. These conservation professionals ensure that TU is at the forefront of fisheries restoration work at the local, state and national levels.',

'contact_name': 'Neil Hancock',

'email': 'neilh20+turc1910@wllw.net',

'link': 'https://www.tu.org/'

}

})])

The debug out looks like this

** elem_svd[{
    'value': '16.0',
    'method_code': '2',
    'quality_control_level_code': '1',
    'censor_code': 'nc',
    'date_time_utc': '2019-11-23T03:51:02',
    'source_code': '611',
    'time_offset': '-08:00',
    'datetime': '2019-11-22T19:51:02'
  }] 
  
** full_value {
    'value': '16.0',
    'method_code': '2',
    'quality_control_level_code': '1',
    'censor_code': 'nc',
    'date_time_utc': '2019-11-23T03:51:02',
    'source_code': '611',
    'time_offset': '-08:00',
    'datetime': '2019-11-22T19:51:02'
  }
  
value(v) 16.0

** elem_svd[{

'value': '16.0',

'method_code': '2',

'quality_control_level_code': '1',

'censor_code': 'nc',

'date_time_utc': '2019-11-23T03:51:02',

'source_code': '611',

'time_offset': '-08:00',

'datetime': '2019-11-22T19:51:02'

}]

** full_value {

'value': '16.0',

'method_code': '2',

'quality_control_level_code': '1',

'censor_code': 'nc',

'date_time_utc': '2019-11-23T03:51:02',

'source_code': '611',

'time_offset': '-08:00',

'datetime': '2019-11-22T19:51:02'

}

value(v) 16.0

More...

Author

Posts

Viewing 13 reply threads

You must be logged in to reply to this topic.