Welcome to EnviroDIY, a community for do-it-yourself environmental science and monitoring. EnviroDIY is part of Home › Forums › Mayfly Data Logger › Publish to AWS?
- This topic has 9 replies, 3 voices, and was last updated 2021-08-04 at 7:34 AM by
dan@wachusett.
-
AuthorPosts
-
-
2021-07-06 at 9:00 AM #15654Hey EnviroDIY,
I am researching the use of AWS buckets as an alternative data endpoint for Mayfly logger output. Has anyone out there already tried doing this?
After some preliminary research it see
Hey EnviroDIY,I am researching the use of AWS buckets as an alternative data endpoint for Mayfly logger output. Has anyone out there already tried doing this?
After some preliminary research it seems like that I would have to do the following steps:
- Create my AWS bucket
- Set up a new publisher script for logging to AWS as the endpoint
- Set up AWS SDK to read the data stream and write it to a file in the bucket
If anyone else has interest in this or prior experience please let me know.
Thanks,
Dan -
2021-07-06 at 4:27 PM #15656Its often seems simple, but the practical side of it can be complex – like if certificates are needed. AWS can scale, but that requires a lot of layers. As I understand it AWS dices and slicesIts often seems simple, but the practical side of it can be complex – like if certificates are needed. AWS can scale, but that requires a lot of layers. As I understand it AWS dices and slices the options and the bills are a big surprise.
In software design the question that comes up is, what’s the functionality that is desired? How do you break the functionality into small layers. What currently works, and what is needed.
Does it have to be AWS or can it be something else?
https://thingspeak.com/ Can do data collection, and then my experience is the API can pull it off pretty easily. I did this with an experimental http://azonde.org/pm2/LiveRainGauge3.htm
The code is in the browser, not that I really follow how it all works!! and also here https://github.com/neilh10/thingspeakmultichannel-highcharts
The thingspeak costs are pretty well defined.
For the Mayfly which ever way you look at data collection, then reliably pushing data to the endpoint is also needed, which I’ve got in my sensors, and its on my list to PR but haven’t got there yet. https://github.com/EnviroDIY/ModularSensors/issues/194
-
2021-08-03 at 11:58 AM #15773@dan
I agree with @neilh in that AWS can be a little cumbersome. I have used it with the ESP32 platforms in the past via WiFi, but in the end opted for using other end points than AWS. Lots of free d
@danI agree with @neilh in that AWS can be a little cumbersome. I have used it with the ESP32 platforms in the past via WiFi, but in the end opted for using other end points than AWS. Lots of free dashboards out there for starting out and learning. Most of the really nice ones start adding costs once you decide to scale up the number of data points and devices, but still great for getting off the ground to you hone in what you are trying to achieve..
What were you planning on using for your network transmission device (cell modem, XBee, ESP)???
-
2021-08-03 at 12:20 PM #15774Thanks for your input @neilh and @vogelrnws. I work for a state agency, so I have limited options when it comes to IT related endeavors, especially things that are new (like technology from the lastThanks for your input @neilh and @vogelrnws. I work for a state agency, so I have limited options when it comes to IT related endeavors, especially things that are new (like technology from the last 20 years). I am basically trying to follow pathways that have been lightly trodden, rather than attempt to blaze my own trail.
I am in the process of building some new monitoring stations with Mayfly loggers and Hydros-21 sensors pushing data over the cell network with 4G LTE modems. I have to use our existing Verizon accounts to add data plans using the M2M service, which actually seems pretty reasonably priced. I don’t mind using MMW for real-time data access, however my reason for wanting to push data to AWS is that we can leverage our existing account with ESRI to ingest data files from AWS into GeoEvent Server, which would enable us to make customized dashboards to display our real-time monitoring data. I know there are other options, but I first want to see if I can go the AWS/ESRI route because this workflow already has momentum at another agency in my state, and I can sort of “draft” behind that effort.
Personally, I am open to other options (like Thingspeak), however the hurdles in setting up new contracts for IT related services are almost insurmountable. Its not even a matter of cost ($650 a year is a pretty easy sell). However, I basically need to prove that this project will not work with our existing IT tools/products/services before anyone will entertain another option.
It took me 18 months just to purchase 1 Mayfly board due to all the bureaucratic and procurement protocols that are in place!
Thanks,
Dan -
2021-08-03 at 12:26 PM #15775Are these Verizon Jetpacks with Wifi or do your modems have a serial connection? We started out with AWS, Verizon Jetpacks w/WiFi and the ESP32. It worked, we just didn’t have an active AWS contAre these Verizon Jetpacks with Wifi or do your modems have a serial connection? We started out with AWS, Verizon Jetpacks w/WiFi and the ESP32. It worked, we just didn’t have an active AWS contract in place so the free trial accounts were running out of room for us. We at that point just set up an internal MQTT Broker so we (National Weather Service) didn’t have to pay anything for the service.
-
2021-08-03 at 12:36 PM #15777@vogelrnws, We are using 4G LTE modems that are recommended by EnviroDIY with the LTE Bee Adapter. Verizon will se@vogelrnws, We are using 4G LTE modems that are recommended by EnviroDIY with the LTE Bee Adapter. Verizon will send us SIM cards once we give them a list of IMEI numbers for the modems. Fortunately, our agency has access (with approval) for certain AWS storage/computing services, so I can take advantage of this since this project is relatively small, data-wise. I will only have 7 sensors sending data at 15 minute increments.
The main hurdles I need to address are items 2 and 3 in the original post on this thread. Basically, setting up a publisher script to push to an amazon endpoint, and configuring that endpoint to process the JSON data streams that get sent every 15 minutes.
-
2021-08-03 at 12:54 PM #15778
-
2021-08-03 at 3:00 PM #15784I wonder if the AWS interface can be done through some form of @vogelrnws python that runs on the Xbee LTE? JMight be worth looking for somI wonder if the AWS interface can be done through some form of @vogelrnws python that runs on the Xbee LTE? JMight be worth looking for something that runs on the raspberryPi in Python for AWS, and then seeing if they could be adapted to Xbee LTE python. (so easy to throw out an idea – but I have no experience with AWS)
-
2021-08-03 at 4:03 PM #15788
-
2021-08-04 at 7:34 AM #15793@vogelrnws, thanks for the digi link… this does seem like something that could work very well to send information to AWS from a XBee modem. There are still several steps to figure out in terms o@vogelrnws, thanks for the digi link… this does seem like something that could work very well to send information to AWS from a XBee modem. There are still several steps to figure out in terms of getting a Mayfly program to execute the python code that is on the modem, perhaps like calling a function?, with the sensor data as the message formatted in a JSON string. These details are way beyond my capabilities, so I can’t really pursue this option on my own. I am hoping to come up with a solution that utilizes existing modular sensor libraries with some minor tweaks. Maybe this is unrealistic, but I have neither the time or ability to develop a novel solution from scratch, on my own. I am very thankful to be able to take advantage of all the existing work and know-how in this community. I will keep researching options and report back if I find a solution that is within my technological reach.
-
-
AuthorPosts
- You must be logged in to reply to this topic.
