Welcome to EnviroDIY, a community for do-it-yourself environmental science and monitoring. EnviroDIY is part of Home › Forums › Mayfly Data Logger › Manual upload of CSV files on MonitorMyWatershed
- This topic has 13 replies, 5 voices, and was last updated 2021-07-15 at 2:09 PM by
Robert S.
-
AuthorPosts
-
-
2020-04-03 at 10:09 AM #14019I don’t know if this is related to the issue reported by @mbarney (monitormywatershed.org is not respondingI don’t know if this is related to the issue reported by @mbarney (monitormywatershed.org is not responding) or to the issue posted on GitHub by @dbressler (monitor my watershed won’t load, just spins #477) but manual uploading of CSV files is not working either.
The page sits idle until the “Failed to upload data” message appears at the bottom of the page. I have tried several time over the past couple of days, but with no change.
Robert
-
2020-04-03 at 6:06 PM #14023Hi Robert,
I just now tried and am also unable to upload a CSV. In the past, I had trouble due to formatting or due to files that were too long, but once I figured that out, I saved examples of files
Hi Robert,I just now tried and am also unable to upload a CSV. In the past, I had trouble due to formatting or due to files that were too long, but once I figured that out, I saved examples of files that I’ve successfully uploaded. However, those are now failing as well. Sample csv attached, FWIW.
Matt
Attachments:
-
2020-04-03 at 7:21 PM #14025
Thanks for the verification Matt.
I was aware that large files can be a problem. I am trying to upload 2 days worth of readings (about 290 lines) but see your file is even smaller.
Robert
Attachments:
-
2020-04-07 at 5:47 AM #14033Well, I have pretty much answered my own question.
After doing some searching on GitHub, I came across Issue #303 ( Allow POST of da
Well, I have pretty much answered my own question.After doing some searching on GitHub, I came across Issue #303 ( Allow POST of data, even when one variable UUID is wrong), which got me looking hard at the .csv data I was trying to upload. It really didn’t become clear to me until I grabbed some new data from the sensor last night.
If you have four headers defined (time, water temp, voltage, board temp) and you drop the last one so there are only three data fields, the upload will fail. Just one of the pitfalls of touching the data too much.
Funny in a way because I had changed the code in my logger (some time ago) to output the exact format I need to upload but haven’t been out to the sensor to upload the new code.
Robert
-
2020-04-07 at 4:03 PM #14034
The csv uploading tools are very sensitive to an irregularities in the files. We would very much like to improve them, but right now that’s pending time and, mostly, budget.
-
2020-04-07 at 5:36 PM #14035I think that sensitivity is a good thing when collecting data and I want the system to tell me I am doing something wrong. If I could change anything, it might be to tell me what I am doing wrong.
Ei
I think that sensitivity is a good thing when collecting data and I want the system to tell me I am doing something wrong. If I could change anything, it might be to tell me what I am doing wrong.Either way, this was a case of “User Error” and Bad Practice Management. I need to get my logger updated so that the .cvs file it is producing is in the EXACT format it needs to be for uploading. Then I don’t have to fiddle with copy&paste and run the risk of corrupting or altering the data inadvertently.
Robert
-
2020-04-08 at 3:55 PM #14043It would be so much better if the website at least told you *why* and upload failed! All you get is a “failed” which could mean anything – the format was bad, theIt would be so much better if the website at least told you *why* and upload failed! All you get is a “failed” which could mean anything – the format was bad, the file was too big, there was an internet blip, who knows. The only way right now to even get an inking as to why it fails is to turn on “developer tools” (F12 in Firefox, I think also in Chrome) in your browser before posting the data and then searching through all the many tiny requests on the console for the right post request and hoping the error message is there and is interpret-able – and often it isn’t.
-
2020-04-08 at 5:08 PM #14045
Hmm, turning on the console never occurred to me. I will have to remember that one.
Thanks for your insight and knowledge Sara. I appreciate all of the useful conversation.
Robert
-
2020-08-17 at 11:28 AM #14473I have been having similar problems uploading to the internet. They seem to have become worse the last few months.
I make it a point not to change the original data file, I open it but don’t
I have been having similar problems uploading to the internet. They seem to have become worse the last few months.I make it a point not to change the original data file, I open it but don’t save it before closing.
I use an Excel template to create an upload file. The template contains the first 8 rows of the upload file. All I have to do to those rows is cut and paste in the Data Logger File name from the data file. This gives me exactly the same first 8 rows every time, except the file name.
I then copy and paste in the data from the original .csv file at row 9.
I have an Excel Macro that deletes the SampNum data col for 1,000 rows below row 9 in the upload file and shifts the cells left so my data is now in the right columns and all the header information correct.
I have a second Excel Macro that formats the Date and Time column for 1,000 rows below row 9.
I’ve been using the Excel template since April. The macros are relatively new.
My upload failure rate has steadily got worse over time. Lately it is successful maybe 10% to 20% of the time.
I upload daily so my upload files are about 60 rows long. That is unless I am not successful uploading a days data. In that case I add the additional days data below the previous days and try to upload all. Right now I have a file with 3 days data I have been unable to upload after about 20 tries at various times throughout the days. I am well below the 1,000 rows I’ve been hearing is the maybe the limit.
The reason for our watershed group funding these stations was to get baseline data on our river. A major justification for our station was for the fishermen in our watershed group and others to have data on river conditions so they know whether or not to make the trip up to fish on their holidays and weekends. About 75% of the residences in the watershed are not owner occupied, they are vacation homes people drive to, very commonly 2 to 4 hours. So we need timely data and since we have NO reliable cell service at this location I am working on an xbee radio link for it and using this method during the long learning curve. My intent is to use a mesh network to establish a base for upload where we have solid internet. Without timely data I risk losing support for the project.
We are in a fairly remote area and satellite internet is the only available option and it may play into this with latency or other issues. The Covid pandemic has put an additional load on the internet and is certainly not helping things. I also have a Wilson cell booster system in my house so I have cell service (well, most of the time) and the results using a mobile hot spot connection are about the same.
I am pretty sure my upload files are consistent and correct and there is a problem in the upload process and not my files. I would really appreciate it if you could get it prioritized to be fixed/upgraded. It appears to be a very common problem.
-
2020-08-17 at 5:13 PM #14478@armas
I can’t think of any one thing that may be wrong (internet service, formatting, etc) without seeing some of the data. Could you maybe post some small samples of the files your are workin
@armasI can’t think of any one thing that may be wrong (internet service, formatting, etc) without seeing some of the data. Could you maybe post some small samples of the files your are working with (raw – from the logger and processed – after Excel macros)?
I found that the best way was to have the Mayfly output the log file data exactly the way I want to upload. I can then upload the .csv file straight to MMW without touching the contents of the file (I do rename the file to the date range it collected).
Robert
-
2020-08-21 at 3:41 PM #14516I’ve attached two of my files from earlier today. The file with 08-20 is the raw data file created yesterday right from the Mayfly, and the 08-21 is the upload file created today as I describeI’ve attached two of my files from earlier today. The file with 08-20 is the raw data file created yesterday right from the Mayfly, and the 08-21 is the upload file created today as I described earlier. I have been unable to upload the 08-21 file as of now. I tried three times earlier this morning and twice this afternoon. The third file, 08-19_20 is the one I uploaded successfully yesterday, but I went back and formatted the date/time column manually when I added the data for the second day. It almost seems like the system is looking for a change flag somewhere once an upload fails because if I go in and do redo something, like reformat the date/time just like it was, it will upload.
My C programming skills back in April when I loaded the program were not such that I was capable of modifying the output file. I was thrilled when I got it to do what is is. I’ve advanced some since and could maybe get in there and do it. The Excel template and macros seemed like the way to go while I figured out the radios and automatic upload. I’ve got the xbee’s working now and am working through the ethernet and web connection, I haven’t got to the upload file formatting part yet…
-
2020-08-21 at 8:24 PM #14520Okay, here is what I see …
Note: I am no expert but having an extra set of eyes helps sometimes.
I set up a dummy site and tried to replicate what I saw in your files. You can have a look at t
Okay, here is what I see …Note: I am no expert but having an extra set of eyes helps sometimes.
I set up a dummy site and tried to replicate what I saw in your files. You can have a look at the site to compare it to what you have set up and let me know if it seems different. I will leave it up for a bit but it will be removed if we have succeeded in solving your problems. Of course the name and specific site information will be different.
https://monitormywatershed.org/sites/STLWTR-TEST/
I managed to get all of the data to upload, but in the process I noticed the following things about your files:
* The file “190363_Armas_2020-08-21.csv” has a whole mess of empty values at the end of the file (,,,,,,). These empty commas should be removed from the file or it will not upload.
* The file “190363_Armas_2020-08-20.csv” has an extra column named “SampNum”. If this is not defined in your site/sensor list, it will fail to upload because it does not match the defined values of the site. It also has no UUIDs defined in it which will also cause it to fail.
* The last file (which you said you were able to upload) “190363_Armas_2020-08-19_20.csv” appears to be a valid file.
I may be wrong, but I think that your files just need to have all the right parts in the right order to upload successfully.
I have attached a file “190363_Armas_Formatted.csv” which contains all of the data from the first two files and “should” upload correctly.
Please let me know if this helps to solve your problem.
Robert
Attachments:
-
2021-07-15 at 1:04 PM #15683I am working through manually uploading site data and I get the “upload successful” message, but it’s not actually uploaded? Has anyone come across this issue?
I attached an example
I am working through manually uploading site data and I get the “upload successful” message, but it’s not actually uploaded? Has anyone come across this issue?I attached an example of a file that I am trying to upload for this site – http://monitormywatershed.org/sites/SL069/
Would anyone be able to look over the file to make sure I have everything right?
Attachments:
-
2021-07-15 at 2:09 PM #15685Not sure what the problem is, but the file seems to be valid.
I set up a test site, added the same sensors and changed the UUIDs in the file you attached to match what I set up and it uploaded with n
Not sure what the problem is, but the file seems to be valid.I set up a test site, added the same sensors and changed the UUIDs in the file you attached to match what I set up and it uploaded with no problem.
Test Site (STLWTR-TEST) (monitormywatershed.org)
My first guess would be to check that the UUIDs are correct for the site you are uploading to.
Please let me know if this is any help to you or if we need to dig deeper.
Robert
-
-
AuthorPosts
- You must be logged in to reply to this topic.
