Manual upload of CSV files on MonitorMyWatershed

Hi Robert,

I just now tried and am also unable to upload a CSV. In the past, I had trouble due to formatting or due to files that were too long, but once I figured that out, I saved examples of files that I’ve successfully uploaded. However, those are now failing as well. Sample csv attached, FWIW.

Matt

The csv uploading tools are very sensitive to an irregularities in the files.  We would very much like to improve them, but right now that’s pending time and, mostly, budget.

It would be so much better if the website at least told you *why* and upload failed!  All you get is a “failed” which could mean anything – the format was bad, the file was too big, there was an internet blip, who knows.  The only way right now to even get an inking as to why it fails is to turn on “developer tools” (F12 in Firefox, I think also in Chrome) in your browser before posting the data and then searching through all the many tiny requests on the console for the right post request and hoping the error message is there and is interpret-able – and often it isn’t.

I have been having similar problems uploading to the internet.  They seem to have become worse the last few months.

I make it a point not to change the original data file, I open it but don’t save it before closing.

I use an Excel template to create an upload file.  The template contains the first 8 rows of the upload file.  All I have to do to those rows is cut and paste in the Data Logger File name from the data file.  This gives me exactly the same first 8 rows every time, except the file name.

I then copy and paste in the data from the original .csv file at row 9.

I have an Excel Macro that deletes the SampNum data col for 1,000 rows below row 9 in the upload file and shifts the cells left so my data is now in the right columns and all the header information correct.

I have a second Excel Macro that formats the Date and Time column for 1,000 rows below row 9.

I’ve been using the Excel template since April.  The macros are relatively new.

My upload failure rate has steadily got worse over time.  Lately it is successful maybe 10% to 20% of the time.

I upload daily so my upload files are about 60 rows long.  That is unless I am not successful uploading a days data.  In that case  I add the additional days data below the previous days and try to upload all.  Right now I have a file with 3 days data I have been unable to upload after about 20 tries at various times throughout the days.  I am well below the 1,000 rows I’ve been hearing is the maybe the limit.

The reason for our watershed group funding these stations was to get baseline data on our river.  A major justification for our station was for the fishermen in our watershed group and others to have data on river conditions so they know whether or not to make the trip up to fish on their holidays and weekends.  About 75% of the residences in the watershed are not owner occupied, they are vacation homes people drive to, very commonly 2 to 4 hours.  So we need timely data and since we have NO reliable cell service at this location I am working on an xbee radio link for it and using this method during the long learning curve.  My intent is to use a mesh network to establish a base for upload where we have solid internet.  Without timely data I risk losing support for the project.

We are in a fairly remote area and satellite internet is the only available option and it may play into this with latency or other issues.  The Covid pandemic has put an additional load on the internet and is certainly not helping things.  I also have a Wilson cell booster system in my house so I have cell service (well, most of the time) and the results using a mobile hot spot connection are about the same.

I am pretty sure my upload files are consistent and correct and there is a problem in the upload process and not my files.  I would really appreciate it if you could get it prioritized to be fixed/upgraded.  It appears to be a very common problem.

I’ve attached two of my files from earlier today.  The file with 08-20 is the raw data file created yesterday right from the Mayfly, and the 08-21 is the upload file created today as I described earlier.  I have been unable to upload the 08-21 file as of now.  I tried three times earlier this morning and twice this afternoon.  The third file, 08-19_20 is the one I uploaded successfully yesterday, but I went back and formatted the date/time column manually when I added the data for the second day.  It almost seems like the system is looking for a change flag somewhere once an upload fails because if I go in and do redo something, like reformat the date/time just like it was, it will upload.

My C programming skills back in April when I loaded the program were not such that I was capable of modifying the output file.  I was thrilled when I got it to do what is is.  I’ve advanced some since and could maybe get in there and do it.  The Excel template and macros seemed like the way to go while I figured out the radios and automatic upload.  I’ve got the xbee’s working now and am working through the ethernet and web connection, I haven’t got to the upload file formatting part yet…

I am working through manually uploading site data and I get the “upload successful” message, but it’s not actually uploaded? Has anyone come across this issue?

I attached an example of a file that I am trying to upload for this site – http://monitormywatershed.org/sites/SL069/

Would anyone be able to look over the file to make sure I have everything right?