2022-02-13 at 3:49 PM #16440
Around 50% of my sites have a problem where the CTD drops out on a regular (not predictable) basis. This results in -9999 values which may persist for a few readings before returning to normal.
I still get a data record out of this, but the resolution is lower and I would like to fix it if possible.
Has anyone experienced this before? I have tried updating the code to the complex loop (to prevent the MODBUS wing interfering with the CTD) and replacing the stereo jack adapter.
I have attached an image below, or you can see for yourself via this link:
Any help is much appreciated.
2022-02-13 at 4:43 PM #16442Shannon HicksModerator
We’ve seen that very rarely, usually it happens when the connection between the Mayfly and the sensor is bad, and the fault is either mechanical damage to the sensor cable (rodents chewing, flood damage, etc) or corrosion on the contacts inside the Grove stereo jack due to moisture inside the logger enclosure, or bad connection of the Grove cable between the stereo board and the Mayfly. So whenever we see this sort of pattern in a CTD sensor, we replace the Grove cable and stereo jack, and 95% of the time, that solves the problem. In the rest of the cases, it’s either damage to the sensor cable or the electronics inside the sensor itself are failing (usually due to impact to the sensor body from large debris during a storm).
But if you’ve already replaced the Grove jack (and the cables?) and you’re seeing this on more than one station, then it’s likely a code or voltage issue. What version of the Mayfly board are you using (v0.5b or v1.0)? I don’t know how you’re powering the RS-485 sensors, but if they are being powered using a boost circuit that’s powered from the same voltage regulator on the Mayfly that powers everything else on the board (including the CTD sensor), then it’s possible that a voltage drop during the sampling period is causing the CTD sensor to now be fully powered, thus causing a communication issue between the Mayfly and CTD, resulting in the Mayfly reporting -9999 as the CTD sensor parameters. Have you tried just unplugging the turbidity sensor from your setup (but don’t change the logger code) and see if you’re able to get reliable communication with the CTD sensor again? If so, that would indicate the problem is related to the turbidity sensor being added. Or it could just be a software issue due to timing or other problems with the sketch. You can email me your sketch if you’d like me to take a look at it, once you’ve ruled out all the other obvious physical failures.
Other troubleshooting ideas: If you’ve got one of the ZSC bluetooth sensor interfaces from Meter Group, you could test that the sensor is operating fine independently of the Mayfly, or you could program a spare Mayfly and connect it to only the CTD sensor (first with and then without a cellular board) and have it take frequent readings (like 1 minute) and let it run for a few hours to see if it drops any readings.
2022-02-14 at 12:29 AM #16443
@James ouch. Looking at the data file, its just the Decagon CTD/SDI-12? readings that are responding -9999 . Typically that means the (SDI-12?) protocol has failed to communicate with the instrument.
At the same time Yosemite/Modbus? is responding well.
The CTD is only occasionally failing, or failing in groups, but still working some of the time. A hard failure where it fails all the time is easiest to debug. Intermittent failures are the hardest.
You started running the setup 2021-10-25 11:16 and
then it started failing 2021-12-09 14:45
and recovered 2021-12-14 13:00
Did you do anything to have it recover?
Since then it has intermittent failures. Ouch that is the hardest to troubleshoot.
I have one LT500 that is not working well with SDI-12, its mostly failing, but sometimes works. Similarly I purchased a device that is tested to the full SDI-12 device, and its also failing a lot. SDI-12 from the Mayfly is not a standard 0-5V, so I’m planning on moving it to the Modbus. I’m also trying to figure out a way of generating a fully specified SDI-12 0-5V
2022-02-14 at 3:19 AM #16445
I’m leaning more towards an interaction with the MODBUS wing. I had one station that produced -9999’s constantly when the Turbidity sensor was plugged in, and operated fine when I removed the MODBUS wing. This drove me nuts until Anthony pointed me towards the complex loop code where I set the Turbidity sensor to low after each reading to avoid bleeding voltage. I have subsequently uploaded the new code to all stations, which I assumed would fix the intermittent problems, but that doesn’t seem to be the case.
Other factors: there is no damage to the cables, no corrosion, and no perceivable storm damage.
Shannon, I’m using v0.5b and the sensor is powered from the grove slot (D10/D11). I changed the voltage jumper to 5v because 3.3v was resulting in -9999 values (I assume because of brownout?).
I haven’t tried unplugging the MODBUS wing, but I bet the CTD record would be perfect if I did. I can test this later this week if we don’t find another solution.
Perhaps it’s my code (see below)? Can either of you see any obvious errors?
Thanks for your help.
2022-02-15 at 10:05 PM #16482
2022-02-15 at 10:07 PM #16483Shannon HicksModerator
I’m not familiar with the code regarding the complex loop since it’s not something I’ve ever used, and I’ve also not used one of those modbus wings, so I think Anthony or Neil will need to be the ones to help with this issue.
2022-02-16 at 1:19 AM #16484
@James_NZ – are all your setups the same?.
Have you got a spare setup that can be debugged in a local setting? (office with terminal connected)
I have a similar setup with an RS485 and SDI-12 on one system – but I maintain a test system to verify it before I deploy it.
I have done work to ruggedize it. All I can do is describe how I do it.
2022-02-16 at 2:24 PM #16497
@neilh20, yes all setups are the same. Some work fine, others have issues with dropping out. I have spare Mayflies and a spare CTD, but no spare turbidity sensor. Did you ever end up producing your Mayfly wings for the 0.5? This might just be another case of rubbish RS485/ttl’s.
I kinda wish my project was delayed by six months so I could get the version 1.0’s with the new modbus wing!
2022-02-16 at 6:58 PM #16498
@james_nz my process is to stage systems in an accessible context to check the reliability of the systems. This comes from the school of hard knocks working with embedded processors like the Mayfly.
IMHO while ModularSensors software has some fantastic classes to be able to access the instruments and also to push to cloud locations, Arduino Open Source is generally at the level of – Arduino teaching ~ this is how it can be done.
Its next to impossible to teach “reliability” – its usually learnt through being on the “bleeding edge”. The 2nd time around one tends to add more foundational capability to better cope with reliability testing.
You reference “dropping out” which can be a whole record or one instrument. A record lost is typically due to “internet failure”, but still stored on the uSD.
Individual readings, usually digital instrument, when the digital comms link to the digital instrument fails.
It seems, for the above referenced node, your RS485 instruments are responding OK, its the SDI-12 instruments that are having comms failures.
Its should be noted that the Mayfly’s xx could be said to support “Arduino SDI-12”, that is they show how it can be done, but there is a lot of technical debt. It isn’t per the SDI-12.org specification for signaling and protection.
I’ve been trying to prototype a solution to this for some time. Right now my RS485board includes SDI-12 protection and level shift. However I still need to work on the SDI-12 software for it to cope with the level shift. https://www.envirodiy.org/topic/sdi-12-level-shifting/
Currently when you access the Mayfly X with the USB cable (or even the FTDI cable), it causes the Mayfly to reset – a nice Arduino feature of always making sure the board can be programmed. However that reset isn’t good for reliability monitoring – that is plugging in and just monitoring how its been working – I’ve created a modified FTDI connector to allow that https://github.com/neilh10/ModularSensors/wiki/Test-Equipment-FTDI-cable
2022-03-01 at 7:14 PM #16651SGFultonParticipant
Hi James, we regularly had this problem when using the Decagon CTD-10s connected to a DIY Arduino Mega data logger. We ended up taking the median value of 10 measurements taken 1x/second every 5 mins. However, what eventually worked (if I’m remembering this correctly; this was my dissertation research done in 2016-2019) was clamping an RF ferrite balun filter to the cables. They looked something like this: https://www.digikey.com/en/products/detail/fair-rite-products-corp./0444173951/8594098?utm_adgroup=Ferrite%20Cores%20-%20Cables%20and%20Wiring&utm_source=google&utm_medium=cpc&utm_campaign=Shopping_Product_Filters_NEW&utm_term=&utm_content=Ferrite%20Cores%20-%20Cables%20and%20Wiring&gclid=CjwKCAiApfeQBhAUEiwA7K_UH7XcVuubBeU3WlGAQFSizFtYSRyiAbgXnIgNb0TLWobKsUVKiNRBVxoCfTAQAvD_BwE
I think it must’ve been noise interference from somewhere on our board. Hope this helps! I have code you’re welcome too if you’re interested.
2022-03-03 at 9:19 PM #16666
2022-03-09 at 2:38 PM #16689Sara DamianoModerator
If the communication works sporadically, it’s more likely to be an electrical issue than a code issue. Coding issues *usually* cause it to not work at all.
The -9999 is the library’s value for a failed reading. When you get in in SDI-12, it most often means the sensor didn’t respond or the response was garbled. Because the Mayfly’s voltage (and thus serial communication) range is only 3.3V, it doesn’t take much electrical noise to garble things up.
- You must be logged in to reply to this topic.