Reply To: Data Dropped Between Mayfly and MMW (2.4%)

Well it’s certainly not intended that less than 100% of the data makes it online, if that’s what your asking.

Beyond that I can’t really help without a lot more information. So, here goes:

Do you know where in the chain the data is being lost? Are you using a Mayfly? What code are you running? If ModularSensors, what version of both ModularSensors itself and of TinyGSM. What firmware version do you have on the XBee3? What firmware version do you have on the SARA R410 inside of the XBee3? Are you using the the XBee3 in transparent or bypass mode?

1. Is 100% of the data on your SD card? If so, the sensors are working and the logging portion of the code on your board is working fine; move on to 2. If not:
1a. Are you recording the “ProcessorStats_SampleNumber?” If so, is this number being reset after a gap? If so, something is causing your board to freeze until the watch-dog timer goes off. That could be an electrical/wiring issue or a code issue.
1b. If you’re not recording sample number, How long is the gap between points when data is dropped? If the gaps are always either 15 or 20 mintues long, that’s probably the watch dog. It bites after 15 minutes, but you may miss the first new point after it bites leaving a 20 minute gap.
1c. If you think it’s the watchdog:
1ci. Are you using any I2C sensors and are disconnecting their power between sampling. If the I2C bus is held low, it will cause the low level I2C/Wire library to hang on the next attempt to communicate with any I2C device (like the RTC).
1cii. There is also an issue with the XBee3 in bypass mode that sometimes causes hangs. I know it happens. I can’t figure out why. You can read about it on GitHub. I’ve only seen this sort of hang once every few days or so, though, not 2.4% of the time.

2. Is the issue with both wifi and cellular or just with the cellular? If both, skip to 4.

3. Can you see your data use by individual sessions for your cellular plan (or wifi connection)? If you are using a hologram data plan, you can see this on their dashboard.
3a. When looking at the data sessions on hologram, do you see a session at every single 5 minute interval. If not, you either don’t have good enough service or there’s some other issue with the modem seeing and registering to the internet.
3b. If you see a session at every interval, do you see actual data being used on that session or are they 0 byte sessions? When running the XBee3 in transparent mode it will sometimes continue to report that it is unable to register to the network even though on hologram’s side the registration has been successful. At other times (again, when in transparent mode) the XBee3 fails to properly packet the data and actually send it out. Both of these things happens more on older version of the XBee3 firmware, but I’ve still seen it on the current version. I’ve tried a number of things to try to help the connection, but unfortunately this is on Digi to fix. I have had many, many fewer issues with empty sessions when running in Bypass mode, at the trade off of a know bug causing rare hangs in bypass.

4. Finally, if you’re see the data on your SD card, you’re seeing sessions that consume at every single five minute interval, and your still not seeing data on the data portal, it could be something wrong either with the portal receiving the data or displaying it.
4a. Is the data missing no matter how you attempt to view it? (ie, sparklines, pop-up table, downloaded csv from the data portal, plots in WebTSA, csv download from WebTSA) If it’s missing everywhere it’s *probably* not being processed and put into the “main” databased correctly.
4b. If you can see data on any place in the database, but just not all of them, there are known communication issues between the “main” database and the higher performance “daughter” databases that produce the various types of plots and other ways of viewing the data. The data isn’t lost and hopefully these issues will be fixed soon, but the only solution in this case is to wait.