Welcome to EnviroDIY, a community for do-it-yourself environmental science and monitoring. EnviroDIY is part of Home › Forums › Mayfly Data Logger › Debug flags for modem comms
Tagged: LTE, ModularSensors, Xbee, XBee3
- This topic has 24 replies, 4 voices, and was last updated 2020-01-24 at 12:45 PM by
Matt Barney.
-
AuthorPosts
-
-
2020-01-16 at 2:36 PM #13670Jake (@jlemontu-org) and I are troubleshooting a new Mayfly (from this thread) that is unable to connect to the internet. WeRJake (@jlemontu-org) and I are troubleshooting a new Mayfly (from this thread) that is unable to connect to the internet. We’ve attempted to install it in place of an existing Mayfly station that has working connectivity. To do so, we move the LTE adapter (with modem, antenna, and SIM) to the new board. Today Jake retrieved the mayfly, with modem, etc., from the field for some desktop testing while connected to the Serial Monitor in PlatformIO. Still no connectivity, so I enabled debug flags (based on an earlier post) as follows:123456789101112131415build_flags =-DSDI12_EXTERNAL_PCINT-DNEOSWSERIAL_EXTERNAL_PCINT-DMQTT_MAX_PACKET_SIZE=240-DTINY_GSM_RX_BUFFER=64-DTINY_GSM_YIELD_MS=2; additional build_flags for modem debugging:-DTINY_GSM_DEBUG=Serial-DMS_LOGGERMODEM_DEBUG-DMS_LOGGERMODEM_DEBUG_DEEP-DMS_DIGIXBEE_DEBUG-DMS_DIGIXBEECELLULARTRANSPARENT_DEBUG-DMS_DIGIXBEECELLULARTRANSPARENT_DEBUG_DEEP-DMS_DIGIXBEELTEBYPASS_DEBUG-DMS_DIGIXBEELTEBYPASS_DEBUG_DEEP
I was then unable to build the code, receiving the errors shown in the attached txt file.
Am I using incompatible build flags? This same combination of flags works on my PC, but apparently not on Jake’s.
Second build flag question: Some of the ModularSensors examples on github include the following 4 flags, and some do not. I’m unsure whether I need them or not:
1234-DNEOSWSERIAL_EXTERNAL_PCINT-DMQTT_MAX_PACKET_SIZE=240-DTINY_GSM_RX_BUFFER=64-DTINY_GSM_YIELD_MS=2(Confession: I haven’t read the docs regarding build flags, so please feel free to just direct me to a good source there.)
Thank you!
MattAttachments:
-
2020-01-16 at 2:42 PM #13672One additional detail: after it failed to build, I commented out –DTINY_GSM_DEBUG=Serial, and it built, but of course, got no debugginOne additional detail: after it failed to build, I commented out –DTINY_GSM_DEBUG=Serial, and it built, but of course, got no debugging info in Serial Monitor.
-
2020-01-16 at 3:36 PM #13674What?? You haven’t read all the docs?? Actually, I don’t remember how well documented the build flags are. Writing good documentation is hard.
Are you using transparent or bypass? Y
What?? You haven’t read all the docs?? Actually, I don’t remember how well documented the build flags are. Writing good documentation is hard.Are you using transparent or bypass? You can remove the build flags for whatever you’re not using. You also don’t need the MQTT flag, it only applies to ThingSpeak (or anything else MQTT). You also don’t need the NeoSWSerial flag unless you’re using that library. Having those extra defines shouldn’t be a problem though. Any build flag that starts with a ‘-D’ is the same as typing “#define” in your code. When you hit “build” one of the first steps is for the pre-processor to essentially find-and-replace everything you’ve “defined” with the value you’ve defined it as.
Jake must have just updated TinyGSM. I’d added a line to the debugging just a few days ago but screwed up and tried to print the value of a pointer. I just fixed that and pushed. You can both update again (“pio lib update”).
If you have an Arduino mega or other board with more than two serial ports available, I find debugging the modem can be a bit easier if you can “snoop” the modem Rx and Tx lines with a separate board.
-
2020-01-16 at 5:21 PM #13675Great, thanks, we’ll try the library update and see if we can get some useful traces. We’re using transparent.
Reading docs is on my to-do list. =:D Good documentation is definitely har
Great, thanks, we’ll try the library update and see if we can get some useful traces. We’re using transparent.Reading docs is on my to-do list. =:D Good documentation is definitely hard! Actually, the Mayfly manual is impressively good, as are the docs in ModularSensors github repo.
-
2020-01-16 at 5:27 PM #13676@ensign and others (Rachel Johnson, Matt Gisondi, David Bressler, @shicks, @heather and a few more) have put a lot of work into that manual!
But we still have a 1-star review on Amazon for i
-
2020-01-16 at 5:55 PM #13677Haha… If your metric is Amazon reviews, well…. 🙂
The doc effort definitely shows. I first came to a Stroud workshop in ~12/2017; then we (TU) jumped in deep last summer or fall w
Haha… If your metric is Amazon reviews, well…. 🙂The doc effort definitely shows. I first came to a Stroud workshop in ~12/2017; then we (TU) jumped in deep last summer or fall with the Mayfly, so I picked it up again, and wow! what an advance in documentation! And y’all even incorporate user input – quickly!
One day, hopefully soon, we intend to be a value-add rather than a resource drain for the Mayfly-EnviroDIY community. 😀
-
2020-01-17 at 11:25 AM #13679Hmm… still having errors with the TINY_GSM_DEBUG macro after updating libraries. Here’s the pio lib update’s output:12$ pio lib updateLibrary StHmm… still having errors with the TINY_GSM_DEBUG macro after updating libraries. Here’s the pio lib update’s output:1234567$ pio lib updateLibrary Storage: C:\Users\Jacob.Lemon\Documents\PlatformIO\Projects\BearCT\.pio\libdeps\mayflyUpdating EnableInterrupt @ 1.1.0 [Up-to-date]Updating EnviroDIY_DS3231 @ 1.3.2 [Up-to-date]Updating EnviroDIY_ModularSensors @ 0.23.16 [Up-to-date]Updating SdFat @ 1.1.0 [Up-to-date]Updating StreamDebugger @ 1.0.1 [Up-to-date]
And the errors from the subsequent build are in the attachment. I also tried Clean, pio lib update, and then Build, with the same result.
Thanks,
MattAttachments:
-
2020-01-17 at 12:45 PM #13682You are exactly 1 commit behind for TinyGSM. Try again. Have you maybe installed TinyGSM to your “global” library registry instead of your local one? You might need to do a “piYou are exactly 1 commit behind for TinyGSM. Try again. Have you maybe installed TinyGSM to your “global” library registry instead of your local one? You might need to do a “pio lib -g update” (-g flag for global). Otherwise you can navigate directly to into the folder and use git to update:
cd .pio\libdeps\mayfly\TinyGSMfollowed bygit pull.Attachments:
-
2020-01-17 at 4:13 PM #13685
Global update did it – thanks!
-
2020-01-17 at 7:04 PM #13687Well, we’re still unable to connect to the internet. We were using transparent mode on the XBee3, but I tried switching to bypass, and still no luck. Then, thinking that the project’s librWell, we’re still unable to connect to the internet. We were using transparent mode on the XBee3, but I tried switching to bypass, and still no luck. Then, thinking that the project’s library/build environment might somehow be corrupt, I created a new PlatformIO project and pasted the code into new files there, still no luck. I’ll attach two logfiles here and hope you can spot something that we can fix. Thanks again!
-Matt
Attachments:
-
2020-01-17 at 9:18 PM #13690You might try using a new SIM card or a new LTEBee module. We’ve had several of them stop working recently for no reason at all. Sometimes using a brand new SIM card will fix the problem, buYou might try using a new SIM card or a new LTEBee module. We’ve had several of them stop working recently for no reason at all. Sometimes using a brand new SIM card will fix the problem, but other times it requires using a new LTEbee module. We haven’t determined the cause of the failure yet, nor a solution to get them working again.
-
2020-01-18 at 8:36 AM #13692OK. What’s perplexing to me about this one is that Jake’s been able to reinstall the same LTEBee and SIM back into the “old” Mayfly (the one we’re trying to swap out), anOK. What’s perplexing to me about this one is that Jake’s been able to reinstall the same LTEBee and SIM back into the “old” Mayfly (the one we’re trying to swap out), and it will connect and upload data successfully.
Thanks for continuing to sleuth out all the LTEBee issues.
-
2020-01-18 at 2:21 PM #13694What is the battery voltage that your Mayfly is recording in your data file? We’ve had a few random failures of a mosfet on the board that will cause the Mayfly to read the incoming Lipo or USWhat is the battery voltage that your Mayfly is recording in your data file? We’ve had a few random failures of a mosfet on the board that will cause the Mayfly to read the incoming Lipo or USB voltage as being much lower than it really is (see this post: https://www.envirodiy.org/topic/innacurate-lipo-battery-voltage-data/). Your logger code probably has a line in it that prohibits the bee from transmitting when the battery gets low, so it’s possible that your board has a bad mosfet which is tricking the Mayfly into not transmitting.
-
2020-01-20 at 11:42 AM #13697I’m sorry. No, I don’t think it’s an issue with the battery or the SIM.
When you were running in Transparent, the XBee3 just didn’t register an internet connection. They j
I’m sorry. No, I don’t think it’s an issue with the battery or the SIM.When you were running in Transparent, the XBee3 just didn’t register an internet connection. They just do that sometimes – refuse to acknowledge they are registered on the network. Sometimes switching from transparent to bypass and back to transparent “magically” fixes the connection issue on the transparent side. Do you know what version of the Digi firmware you have? The newest is a little better than older, but it’s still not perfect.
When you were running in Bypass, you were hitting the same issues that you hit with the UBee, except this time it connected, opened the channel, got data, and then somehow didn’t realize there was a connection and re-opened it several times.
-
2020-01-20 at 6:23 PM #13698I finally actually did some testing with my own board and managed to get most of the same errors you found in Bypass mode and pushed some corrections to TinyGSM. It’s all about the issues withI finally actually did some testing with my own board and managed to get most of the same errors you found in Bypass mode and pushed some corrections to TinyGSM. It’s all about the issues with synchronous and asynchronous opening. I figured out how to make it crash every time and (hopefully) how to make that not happen. And the good news is that since both the XBee3 in Bypass and the Sodaq UBee are using the same code in TinyGSM, it should help both of them.
As for the issues with not connecting in transparent.. I’ve got nothing. Try some prayers to Isidore of Seville, patron of the internet. Sometimes switching the same board back and forth between transparent and bypass mode a couple of times works. More often I’ve been able to “fix” a board that wouldn’t connect in transparent by using bypass, but today I had three different boards that I couldn’t get to connect in bypass and suddenly happily connected in transparent. I have no idea why it happens.
-
2020-01-21 at 11:55 AM #13700The voltage levels logged during these tests were from 4.67 to 4.776V. The mayfly was powered via USB and LiPo.
I don’t know the Digi firmware version number on the XBee, but it was installed J
The voltage levels logged during these tests were from 4.67 to 4.776V. The mayfly was powered via USB and LiPo.I don’t know the Digi firmware version number on the XBee, but it was installed July 25, 2019. Additional background: What we’re attempting to do is to take this station and update it so that it’s more lenient on battery voltage threshold for data transmission, and so that it will flexibly choose the best cell carrier based on signal strength. To do this update, we’ve programmed a new Mayfly with the latest ModularSensors (don’t know whether the existing Mayfly is using that lib or not), and have moved the XBee, LTEBee, SIM, LiPo, and SD card from the existing to the new Mayfly. The existing Mayfly has been successfully communicating (when its battery and cell signal strength are sufficient).
You mentioned us hitting issues with the UBee, but we haven’t been using one here.
It sounds like our next step should be to apply your latest TinyGSM fixes and re-test, switching between Bypass and Transparent to try to beat the XBee into submission.
Question: How many sleep/wake cycles would you suggest I let the Mayfly run before I abort, make a code change (e.g. the Bypass/Transparent switch), and re-test? While debugging, I’ve set the logging interval to 3 minutes.
Thanks,
Matt
Trout Unlimited -
2020-01-21 at 2:47 PM #13705Doh! I’m sorry, I was confusing you with Dan who was troubleshooting issues with the UBee last week. They’re even worse than the LTE XBee3’s.
The power isn’t your
Doh! I’m sorry, I was confusing you with Dan who was troubleshooting issues with the UBee last week. They’re even worse than the LTE XBee3’s.The power isn’t your problem testing in the lab. That seems a hair low for being connected to USB, but not concerning. On the last bypass output that you posted from over the weekend, you did connect and even got the time from NIST, so your SIM and XBee and Mayfly are all OK. It’s just making them play nice.
Yes, I think your next step is to update TinyGSM and re-test. You can try switching between the modes to see if it helps the connection. Don’t bother waiting more than 1 cycle and you can drop your interval down to as low as 1 minute while your watching. If NIST connection doesn’t work in any of the 5 attempts, you can almost give up right there and post the output.
If the XBee3 refuses to submit, try switching to the “bleeding edge” version of ModularSensors on the ModemLast branch. To use that run
pio lib uninstall EnviroDIY_ModularSensorsorpio lib -g uninstall EnviroDIY_ModularSensors(case matters). Then after the uninstall, modify your platformio.ini and removeEnviroDIY_ModularSensorsfrom the lib_deps section and addhttps://github.com/EnviroDIY/ModularSensors.git#modemLastin its place. The modem last branch (as you might guess) restructures the way the internet connection happens. Instead of starting the modem first and waiting for it to connect as if it’s a sensor detecting the signal strength, the library keeps the modem off until all the other sensors are complete. Once the other sensor data is saved, the modem comes on, sends data to the world, and makes note of the signal strength just before going back to sleep. On the next sensor interval, the prior modem strength will be recorded. On the surface, the changes to your code should be pretty trivial, but the library will be doing different things underneath. -
2020-01-21 at 4:19 PM #13706This is all very helpful – thank you!
I just updated TinyGSM via pio:
12From https://github.com/EnviroDIY/TinyGSM30f09b2..5efcc5a master -> origThis is all very helpful – thank you!I just updated TinyGSM via pio:
123From https://github.com/EnviroDIY/TinyGSM30f09b2..5efcc5a master -> origin/masterUpdating 30f09b2..5efcc5aand re-tested (still in Transparent). No joy…and then Skype disconnected me from Jake’s computer, where the Mayfly is… not the ideal way to troubleshoot embedded systems…
So once he gets back to his office and I can reconnect, I’ll continue to troubleshoot as you suggested and report back.
Matt
Attachments:
-
2020-01-21 at 4:21 PM #13707
LOL!
-
2020-01-23 at 5:08 PM #13718@shicks came over to my office yesterday with a pile of LTE XBee3’s that will no longer connect to the internet. We’re going to buy some alternative types of modems to see if any of them@shicks came over to my office yesterday with a pile of LTE XBee3’s that will no longer connect to the internet. We’re going to buy some alternative types of modems to see if any of them are any better.
-
2020-01-23 at 11:18 PM #13720Hi Sara,
I want to update you on the results of our latest testing. I noticed that there was a new release of EnviroDIY/TinyGSM, so I installed that and re-tested. Transparent mode still failed (see
Hi Sara,I want to update you on the results of our latest testing. I noticed that there was a new release of EnviroDIY/TinyGSM, so I installed that and re-tested. Transparent mode still failed (see transparent.txt), but Bypass works. I tried flipping back and forth several times, and Transparent never worked; Bypass always did. Next, I decided to uninstall ModularSensors and install the modemLast branch. However, I got a permissions error during build as PlatformIO was trying to install dependencies (see permissions.txt). I tried creating a brand new project in a newly created folder on the C drive and repeated the test, with the same permissions error. I re-tried the build, and then got build errors in SdFat_ID322\FatLib (see sdfat.txt). I’ve been able to return to the main branch of ModularSensors and rebuild, and the modem works (again, only in Bypass). What would you suggest? Should we go ahead and deploy this Mayfly with the modem in Bypass mode?
There is one error I get when running in Bypass:
123AT+CGACT=0ERROR(See Bypass.txt for context.) Is this expected?
What a bummer to hear about all the nonfunctional XBee3’s. We’ll be very interested to hear of new developments on that front, and happy to assist in any way we can.
Best,
MattAttachments:
-
2020-01-24 at 11:08 AM #13725I’m curious, were those LTE XBee3’s deployed in the field, worked properly then at some point stopped connecting to the internet? Or did you receive them and realize they wouldn’t coI’m curious, were those LTE XBee3’s deployed in the field, worked properly then at some point stopped connecting to the internet? Or did you receive them and realize they wouldn’t connect upon initial testing? Just wanting to understand if this is something I need to watch out for with our LTE stations we’ve already deployed.
Thanks
-
2020-01-24 at 11:30 AM #13726@lemon – most of them were deployed in the field working and stopped for no apparent reason. I didn’t realize we’d had so many failures until Shannon dumped them on my desk the oth@lemon – most of them were deployed in the field working and stopped for no apparent reason. I didn’t realize we’d had so many failures until Shannon dumped them on my desk the other day.
@mbarney – we’ve gone back and forth over and over about the transparent-vs-bypass issue. My experience has generally been that the modem connects more quickly and more consistently and is more frequently successful in sending data when in bypass mode. Despite that, there is some bug that I have not been able to track down and only seems to happen with an XBee3 in bypass that causes the whole system to occasionally completely lock up until the watchdog finally forces a reset. So we’ve been deploying with transparent. The watchdog reset does work meaning you should only lose 15-20 minutes of data when the board freezes, so if you’re ok with that it should be fine to deploy in bypass mode.As for the logs, I’m not sure about that one error. It’s still working, so it’s probably not a big deal, but it should be fixed. The permissions error in installing or compiling sometimes happens when somehow the computer thinks a file is stuck open. Whenever you install a library or ‘build’ anything the PlatformIO and the compiler create and modify a whole bunch of files in the hidden .pio directory. If it thinks one of those files is open by another process it can’t do its thing and give that error. Usually completely closing all open copies of VSCode or Atom will ‘free’ whatever file got locked and you can re-open and it will be fine. Sometimes you have to reboot.
I’m sorry about the issue with the “modemLast” branch. Bleeding edge .. I changed something and was sloppy and didn’t make it backwards compatible. I’ll push a fix.
-
2020-01-24 at 11:43 AM #13727
Fix pushed.
-
2020-01-24 at 12:45 PM #13728Hi Sara, Since we got Bypass working on this modem, and @jlemontu-org (whose display name doesn’t match his username :)) has a window of time today to redeploy it in the field, we opted to justHi Sara, Since we got Bypass working on this modem, and @jlemontu-org (whose display name doesn’t match his username :)) has a window of time today to redeploy it in the field, we opted to just roll with it, so I haven’t taken the time to try #modemLast again, but thank you for the fix (I understand about the bleeding edge!)
Thanks for the tips on file permissions issue – makes sense.
Jake and I are going to take a pause on deploying new XBee3’s, pending any new findings from y’all about the ongoing failures. Please let me know if there is anything I can do to assist.
Best,
Matt
-
-
AuthorPosts
- You must be logged in to reply to this topic.
