Reply To: Stability Testing ~ how to do it?

In the early part of this year 2021, I did some regression testing using an Insitu LT500/SDI12 with a modbus board and telecom LTE, and I ran into a hard to find issue on the Mayfly 0.5b  – the

In the early part of this year 2021, I did some regression testing using an Insitu LT500/SDI12 with a modbus board and telecom LTE, and I ran into a hard to find issue on the Mayfly 0.5b  – the Uart Rx line is unterminated..

When extending features, its not unusual that problems are uncovered in new ways of exercising the software.

As part of a terminal based set the date on a simple Mayfly, I introduced a command line interface.

The objective was to be able to set the Date and Time on a Mayfly when its been installed or about to be installed. Typically a connected Mayfly gets the date when it polls NIST or by special date setting program.

A command line can be very useful in other ways as I brought out here

https://www.envirodiy.org/topic/how-to-dump-contents-of-file-on-sd-card-to-serial/#post-15830

For the initial command line interface I used String.

Now, it turns out that String has some down sides, and its valuable to understand these downsides – so as not to trip over them like I did.

https://cpp4arduino.com/2018/11/06/what-is-heap-fragmentation.html

https://cpp4arduino.com/2018/11/21/eight-tips-to-use-the-string-class-efficiently.html

 

So I created a potential issue in the software, that for incoming characters for the UART RX, I added them to a String. The hardware unfortunately can generate a lot of random characters, and this can cause the String to use up a lot of memory.

In the process of debugging this I wrote a utility to dump the ram and showing how much stack and heap used I called it dumpFreeRam() . This gives a periodic visual of the ram allocation, and what might be using it.

The critical issue is monitoring during integration testing how much free ram is available, and what might be using it.

I did find one bug this way when a lot of ram got used up with the above usage of String. I’ve changed it to a more traditional fast circular buffer.

This utility at the end of each invocation calculates a summary “Free ram never allocated”. However this by itself doesn’t indicate when the ram actually got allocated, so the debug listing needs to be evaluated for what event caused any ram to be used.

The example below shows that a program used up to 273 bytes after running for a couple of hours. 6161-591=273bytes. Fortunately  longer testing doesn’t see more ram leakage.

[2021-08-01 16:22:14.881] Free ram never allocated between (bytes dec) 6144 and 6161

and after two hours

[2021-08-01 18:20:15.013] Free ram never allocated between (bytes dec) 5888 and 5912

 

The output on the terminal using Teraterm looks like this (using a Nanolevel/RS485)

after running for two hours

 

The utility is in my fork, search for dumpFreeRam() in following

https://github.com/neilh10/ModularSensors/commit/842eab880fdf6548d8b6d14951e4f0ad45727ecd

Less...