Welcome to EnviroDIY, a community for do-it-yourself environmental science and monitoring. EnviroDIY is part of Home › Forums › Mayfly Data Logger › Stability Testing ~ how to do it? › Reply To: Stability Testing ~ how to do it?
In the early part of this year 2021, I did some regression testing using an Insitu LT500/SDI12 with a modbus board and telecom LTE, and I ran into a hard to find issue on the Mayfly 0.5b – the Uart Rx line is unterminated..
When extending features, its not unusual that problems are uncovered in new ways of exercising the software.
As part of a terminal based set the date on a simple Mayfly, I introduced a command line interface.
The objective was to be able to set the Date and Time on a Mayfly when its been installed or about to be installed. Typically a connected Mayfly gets the date when it polls NIST or by special date setting program.
A command line can be very useful in other ways as I brought out here
https://www.envirodiy.org/topic/how-to-dump-contents-of-file-on-sd-card-to-serial/#post-15830
For the initial command line interface I used String.
Now, it turns out that String has some down sides, and its valuable to understand these downsides – so as not to trip over them like I did.
https://cpp4arduino.com/2018/11/06/what-is-heap-fragmentation.html
https://cpp4arduino.com/2018/11/21/eight-tips-to-use-the-string-class-efficiently.html
So I created a potential issue in the software, that for incoming characters for the UART RX, I added them to a String. The hardware unfortunately can generate a lot of random characters, and this can cause the String to use up a lot of memory.
In the process of debugging this I wrote a utility to dump the ram and showing how much stack and heap used I called it dumpFreeRam() . This gives a periodic visual of the ram allocation, and what might be using it.
The critical issue is monitoring during integration testing how much free ram is available, and what might be using it.
I did find one bug this way when a lot of ram got used up with the above usage of String. I’ve changed it to a more traditional fast circular buffer.
This utility at the end of each invocation calculates a summary “Free ram never allocated”. However this by itself doesn’t indicate when the ram actually got allocated, so the debug listing needs to be evaluated for what event caused any ram to be used.
The example below shows that a program used up to 273 bytes after running for a couple of hours. 6161-591=273bytes. Fortunately longer testing doesn’t see more ram leakage.
[2021-08-01 16:22:14.881] Free ram never allocated between (bytes dec) 6144 and 6161
and after two hours
[2021-08-01 18:20:15.013] Free ram never allocated between (bytes dec) 5888 and 5912
The output on the terminal using Teraterm looks like this (using a Nanolevel/RS485)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
[2021-08-01 16:22:14.710] dumpFreeRam from heap(low)=0x26C8to stack(high)=0x40C3) size(dec):6651 [2021-08-01 16:22:14.710] 0x26C0 2D30312E 63737600 40000000 6C6C6572 ;-01.csv.@...ller [2021-08-01 16:22:14.710] 0x26D0 4E616E6F 6C657665 6C206174 206D6F64 ;Nanolevel at mod [2021-08-01 16:22:14.710] 0x26E0 6275735F 30783031 001F0000 006C6C65 ;bus_0x01.....lle [2021-08-01 16:22:14.710] 0x26F0 724E616E 6F6C6576 656C2061 74206D6F ;rNanolevel at mo [2021-08-01 16:22:14.710] 0x2700 64627573 5F307830 31001F00 00006C6C ;dbus_0x01.....ll [2021-08-01 16:22:14.710] 0x2710 65724E61 6E6F6C65 76656C20 6174206D ;erNanolevel at m [2021-08-01 16:22:14.710] 0x2720 6F646275 735F3078 3031006D 6F646275 ;odbus_0x01.modbu [2021-08-01 16:22:14.710] 0x2730 735F3078 30310061 79666C79 00000000 ;s_0x01.ayfly.... [2021-08-01 16:22:14.710] UnUsed 1k 0x2740............................................................... [2021-08-01 16:22:14.710] UnUsed 1k 0x2B40............................................................... [2021-08-01 16:22:14.710] UnUsed 1k 0x2F40............................................................... [2021-08-01 16:22:14.710] UnUsed 1k 0x3340............................................................... [2021-08-01 16:22:14.710] UnUsed 1k 0x3740............................................................... [2021-08-01 16:22:14.725] UnUsed 1k 0x3B40............................................................... [2021-08-01 16:22:14.725] 0x3F40 00000000 00000000 00000000 00744800 ;.............tH. [2021-08-01 16:22:14.725] 0x3F50 C11A81E1 013F3000 04000001 61CB0069 ;.....?0.....a..i [2021-08-01 16:22:14.725] 0x3F60 EA3F9574 483F993F 993F9869 88921D88 ;.?.tH?.?.?.i.... [2021-08-01 16:22:14.725] 0x3F70 921D89E2 1DD81202 00250089 E21DD80C ;.........%...... [2021-08-01 16:22:14.725] 0x3F80 0088921D D80074FB 1DD875B1 1BAB0077 ;......t...u....w [2021-08-01 16:22:14.725] 0x3F90 271DB600 003F0000 0400845B 3FA90000 ;'....?.....[?... [2021-08-01 16:22:14.725] 0x3FA0 1D3FB21D F3030001 BF27C40F DF000000 ;.?.......'...... [2021-08-01 16:22:14.725] 0x3FB0 0146A000 00B5D401 0200002A 073FCA40 ;.F.........*.?.@ [2021-08-01 16:22:14.725] 0x3FC0 2B010140 0E402A1D F385E200 85728571 ;+..@.@*......r.q [2021-08-01 16:22:14.725] 0x3FD0 74483FFD 3FFC0000 00027448 01143FF7 ;tH?.?.....tH..?. [2021-08-01 16:22:14.725] 0x3FE0 F3007448 40180001 3838FFFD 0000FFFC ;..tH@...88...... [2021-08-01 16:22:14.725] 0x3FF0 40500006 4014046E 14FFB500 0088921D ;@P..@..n........ [2021-08-01 16:22:14.725] 0x4000 89E21DD8 12000025 0089E21D D80C0088 ;.......%........ [2021-08-01 16:22:14.725] 0x4010 921DD800 74FB1DD8 75B11BAB 0077271D ;....t...u....w'. [2021-08-01 16:22:14.725] 0x4020 B600A040 000000A0 845B4037 00051D40 ;...@.....[@7...@ [2021-08-01 16:22:14.725] 0x4030 40888892 1D8940C3 00014056 000F1429 ;@.....@...@V...) [2021-08-01 16:22:14.725] 0x4040 1E4E40C3 00016E74 4869CB1A 813D3C74 ;.N@...ntHi...=<t [2021-08-01 16:22:14.881] 0x4050 4802F302 000135CB 00744840 8F008F40 ;H.....5..tH@...@ [2021-08-01 16:22:14.881] 0x4060 8B694A40 6E408D1A 80180000 0071E348 ;.iJ@n@.......q.H [2021-08-01 16:22:14.881] 0x4070 00C11A81 744869A0 1A817448 744800C1 ;....tHi...tHtH.. [2021-08-01 16:22:14.881] 0x4080 001B0074 481ABB1A 814840A4 00003000 ;...tH....H@...0. [2021-08-01 16:22:14.881] 0x4090 1A9B40C3 40900500 404EAB05 2E2E402E ;..@.@...@N....@. [2021-08-01 16:22:14.881] 0x40A0 402E2E2E 404E2E2E 2E2E402E 00201440 ;@...@N....@.. .@ [2021-08-01 16:22:14.881] 0x40B0 D7701A40 B64C566E 260B000B 008E261E ;.p.@.LVn&.....&. [2021-08-01 16:22:14.881] 0x40C0 001E0011 18060006 0040D000 1B2C0000 ;.........@...,.. [2021-08-01 16:22:14.881] Free ram never allocated between (bytes dec) 6144 and 6161 |
after running for two hours
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
[2021-08-01 18:20:14.715] dumpFreeRam from heap(low)=0x2721to stack(high)=0x40B9) size(dec):6552 [2021-08-01 18:20:14.715] 0x2720 31150000 0032312D 30382D30 31203137 ;1....21-08-01 17 [2021-08-01 18:20:14.715] 0x2730 3A32303A 30302C00 2D30312E 63737600 ;:20:00,.-01.csv. [2021-08-01 18:20:14.715] 0x2740 1D000000 5F52435F 74657374 3034625F ;...._RC_test04b_ [2021-08-01 18:20:14.715] 0x2750 32303231 2D30382D 30312E63 73760078 ;2021-08-01.csv.x [2021-08-01 18:20:14.715] 0x2760 3031001F 0000006C 6C65724E 616E6F6C ;01.....llerNanol [2021-08-01 18:20:14.715] 0x2770 6576656C 20617420 6D6F6462 75735F30 ;evel at modbus_0 [2021-08-01 18:20:14.715] 0x2780 78303100 00006C6C 65724E61 6E6F6C65 ;x01...llerNanole [2021-08-01 18:20:14.715] 0x2790 76656C20 6174206D 6F646275 735F3078 ;vel at modbus_0x [2021-08-01 18:20:14.715] 0x27A0 30310061 79666C79 003A3030 00003330 ;01.ayfly.:00..30 [2021-08-01 18:20:14.715] 0x27B0 00050000 00323600 05000000 33300005 ;.....26.....30.. [2021-08-01 18:20:14.715] 0x27C0 00000032 36000500 00003236 00000000 ;...26.....26.... [2021-08-01 18:20:14.715] UnUsed 1k 0x27D0............................................................... [2021-08-01 18:20:14.715] UnUsed 1k 0x2BD0............................................................... [2021-08-01 18:20:14.715] UnUsed 1k 0x2FD0............................................................... [2021-08-01 18:20:14.731] UnUsed 1k 0x33D0............................................................... [2021-08-01 18:20:14.731] UnUsed 1k 0x37D0............................................................... [2021-08-01 18:20:14.731] UnUsed 1k 0x3BD0............................................... [2021-08-01 18:20:14.731] 0x3ED0 00000000 00000000 00000000 00005003 ;..............P. [2021-08-01 18:20:14.731] 0x3EE0 000C0040 0089201D 8A701DE3 12010025 ;...@.. ..p.....% [2021-08-01 18:20:14.731] 0x3EF0 008A701D E30C0089 201DE300 75021DE3 ;..p..... ...u... [2021-08-01 18:20:14.731] 0x3F00 75B81BB6 00772E1D C100203F 00000220 ;u....w.... ?... [2021-08-01 18:20:14.731] 0x3F10 84E93F1F 00011DC1 02000000 022022DC ;..?.......... ". [2021-08-01 18:20:14.731] 0x3F20 E53F283F E4898920 1D8A701D E3120000 ;.?(?... ..p..... [2021-08-01 18:20:14.731] 0x3F30 89201D00 251D0000 A189201D E3017502 ;. ..%..... ...u. [2021-08-01 18:20:14.731] 0x3F40 1DE324CF 1BB60104 1B89201D 8A701DE3 ;..$....... ..p.. [2021-08-01 18:20:14.731] 0x3F50 12017572 1D8A701D E30C0089 201DE300 ;..ur..p..... ... [2021-08-01 18:20:14.731] 0x3F60 75021DE3 7589201D 8A701DE3 12020025 ;u...u. ..p.....% [2021-08-01 18:20:14.731] 0x3F70 008A701D E30C0089 201DE300 75021DE3 ;..p..... ...u... [2021-08-01 18:20:14.731] 0x3F80 75B81BB6 00772E1D C100003F 00000400 ;u....w.....?.... [2021-08-01 18:20:14.731] 0x3F90 84E93F9F 00001D3F A81DFE03 0001BF28 ;..?....?.......( [2021-08-01 18:20:14.750] 0x3FA0 FE0FE900 0000007B 480000B5 69010200 ;.......{H...i... [2021-08-01 18:20:14.750] 0x3FB0 002B963F C0402101 01400440 201DFE86 ;.+.?.@!..@.@ ... [2021-08-01 18:20:15.012] 0x3FC0 70744F3F F43FF300 03000300 02272301 ;ptO?.?.......'#. [2021-08-01 18:20:15.012] 0x3FD0 744F0174 4F401040 10000700 070006FF ;tO.tO@.@........ [2021-08-01 18:20:15.012] 0x3FE0 FB012200 00C36646 00064000 2500FF00 ;.."...fF..@.%... [2021-08-01 18:20:15.012] 0x3FF0 A1210189 201D8A70 1DE30025 00FF008A ;.!.. ..p...%.... [2021-08-01 18:20:15.012] 0x4000 701DE30C 0089201D E3007502 1DE375B8 ;p..... ...u...u. [2021-08-01 18:20:15.012] 0x4010 1BB60077 2E1DC100 A0400000 00A084E9 ;...w.....@...... [2021-08-01 18:20:15.012] 0x4020 402D0005 1D403689 89201D8A 40B90002 ;@-...@6.. ..@... [2021-08-01 18:20:15.012] 0x4030 404C00D8 0011154F 40B90002 6E744F00 ;@L.....O@...ntO. [2021-08-01 18:20:15.012] 0x4040 C11A8CA1 01403100 01000140 31000021 ;.....@1....@1..! [2021-08-01 18:20:15.012] 0x4050 4F408540 8540836A D0406440 831A8B40 ;O@.@.@.j.@d@...@ [2021-08-01 18:20:15.012] 0x4060 A2000071 EA1D8A70 744F744F 1A744F6B ;...q...ptOtO.tOk [2021-08-01 18:20:15.012] 0x4070 8C1A744F 00C10830 0002FF2E 0100744F ;..tO...0......tO [2021-08-01 18:20:15.012] 0x4080 00000000 1A004030 73B94000 40000000 ;......@0s.@.@... [2021-08-01 18:20:15.012] 0x4090 AE382E38 2E382E38 2E382E38 2E382E38 ;.8.8.8.8.8.8.8.8 [2021-08-01 18:20:15.012] 0x40A0 2E380074 0040D700 1E6FF740 D7204D40 ;.8.t.@...o.@. M@ [2021-08-01 18:20:15.012] 0x40B0 D7702140 B64DDCA7 26181706 0006001E ;.p!@.M..&....... [2021-08-01 18:20:15.013] Free ram never allocated between (bytes dec) 5888 and 5912 |
The utility is in my fork, search for dumpFreeRam() in following
https://github.com/neilh10/ModularSensors/commit/842eab880fdf6548d8b6d14951e4f0ad45727ecd
