collectgarbage("count")
Locked
|
On my LM3S6965 board, I get "not enough memory" errors when collectgarbage("count") returns a value that is way lower than the available memory, between 30 and 40k, with 64k on the chip. Shouldn't I get close to the board's limit of 64k? Or do I not understand the value returned by collectgarbage("count") correctly? I tested with one collectgarbage("collect") call per duty cycle of my program to hopefully keep the influence of the dynamics of the garbage collector small.
Thanks for any enlightening! Cheers -- Stefan _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
Hi -
There are a few possibilities here for where your extra memory may be disappearing. The "count" option for collectgarbage is only counting garbage-collectable objects allocated by Lua. So, it isn't counting every chunk of SRAM allocated or how much stack space is being used. Additionally, depending ont he size of additional objects that you're needing when it runs out of memory, it may be that there's just too much fragmentation and a large enough chunk can't be found at those times. You don't need to do a collect every cycle, you might want to look into enabling the emergency garbage collector: http://www.eluaproject.net/en_elua_egc.html I don't think the defines are in the platform_conf.h files for any of the platforms so you would need to enable it yourself. See the EGC variables (EGC_INITIAL_MODE, EGC_INITIAL_MEMLIMIT) on this page: http://www.eluaproject.net/en_building.html On Thu, Jun 10, 2010 at 2:32 PM, Stefan Brantschen <[hidden email]> wrote: > On my LM3S6965 board, I get "not enough memory" errors when collectgarbage("count") returns a value that is way lower than the available memory, between 30 and 40k, with 64k on the chip. Shouldn't I get close to the board's limit of 64k? Or do I not understand the value returned by collectgarbage("count") correctly? I tested with one collectgarbage("collect") call per duty cycle of my program to hopefully keep the influence of the dynamics of the garbage collector small. > > Thanks for any enlightening! > > Cheers > -- Stefan > > _______________________________________________ > eLua-dev mailing list > [hidden email] > https://lists.berlios.de/mailman/listinfo/elua-dev > -- James Snyder Biomedical Engineering Northwestern University [hidden email] PGP: http://fanplastic.org/key.txt Phone: (847) 448-0386 _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Stefan Brantschen-3
Hello,
On Thu, Jun 10, 2010 at 16:32, Stefan Brantschen <[hidden email]> wrote: On my LM3S6965 board, I get "not enough memory" errors when collectgarbage("count") returns a value that is way lower than the available memory, between 30 and 40k, with 64k on the chip. Do you mean, you get a "not enough memory" error when simply executing a collectgarbage("count") instruction ? Or your program gets "not enough memory" errors when the returned "count" value is under 30~40K ? Shouldn't I get close to the board's limit of 64k? Not really, because the Lua (and possibly your program too) eats part of the available RAM. Or do I not understand the value returned by collectgarbage("count") correctly? collectgarbage("count") returns the ammount of RAM currently allocated/in use by Lua in KB ( * 1024 = in Bytes). Thus, 64 - collectgarbage("count") should give you the ammount of free RAM you still have in your app. I tested with one collectgarbage("collect") call per duty cycle of my program to hopefully keep the influence of the dynamics of the garbage collector small. The Lua GC is actually much less frightening to our embedded apps than what it may seem at first :) If your app is not _that_ critical, just let it do it's job and it is usually enough. You can tune a couple of variables to tune the GC to your app but you'll hardly need it.
Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
 
|
Stefan Brantschen-3 |
Re: collectgarbage("count")
|
|
In reply to this post by jbsnyder
Hi
On 10 Jun 2010, at 22:15, James Snyder wrote: > The "count" option for collectgarbage is only counting > garbage-collectable objects allocated by Lua. So, it isn't counting > every chunk of SRAM allocated or how much stack space is being used. I understand. Wasn't aware of this. Thanks. Still digging. Cheers -- Stefan _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
Stefan Brantschen-3 |
Re: collectgarbage("count")
|
|
In reply to this post by Dado Sutter
Hi
On 10 Jun 2010, at 22:19, Dado Sutter wrote: >> Or do I not understand the value returned by collectgarbage("count") correctly? > > collectgarbage("count") returns the ammount of RAM currently allocated/in use by Lua in KB ( * 1024 = in Bytes). Thus, 64 - collectgarbage("count") should give you the ammount of free RAM you still have in your app. Isn't the free RAM 64k - collectgarbage("count") - memory used by non-collectable elements (eg. stack), according to James' post? Cheers -- Stefan _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
On Fri, Jun 11, 2010 at 03:28, Stefan Brantschen <[hidden email]> wrote: ......... Yes, right. It is just that the we tend not to consider the space taken by the non-collectable elements because they're usually small. Thanks for pointing it James. But I didn't understant very well your problem. Are you getting the "not enough memory" errors during/right_after the collectgarbage( "count" ) instruction ? By the simply fact of executing it ? This sounds really strange.
Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
Stefan Brantschen-3 |
Re: collectgarbage("count")
|
|
Hi
On 11 Jun 2010, at 14:22, Dado Sutter wrote: > But I didn't understant very well your problem. Are you getting the "not enough memory" errors during/right_after the collectgarbage( "count" ) instruction ? By the simply fact of executing it ? This sounds really strange. OK, here's my test set-up. Sorry if am a bit lengthy, but you asked... :-) I try to get my head around eLua, so I thought I'd create some simple re-usable framework for handling several threads of control ("agents") on the LM3S board, in order to learn how far I get with 64k of memory. So I have many agent instances going after their job, executing independently of each other. This way I handle events, one agent can signal the other for synchronization, and I plan non-blocking timing of these "threads". You get the idea. In order to push the limits to see how far I can go (and to learn where to streamline my code to find a practical balance between abstraction in the framework and performance/execution size), I increased the number of agent instances, each of which is just stupidly counting up in order to have something to do for this test, and each 10th agent displays the counter value on the display on the board so I can follow what's going on. Now, my scheduler periodically executes and displays collectgarbage("count"). As I increase the number of agent instances (implemented as closures over the same function for testing), the value of collectgarbage("count") goes up as well. And here's finally my observation and my answer to your question: with a sufficient number of agents, I get the "not enough memory" message, and at this point the value of collectgarbage("count") is between 30-40k. Of course it makes a difference if I call collectgarbage("collect") at each duty cycle, which allows me to run more agents at the cost of performance, but what happens is the same, just at a higher number of agents. I had expected to see the "not enough memory" error at or close to 64k, naively assuming that collectgarbage("count") gives the overall memory consumption. I know better now, so I guess I can infer that my test program with its non-collectable items plus stack is 24k if I get the above error message at collectgarbage("count") == 40k. Correct? Thanks for your help. Cheers -- Stefan PS: BTW, I can run about 100 counting agents in parallel with tight garbage collection. _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
On Fri, Jun 11, 2010 at 10:26, Stefan Brantschen <[hidden email]> wrote: ........................... Very impressive for such a tiny board, simple code and on bare metal ! :) Tks for your presentation too. I'll come back to it asap. Meanwhile, does the performance change too much if you let the GC do it's job automatically ? (not forcing collect ...) Cheers Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Stefan Brantschen-3
On Fri, Jun 11, 2010 at 8:26 AM, Stefan Brantschen
<[hidden email]> wrote: > I had expected to see the "not enough memory" error at or close to 64k, naively assuming that collectgarbage("count") gives the overall memory consumption. I know better now, so I guess I can infer that my test program with its non-collectable items plus stack is 24k if I get the above error message at collectgarbage("count") == 40k. Correct? I believe the main stack (whose size is settable in stacks.h) is 2kB on that platform. I'm not sure if the remaining 22k is really non-collectable items or if the problem is more related to fragmentation. We would really need to instrument malloc and see what the memory space looks like to get a better idea here, but we don't currently have something like this implemented (it's on the wishlist somewhere). I believe that Bogdan might have had a tool at some point that would collect some data on Lua's collectable allocations, but it's not in SVN and I don't think I have the sources for it. -- James Snyder Biomedical Engineering Northwestern University [hidden email] PGP: http://fanplastic.org/key.txt Phone: (847) 448-0386 _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
Stefan Brantschen-3 |
Re: collectgarbage("count")
|
|
In reply to this post by Dado Sutter
Hi
On 11 Jun 2010, at 15:55, Dado Sutter wrote: > PS: BTW, I can run about 100 counting agents in parallel with tight garbage collection. > > Very impressive for such a tiny board, simple code and on bare metal ! :) Bear in mind that they don't do anything useful though :-) > Meanwhile, does the performance change too much if you let the GC do it's job automatically ? (not forcing collect ...) Running 50 agents doing their counting and displaying, with t-cycle = time to run all agents once c-max = max value of collectgarbage("count") 1) GC let alone: t-cycle = 27.7 ms, c-max = 32.2k 2) GC enforced each scheduler cycle: t-cycle = 30.8 ms, c-max = 23.3k 3) GC enforced each 10th cycle: t-cycle = 28.6 ms, c-max = 24.6k In this set-up, variant 3) looks like a decent compromise. Times measures with the board's timer and the tmr module. Cheers -- Stefan _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
Hi,
It's true that collectgarbage('count') is only a part of the story. A lot of stuff happens in the background, such as buffered I/O (fopen and friends), eLua's own buffers, malloc overhead and so on. I still don't have a good way to see a clear "memory map" of the system when an "out of memory" error is returned, and I'm not even sure if this would help. In the end, the truth is that 64k is still quite a limitation for eLua. I'm afraid that the only thing left to do to decrease memory consumption is to have a "run from memory" patch (run a Lua script directly from Flash, thus saving RAM; this obviously will work only for Flash memory mapped on the CPU's bus, not serial memory or anything similar, but it would still be an improvement). Well, this and rewriting the VM, of course :) Best, Bogdan On Fri, Jun 11, 2010 at 8:08 PM, Stefan Brantschen <[hidden email]> wrote: Hi _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
On Mon, Jun 14, 2010 at 5:57 AM, Bogdan Marinescu
<[hidden email]> wrote: > Hi, > > It's true that collectgarbage('count') is only a part of the story. A lot of > stuff happens in the background, such as buffered I/O (fopen and friends), > eLua's own buffers, malloc overhead and so on. I still don't have a good way > to see a clear "memory map" of the system when an "out of memory" error is > returned, and I'm not even sure if this would help. In the end, the truth is > that 64k is still quite a limitation for eLua. I'm afraid that the only > thing left to do to decrease memory consumption is to have a "run from > memory" patch (run a Lua script directly from Flash, thus saving RAM; this > obviously will work only for Flash memory mapped on the CPU's bus, not > serial memory or anything similar, but it would still be an improvement). > Well, this and rewriting the VM, of course :) I haven't investigated, but one other thing that has occurred to me would be making things like table creation a little more tunable so that pre-allocation doesn't grow automatically up to the next power of two. Either that, or perhaps having GC compact tables when needed (this might be done by Jakabosky's patch, though I haven't read through the memory optimization features that are part of the main patch (and also broken out): http://lua-users.org/wiki/EmergencyGarbageCollector If the EGC code actually does this already, then I don't think I'd worry too much about the creation side. > > Best, > Bogdan > > On Fri, Jun 11, 2010 at 8:08 PM, Stefan Brantschen > <[hidden email]> wrote: >> >> Hi >> >> On 11 Jun 2010, at 15:55, Dado Sutter wrote: >> >> > PS: BTW, I can run about 100 counting agents in parallel with tight >> > garbage collection. >> > >> > Very impressive for such a tiny board, simple code and on bare metal ! >> > :) >> >> Bear in mind that they don't do anything useful though :-) >> >> > Meanwhile, does the performance change too much if you let the GC do >> > it's job automatically ? (not forcing collect ...) >> >> Running 50 agents doing their counting and displaying, with >> >> t-cycle = time to run all agents once >> c-max = max value of collectgarbage("count") >> >> 1) GC let alone: t-cycle = 27.7 ms, c-max = 32.2k >> >> 2) GC enforced each scheduler cycle: t-cycle = 30.8 ms, c-max = 23.3k >> >> 3) GC enforced each 10th cycle: t-cycle = 28.6 ms, c-max = 24.6k >> >> In this set-up, variant 3) looks like a decent compromise. >> >> Times measures with the board's timer and the tmr module. >> >> Cheers >> -- Stefan >> >> >> _______________________________________________ >> eLua-dev mailing list >> [hidden email] >> https://lists.berlios.de/mailman/listinfo/elua-dev > > > _______________________________________________ > eLua-dev mailing list > [hidden email] > https://lists.berlios.de/mailman/listinfo/elua-dev > > -- James Snyder Biomedical Engineering Northwestern University [hidden email] PGP: http://fanplastic.org/key.txt Phone: (847) 448-0386 _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
«
Return to eLua Development
|
1 view|%1 views
| Free forum by Nabble | Edit this page |