eLua!

[BogdanM]

Hi,

You'll find my answers below.

Best,
Bogdan

> Just a quick email to ask if you could look over the RAM numbers we're
> getting and see if they are what you'd expect, and whether you had any
> thoughts on other variables/ways to get the RAM footprint down. Also,
> any thoughts on how much executing the script from flash would save (I
> assume it would just be the size of the plain text?).
Yes, it would just be the size of the plain text.

> ---
>
> Standard lua, w/ elua libs (pio, uart, pd, timer, term, shell, xmodem,
> terminal, romfs):
>
> eLua# lua -e 'print(collectgarbage("count").."\n")'
> Press CTRL+Z to exit Lua
> 23.0869140625
Hmmm, 23k only for Lua + platfrom libs. Interesting, I didn't test
this so far. Seems quite large.

> Standard lua, no elua libs (but still with shell)
The shell itself uses very little RAM. The only part of the shell that
uses more RAM (currently 4k) is the "recv" function, but it only
allocates RAM on demand (that is, when "recv" is called) and frees it
before returning from "recv".

> eLua# lua -e 'print(collectgarbage("count").."\n")'
> Press CTRL+Z to exit Lua
> 16.80859375
So, all the auxiliary libraries take about 6k of RAM. That's not so
bad, but I think this could be improve (more on this later).

> Standard lua, no elua libs (but still with shell) + romfs
>
> eLua# lua -e 'print(collectgarbage("count").."\n")'
> Press CTRL+Z to exit Lua
> 16.80859375
>
> (so romfs makes no difference to initial RAM usage)
It shouldn't, as it doesn't use buffers, it reads its data directly
from the flash memory. When the buffered file systems will be in,
things will change, unfortunately. Whatch it though, when you open a
file from Lua, it calls "fopen", which in turn creates a buffer for
that file when you first access it (via fread/fwrite). This is how the
standard fopen/fread/fwrite... functions work. To overcome this, you
could find the implementation of the io.open function (somewhere in
liolib.c) and add a call to "setbuf( filepointer, NULL )" after the
"fopen" call, thus effectively removing any buffering on that file
(eLua already does this for stdout/stderr, but not for stdin, because
you need the buffer on that one).

> Integer-only lua, w/ elua libs (pio, uart, pd, timer, term, shell,
> xmodem, terminal, romfs):
>
> eLua# lua -e 'print(collectgarbage("count").."\n")'
> Press CTRL+Z to exit Lua
> 17
>
> Integer-only lua, no elua libs (but still with shell)
>
> eLua# lua -e 'print(collectgarbage("count").."\n")'
> Press CTRL+Z to exit Lua
> 12
>

So the difference between "bare bones" eLua (no libs, just the shell)
in floating point mode and integer only mode is about 4k of memory.
Now THIS is interesting, we need to figure out where this comes from
and work on this.
Now, finally, some ideas to improve RAM usage:

- the constants in some of our modules (like PIO.Px_y, term.KC_xxxx)
are now written explicitely, and thus consume RAM memory. We could use
methamethods (index) instead to "resolve" these constants. This will
of course add to the execution time, but I think it's a perfectly good
compromise.

- remove some of the modules that Lua preloads. The newly released
0.4.1 already removes the "os" and "package" modules, as they're not
used anyway, and makes "math" configurable just like the platform
modules.

- implement "emergency garbage collection"
(http://lua-users.org/wiki/EmergencyGarbageCollector). I did some
tests with this, and it has a very strong impact on the memory
consumption (see
http://lua-users.org/lists/lua-l/2008-05/threads.html#00023 for more
details). Unfortunately, it has a very strong impact on the execution
speed two (at least on its most aggressive working mode) but this
compromise is something one must live with in programming. This is a
"must have" for eLua 0.5.

- use a memory allocator with a smaller overhead. dlmalloc/TLSF/other
seggregated allocators use some sort of "page tables" (TLSF has even a
two-level page table) that consumes memory (on TLSF the overhead is as
large as 3k!). I have a very simple, very fragmentable, yet very low
overhead chained allocator that I wrote myself, and I intend to
include it in a future version (maybe 0.5, but I'm not sure at this
point). Then again, in my tests, this allocator didn't perform all
that well, but this is VERY dependent on the application.

- figure out what I mentioned earlier: why the difference between
"bare bones" Lua in floating point mode and integer only mode is that
large. Obviously it can't be because of the math library, I can't
imagine a math library with such a huge RAM consumption (and I should
know, I worked on two different math emulation libraries in my past).
Could it be just because we don't register the "math" module anymore?
It still seems that 4k is a LOT of memory gained from just removing
the "math" module. I don't know how to test this, unfortunately.

Thanks a lot for this e-mail, I was working on some memory statistics
myself, but never for a minute thought about running
collectgarbage("count"), as I completely forgot that collectgarbage
can also return the ammount of memory used by Lua :)
And, as a personal preference, could we continue this discussion on
the elua-dev list?

Best,
Bogdan

[Dado Sutter]

Hello eLuers :)
   Simon; thank you for the great analisys and suggestions.
   Bogdan; thank you for bringing this to the list.

Best
Dado




On Thu, Sep 11, 2008 at 4:12 AM, Bogdan Marinescu <
bogdan.marinescu at gmail.com> wrote:

> Hi,
>
> You'll find my answers below.
>
> Best,
> Bogdan
>
> > Just a quick email to ask if you could look over the RAM numbers we're
> > getting and see if they are what you'd expect, and whether you had any
> > thoughts on other variables/ways to get the RAM footprint down. Also,
> > any thoughts on how much executing the script from flash would save (I
> > assume it would just be the size of the plain text?).
> Yes, it would just be the size of the plain text.
>
> > ---
> >
> > Standard lua, w/ elua libs (pio, uart, pd, timer, term, shell, xmodem,
> > terminal, romfs):
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 23.0869140625
> Hmmm, 23k only for Lua + platfrom libs. Interesting, I didn't test
> this so far. Seems quite large.
>
> > Standard lua, no elua libs (but still with shell)
> The shell itself uses very little RAM. The only part of the shell that
> uses more RAM (currently 4k) is the "recv" function, but it only
> allocates RAM on demand (that is, when "recv" is called) and frees it
> before returning from "recv".
>
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 16.80859375
> So, all the auxiliary libraries take about 6k of RAM. That's not so
> bad, but I think this could be improve (more on this later).
>
> > Standard lua, no elua libs (but still with shell) + romfs
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 16.80859375
> >
> > (so romfs makes no difference to initial RAM usage)
> It shouldn't, as it doesn't use buffers, it reads its data directly
> from the flash memory. When the buffered file systems will be in,
> things will change, unfortunately. Whatch it though, when you open a
> file from Lua, it calls "fopen", which in turn creates a buffer for
> that file when you first access it (via fread/fwrite). This is how the
> standard fopen/fread/fwrite... functions work. To overcome this, you
> could find the implementation of the io.open function (somewhere in
> liolib.c) and add a call to "setbuf( filepointer, NULL )" after the
> "fopen" call, thus effectively removing any buffering on that file
> (eLua already does this for stdout/stderr, but not for stdin, because
> you need the buffer on that one).
>
> > Integer-only lua, w/ elua libs (pio, uart, pd, timer, term, shell,
> > xmodem, terminal, romfs):
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 17
> >
> > Integer-only lua, no elua libs (but still with shell)
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 12
> >
> So the difference between "bare bones" eLua (no libs, just the shell)
> in floating point mode and integer only mode is about 4k of memory.
> Now THIS is interesting, we need to figure out where this comes from
> and work on this.
> Now, finally, some ideas to improve RAM usage:
>
> - the constants in some of our modules (like PIO.Px_y, term.KC_xxxx)
> are now written explicitely, and thus consume RAM memory. We could use
> methamethods (index) instead to "resolve" these constants. This will
> of course add to the execution time, but I think it's a perfectly good
> compromise.
>
> - remove some of the modules that Lua preloads. The newly released
> 0.4.1 already removes the "os" and "package" modules, as they're not
> used anyway, and makes "math" configurable just like the platform
> modules.
>
> - implement "emergency garbage collection"
> (http://lua-users.org/wiki/EmergencyGarbageCollector). I did some
> tests with this, and it has a very strong impact on the memory
> consumption (see
> http://lua-users.org/lists/lua-l/2008-05/threads.html#00023 for more
> details). Unfortunately, it has a very strong impact on the execution
> speed two (at least on its most aggressive working mode) but this
> compromise is something one must live with in programming. This is a
> "must have" for eLua 0.5.
>
> - use a memory allocator with a smaller overhead. dlmalloc/TLSF/other
> seggregated allocators use some sort of "page tables" (TLSF has even a
> two-level page table) that consumes memory (on TLSF the overhead is as
> large as 3k!). I have a very simple, very fragmentable, yet very low
> overhead chained allocator that I wrote myself, and I intend to
> include it in a future version (maybe 0.5, but I'm not sure at this
> point). Then again, in my tests, this allocator didn't perform all
> that well, but this is VERY dependent on the application.
>
> - figure out what I mentioned earlier: why the difference between
> "bare bones" Lua in floating point mode and integer only mode is that
> large. Obviously it can't be because of the math library, I can't
> imagine a math library with such a huge RAM consumption (and I should
> know, I worked on two different math emulation libraries in my past).
> Could it be just because we don't register the "math" module anymore?
> It still seems that 4k is a LOT of memory gained from just removing
> the "math" module. I don't know how to test this, unfortunately.
>
> Thanks a lot for this e-mail, I was working on some memory statistics
> myself, but never for a minute thought about running
> collectgarbage("count"), as I completely forgot that collectgarbage
> can also return the ammount of memory used by Lua :)
> And, as a personal preference, could we continue this discussion on
> the elua-dev list?
>
> Best,
> Bogdan
> _______________________________________________
> Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20080911/6dae6f4b/attachment.html 

[Frédéric THOMAS-5]

FWIW, I can confirm these numbers, they are similar to what I get.

What we'd need now is an idea of the amount of RAM used by the rest of  
eLua (shell, etc).

> - use a memory allocator with a smaller overhead. dlmalloc/TLSF/other

The Luminary BSP includes bget  (in third_party) ? I have no  
experience with it.

Fred


On Sep 11, 2008, at 16:35 , Dado Sutter wrote:

> Hello eLuers :)
>    Simon; thank you for the great analisys and suggestions.
>    Bogdan; thank you for bringing this to the list.
>
> Best
> Dado
>
>
>
>
> On Thu, Sep 11, 2008 at 4:12 AM, Bogdan Marinescu <bogdan.marinescu at gmail.com
> > wrote:
> Hi,
>
> You'll find my answers below.
>
> Best,
> Bogdan
>
> > Just a quick email to ask if you could look over the RAM numbers  
> we're
> > getting and see if they are what you'd expect, and whether you had  
> any
> > thoughts on other variables/ways to get the RAM footprint down.  
> Also,
> > any thoughts on how much executing the script from flash would  
> save (I
> > assume it would just be the size of the plain text?).
> Yes, it would just be the size of the plain text.
>
> > ---
> >
> > Standard lua, w/ elua libs (pio, uart, pd, timer, term, shell,  
> xmodem,
> > terminal, romfs):
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 23.0869140625
> Hmmm, 23k only for Lua + platfrom libs. Interesting, I didn't test
> this so far. Seems quite large.
>
> > Standard lua, no elua libs (but still with shell)
> The shell itself uses very little RAM. The only part of the shell that
> uses more RAM (currently 4k) is the "recv" function, but it only
> allocates RAM on demand (that is, when "recv" is called) and frees it
> before returning from "recv".
>
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 16.80859375
> So, all the auxiliary libraries take about 6k of RAM. That's not so
> bad, but I think this could be improve (more on this later).
>
> > Standard lua, no elua libs (but still with shell) + romfs
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 16.80859375
> >
> > (so romfs makes no difference to initial RAM usage)
> It shouldn't, as it doesn't use buffers, it reads its data directly
> from the flash memory. When the buffered file systems will be in,
> things will change, unfortunately. Whatch it though, when you open a
> file from Lua, it calls "fopen", which in turn creates a buffer for
> that file when you first access it (via fread/fwrite). This is how the
> standard fopen/fread/fwrite... functions work. To overcome this, you
> could find the implementation of the io.open function (somewhere in
> liolib.c) and add a call to "setbuf( filepointer, NULL )" after the
> "fopen" call, thus effectively removing any buffering on that file
> (eLua already does this for stdout/stderr, but not for stdin, because
> you need the buffer on that one).
>
> > Integer-only lua, w/ elua libs (pio, uart, pd, timer, term, shell,
> > xmodem, terminal, romfs):
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 17
> >
> > Integer-only lua, no elua libs (but still with shell)
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 12
> >
> So the difference between "bare bones" eLua (no libs, just the shell)
> in floating point mode and integer only mode is about 4k of memory.
> Now THIS is interesting, we need to figure out where this comes from
> and work on this.
> Now, finally, some ideas to improve RAM usage:
>
> - the constants in some of our modules (like PIO.Px_y, term.KC_xxxx)
> are now written explicitely, and thus consume RAM memory. We could use
> methamethods (index) instead to "resolve" these constants. This will
> of course add to the execution time, but I think it's a perfectly good
> compromise.
>
> - remove some of the modules that Lua preloads. The newly released
> 0.4.1 already removes the "os" and "package" modules, as they're not
> used anyway, and makes "math" configurable just like the platform
> modules.
>
> - implement "emergency garbage collection"
> (http://lua-users.org/wiki/EmergencyGarbageCollector). I did some
> tests with this, and it has a very strong impact on the memory
> consumption (see
> http://lua-users.org/lists/lua-l/2008-05/threads.html#00023 for more
> details). Unfortunately, it has a very strong impact on the execution
> speed two (at least on its most aggressive working mode) but this
> compromise is something one must live with in programming. This is a
> "must have" for eLua 0.5.
>
> - use a memory allocator with a smaller overhead. dlmalloc/TLSF/other
> seggregated allocators use some sort of "page tables" (TLSF has even a
> two-level page table) that consumes memory (on TLSF the overhead is as
> large as 3k!). I have a very simple, very fragmentable, yet very low
> overhead chained allocator that I wrote myself, and I intend to
> include it in a future version (maybe 0.5, but I'm not sure at this
> point). Then again, in my tests, this allocator didn't perform all
> that well, but this is VERY dependent on the application.
>
> - figure out what I mentioned earlier: why the difference between
> "bare bones" Lua in floating point mode and integer only mode is that
> large. Obviously it can't be because of the math library, I can't
> imagine a math library with such a huge RAM consumption (and I should
> know, I worked on two different math emulation libraries in my past).
> Could it be just because we don't register the "math" module anymore?
> It still seems that 4k is a LOT of memory gained from just removing
> the "math" module. I don't know how to test this, unfortunately.
>
> Thanks a lot for this e-mail, I was working on some memory statistics
> myself, but never for a minute thought about running
> collectgarbage("count"), as I completely forgot that collectgarbage
> can also return the ammount of memory used by Lua :)
> And, as a personal preference, could we continue this discussion on
> the elua-dev list?
>
> Best,
> Bogdan
> _______________________________________________
> Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
> _______________________________________________
> Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20080911/4e357a9a/attachment-0001.html 

[BogdanM]

Re-sending a message to list, see below for original message.


---------- Forwarded message ----------
From: Bogdan Marinescu <bogdan.marinescu at gmail.com>
Date: Fri, Sep 12, 2008 at 1:46 PM
Subject: Re: [Elua-dev] eLua!
To: Fr?d?ric Thomas <berlios.de at thomascorner.com>


> What we'd need now is an idea of the amount of RAM used by the rest of eLua
> (shell, etc).
The shell takes very little memory, as explained. The platform
modules, on the other hand ... I'd really like to estimate their
memory consumption. I don't know how though, since when the module is
registered it's "translated" into a dictionary, and I have no idea how
to estimate the memory consumption of that ...

> - use a memory allocator with a smaller overhead. dlmalloc/TLSF/other
>
> The Luminary BSP includes bget  (in third_party) ? I have no experience with
> it.
I looked it at, it seems also a very basic allocator, with little
overhead, quite similar to mine. Will study it further.

Best,
Bogdan

[BogdanM]

Hi,

FYI, I tried the first suggestion below (handle pio.PIOx_y and term.KC_xxxx
via metamethods instead of adding them directly into the table) and the
memory footprint was reduced.

Before:

eLua# lua -e "print(collectgarbage'count')"
Press CTRL+Z to exit Lua
25.03515625

After:

eLua# lua -e "print(collectgarbage'count')"
Press CTRL+Z to exit Lua
21.572265625

About 3.5k of memory saved. Quite a save, given the fact that the newly
introduced uIP port to eLua uses a buffer of only 1024 bytes :)
The new code for the forementioned modules (pio and term) is in SVN.

Best,
Bogdan

On Thu, Sep 11, 2008 at 10:12 AM, Bogdan Marinescu <
bogdan.marinescu at gmail.com> wrote:

> Hi,
>
> You'll find my answers below.
>
> Best,
> Bogdan
>
> > Just a quick email to ask if you could look over the RAM numbers we're
> > getting and see if they are what you'd expect, and whether you had any
> > thoughts on other variables/ways to get the RAM footprint down. Also,
> > any thoughts on how much executing the script from flash would save (I
> > assume it would just be the size of the plain text?).
> Yes, it would just be the size of the plain text.
>
> > ---
> >
> > Standard lua, w/ elua libs (pio, uart, pd, timer, term, shell, xmodem,
> > terminal, romfs):
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 23.0869140625
> Hmmm, 23k only for Lua + platfrom libs. Interesting, I didn't test
> this so far. Seems quite large.
>
> > Standard lua, no elua libs (but still with shell)
> The shell itself uses very little RAM. The only part of the shell that
> uses more RAM (currently 4k) is the "recv" function, but it only
> allocates RAM on demand (that is, when "recv" is called) and frees it
> before returning from "recv".
>
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 16.80859375
> So, all the auxiliary libraries take about 6k of RAM. That's not so
> bad, but I think this could be improve (more on this later).
>
> > Standard lua, no elua libs (but still with shell) + romfs
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 16.80859375
> >
> > (so romfs makes no difference to initial RAM usage)
> It shouldn't, as it doesn't use buffers, it reads its data directly
> from the flash memory. When the buffered file systems will be in,
> things will change, unfortunately. Whatch it though, when you open a
> file from Lua, it calls "fopen", which in turn creates a buffer for
> that file when you first access it (via fread/fwrite). This is how the
> standard fopen/fread/fwrite... functions work. To overcome this, you
> could find the implementation of the io.open function (somewhere in
> liolib.c) and add a call to "setbuf( filepointer, NULL )" after the
> "fopen" call, thus effectively removing any buffering on that file
> (eLua already does this for stdout/stderr, but not for stdin, because
> you need the buffer on that one).
>
> > Integer-only lua, w/ elua libs (pio, uart, pd, timer, term, shell,
> > xmodem, terminal, romfs):
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 17
> >
> > Integer-only lua, no elua libs (but still with shell)
> >
> > eLua# lua -e 'print(collectgarbage("count").."\n")'
> > Press CTRL+Z to exit Lua
> > 12
> >
> So the difference between "bare bones" eLua (no libs, just the shell)
> in floating point mode and integer only mode is about 4k of memory.
> Now THIS is interesting, we need to figure out where this comes from
> and work on this.
> Now, finally, some ideas to improve RAM usage:
>
> - the constants in some of our modules (like PIO.Px_y, term.KC_xxxx)
> are now written explicitely, and thus consume RAM memory. We could use
> methamethods (index) instead to "resolve" these constants. This will
> of course add to the execution time, but I think it's a perfectly good
> compromise.
>
> - remove some of the modules that Lua preloads. The newly released
> 0.4.1 already removes the "os" and "package" modules, as they're not
> used anyway, and makes "math" configurable just like the platform
> modules.
>
> - implement "emergency garbage collection"
> (http://lua-users.org/wiki/EmergencyGarbageCollector). I did some
> tests with this, and it has a very strong impact on the memory
> consumption (see
> http://lua-users.org/lists/lua-l/2008-05/threads.html#00023 for more
> details). Unfortunately, it has a very strong impact on the execution
> speed two (at least on its most aggressive working mode) but this
> compromise is something one must live with in programming. This is a
> "must have" for eLua 0.5.
>
> - use a memory allocator with a smaller overhead. dlmalloc/TLSF/other
> seggregated allocators use some sort of "page tables" (TLSF has even a
> two-level page table) that consumes memory (on TLSF the overhead is as
> large as 3k!). I have a very simple, very fragmentable, yet very low
> overhead chained allocator that I wrote myself, and I intend to
> include it in a future version (maybe 0.5, but I'm not sure at this
> point). Then again, in my tests, this allocator didn't perform all
> that well, but this is VERY dependent on the application.
>
> - figure out what I mentioned earlier: why the difference between
> "bare bones" Lua in floating point mode and integer only mode is that
> large. Obviously it can't be because of the math library, I can't
> imagine a math library with such a huge RAM consumption (and I should
> know, I worked on two different math emulation libraries in my past).
> Could it be just because we don't register the "math" module anymore?
> It still seems that 4k is a LOT of memory gained from just removing
> the "math" module. I don't know how to test this, unfortunately.
>
> Thanks a lot for this e-mail, I was working on some memory statistics
> myself, but never for a minute thought about running
> collectgarbage("count"), as I completely forgot that collectgarbage
> can also return the ammount of memory used by Lua :)
> And, as a personal preference, could we continue this discussion on
> the elua-dev list?
>
> Best,
> Bogdan
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20080921/98945954/attachment.html