memory usage

classic Classic list List threaded Threaded
4 messages Options
Tim Van Der Hulst Tim Van Der Hulst
Reply | Threaded
Open this post in threaded view
|

memory usage

Bogdan,

Just following on from memory usage.

>with the latest eLua patches to Lua, you can execute _bytecode_ (only bytecode!) directly from the Flash, without having to
load it to RAM. To do that, you need to cross-compile your Lua code on the host side (http://www.eluaproject.net/doc/master/en_using.html#cross)
and upload the resulting bytecode file to your board using either ROMFS or WOFS.

I have been doing this for some time, to the point that I had all my modules compiled on the ROM and still getting memory errors. It seems that it still requires a large amount of RAM (more than size of module!) :(

Here is an example using a single module.

logFrameBuffer.lua - 280~ lines, implements a crude ring buffer with both writer/reader

5,293 bytes - 19 functions, pretty minimal amount of string/number. Some math, local variable usage etc. Nothing extraordinary.

Cross compiled (with -s flag) this is 4,672 bytes).

Encoded file logFrameBufferROM.lc (4676 bytes real size, 4704 bytes encoded size

Confirming via Elf file ".text" section increased 4,704 bytes. All good. Now..

---------

function mem()
        print(collectgarbage('count') * 1024)
end

mem()

require 'logFrameBufferROM'

mem()

----------

5179
10709

Eg - Memory has increased by 5,530 bytes!!!

That is more than the original bytecode file :(

--------------

> Each type in Lua is allocated on the heap (malloc and
friends) (yes, even numbers). Unfortunately, this is something we
can't really control much (although we have some patches here too,
their effect isn't nearly as dramatic as the above "bytecode from
Flash" optimization).

If I follow this guide for memory:

http://www.wowwiki.com/Lua_object_memory_sizes

Functions: 19 * 20 bytes = 380 bytes

Numbers: Approx 20
Strings: Approx 50 bytes worth

So I could probably realistically say it shouldn't be more than 1k of data that should go on the heap I suppose!?!

> You can also try to change the EGC patch with aggresive triggering

Tried all options and made little difference, might affect working memory but by the time my modules are loaded its around 40K which is pretty close to the threshold for the 64k SRAM on the Chip.

-----------

I hope there is something that I am doing wrong as it's unfortunate that even with bytecode in Flash it still needs lots of(SRAM) program memory for your average Lua code. I am starting to wonder if eLua is all that suitable for chips with limited RAM (64K and under), seems its fine for trivial applications with small amounts of code perhaps.

I really think I am in need of a chip with 128K SRAM but its kind of rare prospect as indicated in these old posts:

http://elua-development.2368040.n2.nabble.com/Single-Chip-Lua-Prospects-td3132204.html
http://elua-development.2368040.n2.nabble.com/Re-Single-Chip-Lua-Prospects-td4677969.html

Cheers,

TiM





_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
BogdanM BogdanM
Reply | Threaded
Open this post in threaded view
|

Re: memory usage

Hi,

On Tue, Jul 31, 2012 at 12:59 AM, Tim Van Der Hulst
<[hidden email]> wrote:

> Bogdan,
>
> Just following on from memory usage.
>
>>with the latest eLua patches to Lua, you can execute _bytecode_ (only bytecode!) directly from the Flash, without having to
> load it to RAM. To do that, you need to cross-compile your Lua code on the host side (http://www.eluaproject.net/doc/master/en_using.html#cross)
> and upload the resulting bytecode file to your board using either ROMFS or WOFS.
>
> I have been doing this for some time, to the point that I had all my modules compiled on the ROM and still getting memory errors. It seems that it still requires a large amount of RAM (more than size of module!) :(
>
> Here is an example using a single module.
>
> logFrameBuffer.lua - 280~ lines, implements a crude ring buffer with both writer/reader
>
> 5,293 bytes - 19 functions, pretty minimal amount of string/number. Some math, local variable usage etc. Nothing extraordinary.
>
> Cross compiled (with -s flag) this is 4,672 bytes).
>
> Encoded file logFrameBufferROM.lc (4676 bytes real size, 4704 bytes encoded size
>
> Confirming via Elf file ".text" section increased 4,704 bytes. All good. Now..
>
> ---------
>
> function mem()
>         print(collectgarbage('count') * 1024)
> end
>
> mem()
>
> require 'logFrameBufferROM'
>
> mem()
>
> ----------
>
> 5179
> 10709
>
> Eg - Memory has increased by 5,530 bytes!!!
>
> That is more than the original bytecode file :(

There is little connection between the size of the original bytecode
file and the amount of memory it uses. As an example, it's perfectly
possible to allocate megs of memory with a single Lua line. For
example:

s = string.rep( 'a', 32 * 1048576 ) -- this will allocate 32M of memory

Keeping in mind that 'require' actually executes the file it receives
as argument, it becomes clear that the actual content of
logFrameBufferROM is significant. If you can upload it here, we can
try to find a reason for that relatively high memory consumption.

>
> --------------
>
>> Each type in Lua is allocated on the heap (malloc and
> friends) (yes, even numbers). Unfortunately, this is something we
> can't really control much (although we have some patches here too,
> their effect isn't nearly as dramatic as the above "bytecode from
> Flash" optimization).
>
> If I follow this guide for memory:
>
> http://www.wowwiki.com/Lua_object_memory_sizes
>
> Functions: 19 * 20 bytes = 380 bytes
>
> Numbers: Approx 20
> Strings: Approx 50 bytes worth
>
> So I could probably realistically say it shouldn't be more than 1k of data that should go on the heap I suppose!?!

Probably, but again, the actual code in 'logFrameBufferROM' is very important.

>
>> You can also try to change the EGC patch with aggresive triggering
>
> Tried all options and made little difference, might affect working memory but by the time my modules are loaded its around 40K which is pretty close to the threshold for the 64k SRAM on the Chip.

I agree that this is quite weird. Unless your modules create a lot of
strings and/or tables at initialization time, I can't find a good
reason for this behaviour.

>
> -----------
>
> I hope there is something that I am doing wrong as it's unfortunate that even with bytecode in Flash it still needs lots of(SRAM) program memory for your average Lua code. I am starting to wonder if eLua is all that suitable for chips with limited RAM (64K and under), seems its fine for trivial applications with small amounts of code perhaps.

That is true, unfortunately. There are some tricks to work around
this, but ultimately 64k is in the "almost usable" realm for anything
but trivial applications. Changing this is extremely difficult. That
said, I'm extremely willing to help you investigate this issue, as it
might lead to finding new ways to make eLua more memory friendly.
Please try to find a simple example that can be used to reproduce the
problem you described and send it to me (or to the list, whichever you
prefer). Your 'logFrameBufferROM' seems to be a good example.

>
> I really think I am in need of a chip with 128K SRAM but its kind of rare prospect as indicated in these old posts:
>
> http://elua-development.2368040.n2.nabble.com/Single-Chip-Lua-Prospects-td3132204.html
> http://elua-development.2368040.n2.nabble.com/Re-Single-Chip-Lua-Prospects-td4677969.html

Things are not so bad in this area anymore. Some of the latest M4
cores, for example, come with a lot of internal RAM, although not
without some quirks (for example multiple internal RAM regions). Look
for example an the STM32F4 from ST or the LPC4300 from NXP (or the
Cortex-M3 LPC1800 family for even more internal RAM). Things are
getting better and better in this area.

Best,
Bogdan

>
> Cheers,
>
> TiM
>
>
>
>
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
Tim Van Der Hulst Tim Van Der Hulst
Reply | Threaded
Open this post in threaded view
|

Re: memory usage

Hi Bogdan,

You give me hope! :)

> Keeping in mind that 'require' actually executes the file it receives
as argument, it becomes clear that the actual content of
logFrameBufferROM is significant. If you can upload it here, we can
try to find a reason for that relatively high memory consumption

Ok attached module and associated unit test. As mentioned prior the code is pretty crude (but works without obvious bugs hah), the point is to improve it over time, I am sure it could be written much cleaner/simpler in C code but I am a web developer and would rather avoid writing C if and where possible for embedded projects except where it makes sense to, ie low level timing/device driver stuff maybe.

> I agree that this is quite weird. Unless your modules create a lot of
strings and/or tables at initialization time, I can't find a good
reason for this behaviour.

Feel free to rewrite my code to awesome, just kidding :P

> Some of the latest M4
cores, for example, come with a lot of internal RAM, although not
without some quirks (for example multiple internal RAM regions). Look
for example an the STM32F4 from ST or the LPC4300 from NXP (or the
Cortex-M3 LPC1800 family for even more internal RAM). Things are
getting better and better in this area.

Indeed, my colleague did tell me early on that he could solder on a replacement MCU which has 128K SRAM I believe but we were hoping to get through a prototype without having to go to that extreme.

Cheers,

TiM


----- Original Message -----
From: "Bogdan Marinescu" <[hidden email]>
To: "eLua Users and Development List (www.eluaproject.net)" <[hidden email]>
Sent: Tuesday, July 31, 2012 6:58:55 PM
Subject: Re: [eLua-dev] memory usage

Hi,

On Tue, Jul 31, 2012 at 12:59 AM, Tim Van Der Hulst
<[hidden email]> wrote:

> Bogdan,
>
> Just following on from memory usage.
>
>>with the latest eLua patches to Lua, you can execute _bytecode_ (only bytecode!) directly from the Flash, without having to
> load it to RAM. To do that, you need to cross-compile your Lua code on the host side (http://www.eluaproject.net/doc/master/en_using.html#cross)
> and upload the resulting bytecode file to your board using either ROMFS or WOFS.
>
> I have been doing this for some time, to the point that I had all my modules compiled on the ROM and still getting memory errors. It seems that it still requires a large amount of RAM (more than size of module!) :(
>
> Here is an example using a single module.
>
> logFrameBuffer.lua - 280~ lines, implements a crude ring buffer with both writer/reader
>
> 5,293 bytes - 19 functions, pretty minimal amount of string/number. Some math, local variable usage etc. Nothing extraordinary.
>
> Cross compiled (with -s flag) this is 4,672 bytes).
>
> Encoded file logFrameBufferROM.lc (4676 bytes real size, 4704 bytes encoded size
>
> Confirming via Elf file ".text" section increased 4,704 bytes. All good. Now..
>
> ---------
>
> function mem()
>         print(collectgarbage('count') * 1024)
> end
>
> mem()
>
> require 'logFrameBufferROM'
>
> mem()
>
> ----------
>
> 5179
> 10709
>
> Eg - Memory has increased by 5,530 bytes!!!
>
> That is more than the original bytecode file :(
There is little connection between the size of the original bytecode
file and the amount of memory it uses. As an example, it's perfectly
possible to allocate megs of memory with a single Lua line. For
example:

s = string.rep( 'a', 32 * 1048576 ) -- this will allocate 32M of memory

Keeping in mind that 'require' actually executes the file it receives
as argument, it becomes clear that the actual content of
logFrameBufferROM is significant. If you can upload it here, we can
try to find a reason for that relatively high memory consumption.

>
> --------------
>
>> Each type in Lua is allocated on the heap (malloc and
> friends) (yes, even numbers). Unfortunately, this is something we
> can't really control much (although we have some patches here too,
> their effect isn't nearly as dramatic as the above "bytecode from
> Flash" optimization).
>
> If I follow this guide for memory:
>
> http://www.wowwiki.com/Lua_object_memory_sizes
>
> Functions: 19 * 20 bytes = 380 bytes
>
> Numbers: Approx 20
> Strings: Approx 50 bytes worth
>
> So I could probably realistically say it shouldn't be more than 1k of data that should go on the heap I suppose!?!
Probably, but again, the actual code in 'logFrameBufferROM' is very important.

>
>> You can also try to change the EGC patch with aggresive triggering
>
> Tried all options and made little difference, might affect working memory but by the time my modules are loaded its around 40K which is pretty close to the threshold for the 64k SRAM on the Chip.

I agree that this is quite weird. Unless your modules create a lot of
strings and/or tables at initialization time, I can't find a good
reason for this behaviour.

>
> -----------
>
> I hope there is something that I am doing wrong as it's unfortunate that even with bytecode in Flash it still needs lots of(SRAM) program memory for your average Lua code. I am starting to wonder if eLua is all that suitable for chips with limited RAM (64K and under), seems its fine for trivial applications with small amounts of code perhaps.

That is true, unfortunately. There are some tricks to work around
this, but ultimately 64k is in the "almost usable" realm for anything
but trivial applications. Changing this is extremely difficult. That
said, I'm extremely willing to help you investigate this issue, as it
might lead to finding new ways to make eLua more memory friendly.
Please try to find a simple example that can be used to reproduce the
problem you described and send it to me (or to the list, whichever you
prefer). Your 'logFrameBufferROM' seems to be a good example.

>
> I really think I am in need of a chip with 128K SRAM but its kind of rare prospect as indicated in these old posts:
>
> http://elua-development.2368040.n2.nabble.com/Single-Chip-Lua-Prospects-td3132204.html
> http://elua-development.2368040.n2.nabble.com/Re-Single-Chip-Lua-Prospects-td4677969.html

Things are not so bad in this area anymore. Some of the latest M4
cores, for example, come with a lot of internal RAM, although not
without some quirks (for example multiple internal RAM regions). Look
for example an the STM32F4 from ST or the LPC4300 from NXP (or the
Cortex-M3 LPC1800 family for even more internal RAM). Things are
getting better and better in this area.

Best,
Bogdan

>
> Cheers,
>
> TiM
>
>
>
>
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

logFrameBuffer.lua (7K) Download Attachment
test.lua (2K) Download Attachment
BogdanM BogdanM
Reply | Threaded
Open this post in threaded view
|

Re: memory usage

Hi Tim,

On Wed, Aug 1, 2012 at 2:11 AM, Tim Van Der Hulst <[hidden email]> wrote:
> Hi Bogdan,
>
> You give me hope! :)

I'm glad I do :)

>
>> Keeping in mind that 'require' actually executes the file it receives
> as argument, it becomes clear that the actual content of
> logFrameBufferROM is significant. If you can upload it here, we can
> try to find a reason for that relatively high memory consumption
>
> Ok attached module and associated unit test. As mentioned prior the code is pretty crude (but works without obvious bugs hah), the point is to improve it over time, I am sure it could be written much cleaner/simpler in C code but I am a web developer and would rather avoid writing C if and where possible for embedded projects except where it makes sense to, ie low level timing/device driver stuff maybe.

No need to excuse yourself, eLua was created to encourage exactly
people like you to get into embedded programming. Plus, your code
doesn't look bad at all. That said, I must confess that at this point
I have no idea why it takes so much RAM. I'll have to look at this
more carefully, probably doing quite a bit of debugging/logging in the
process.

>
>> I agree that this is quite weird. Unless your modules create a lot of
> strings and/or tables at initialization time, I can't find a good
> reason for this behaviour.
>
> Feel free to rewrite my code to awesome, just kidding :P
>
>> Some of the latest M4
> cores, for example, come with a lot of internal RAM, although not
> without some quirks (for example multiple internal RAM regions). Look
> for example an the STM32F4 from ST or the LPC4300 from NXP (or the
> Cortex-M3 LPC1800 family for even more internal RAM). Things are
> getting better and better in this area.
>
> Indeed, my colleague did tell me early on that he could solder on a replacement MCU which has 128K SRAM I believe but we were hoping to get through a prototype without having to go to that extreme.

Actually ... if you trust him to do it, that couldn't possibly hurt :)
Even if we are able to find the root of your problems and come up with
a fix, you can never have enough RAM.

Best,
Bogdan

>
> Cheers,
>
> TiM
>
>
> ----- Original Message -----
> From: "Bogdan Marinescu" <[hidden email]>
> To: "eLua Users and Development List (www.eluaproject.net)" <[hidden email]>
> Sent: Tuesday, July 31, 2012 6:58:55 PM
> Subject: Re: [eLua-dev] memory usage
>
> Hi,
>
> On Tue, Jul 31, 2012 at 12:59 AM, Tim Van Der Hulst
> <[hidden email]> wrote:
>> Bogdan,
>>
>> Just following on from memory usage.
>>
>>>with the latest eLua patches to Lua, you can execute _bytecode_ (only bytecode!) directly from the Flash, without having to
>> load it to RAM. To do that, you need to cross-compile your Lua code on the host side (http://www.eluaproject.net/doc/master/en_using.html#cross)
>> and upload the resulting bytecode file to your board using either ROMFS or WOFS.
>>
>> I have been doing this for some time, to the point that I had all my modules compiled on the ROM and still getting memory errors. It seems that it still requires a large amount of RAM (more than size of module!) :(
>>
>> Here is an example using a single module.
>>
>> logFrameBuffer.lua - 280~ lines, implements a crude ring buffer with both writer/reader
>>
>> 5,293 bytes - 19 functions, pretty minimal amount of string/number. Some math, local variable usage etc. Nothing extraordinary.
>>
>> Cross compiled (with -s flag) this is 4,672 bytes).
>>
>> Encoded file logFrameBufferROM.lc (4676 bytes real size, 4704 bytes encoded size
>>
>> Confirming via Elf file ".text" section increased 4,704 bytes. All good. Now..
>>
>> ---------
>>
>> function mem()
>>         print(collectgarbage('count') * 1024)
>> end
>>
>> mem()
>>
>> require 'logFrameBufferROM'
>>
>> mem()
>>
>> ----------
>>
>> 5179
>> 10709
>>
>> Eg - Memory has increased by 5,530 bytes!!!
>>
>> That is more than the original bytecode file :(
>
> There is little connection between the size of the original bytecode
> file and the amount of memory it uses. As an example, it's perfectly
> possible to allocate megs of memory with a single Lua line. For
> example:
>
> s = string.rep( 'a', 32 * 1048576 ) -- this will allocate 32M of memory
>
> Keeping in mind that 'require' actually executes the file it receives
> as argument, it becomes clear that the actual content of
> logFrameBufferROM is significant. If you can upload it here, we can
> try to find a reason for that relatively high memory consumption.
>
>>
>> --------------
>>
>>> Each type in Lua is allocated on the heap (malloc and
>> friends) (yes, even numbers). Unfortunately, this is something we
>> can't really control much (although we have some patches here too,
>> their effect isn't nearly as dramatic as the above "bytecode from
>> Flash" optimization).
>>
>> If I follow this guide for memory:
>>
>> http://www.wowwiki.com/Lua_object_memory_sizes
>>
>> Functions: 19 * 20 bytes = 380 bytes
>>
>> Numbers: Approx 20
>> Strings: Approx 50 bytes worth
>>
>> So I could probably realistically say it shouldn't be more than 1k of data that should go on the heap I suppose!?!
>
> Probably, but again, the actual code in 'logFrameBufferROM' is very important.
>
>>
>>> You can also try to change the EGC patch with aggresive triggering
>>
>> Tried all options and made little difference, might affect working memory but by the time my modules are loaded its around 40K which is pretty close to the threshold for the 64k SRAM on the Chip.
>
> I agree that this is quite weird. Unless your modules create a lot of
> strings and/or tables at initialization time, I can't find a good
> reason for this behaviour.
>
>>
>> -----------
>>
>> I hope there is something that I am doing wrong as it's unfortunate that even with bytecode in Flash it still needs lots of(SRAM) program memory for your average Lua code. I am starting to wonder if eLua is all that suitable for chips with limited RAM (64K and under), seems its fine for trivial applications with small amounts of code perhaps.
>
> That is true, unfortunately. There are some tricks to work around
> this, but ultimately 64k is in the "almost usable" realm for anything
> but trivial applications. Changing this is extremely difficult. That
> said, I'm extremely willing to help you investigate this issue, as it
> might lead to finding new ways to make eLua more memory friendly.
> Please try to find a simple example that can be used to reproduce the
> problem you described and send it to me (or to the list, whichever you
> prefer). Your 'logFrameBufferROM' seems to be a good example.
>
>>
>> I really think I am in need of a chip with 128K SRAM but its kind of rare prospect as indicated in these old posts:
>>
>> http://elua-development.2368040.n2.nabble.com/Single-Chip-Lua-Prospects-td3132204.html
>> http://elua-development.2368040.n2.nabble.com/Re-Single-Chip-Lua-Prospects-td4677969.html
>
> Things are not so bad in this area anymore. Some of the latest M4
> cores, for example, come with a lot of internal RAM, although not
> without some quirks (for example multiple internal RAM regions). Look
> for example an the STM32F4 from ST or the LPC4300 from NXP (or the
> Cortex-M3 LPC1800 family for even more internal RAM). Things are
> getting better and better in this area.
>
> Best,
> Bogdan
>
>>
>> Cheers,
>>
>> TiM
>>
>>
>>
>>
>>
>> _______________________________________________
>> eLua-dev mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/elua-dev
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev