Lua Tiny RAM patch (finally) added to eLua

[jbsnyder]

Wow, this is amazing :-)

Based on a cursory look through, it sounds as if some tables and maybe functions are now classed as read-only? Is this so they don't have to be allocated in SRAM? If the numbers below are accurate, it seems there's quite a benefit to doing so. Is there much of a performance hit?

-jsnyder

----- Original Message -----
From: "Bogdan Marinescu" <bogdan.marinescu at gmail.com>
To: "eLua Users and Development List" <elua-dev at lists.berlios.de>
Sent: Sunday, January 11, 2009 2:57:19 PM GMT -06:00 US/Canada Central
Subject: [eLua-dev] Lua Tiny RAM patch (finally) added to eLua

Hi,

If you've been wondering why I suddenly stopped replying in the threads about the platform specific modules/ADC platform interface/whatever else, here's your answer: in the last couple of days I've been working like a crazy maniac on integrating my Lua Tiny RAM (LTR) patch in eLua. I'll send a larger e-mail explaining what's LTR later, for now let's review the svn commit log:

- Lua Tiny RAM (LTR) patch is now integrated in eLua and is enabled by default
- all eLua modules updated to work with LTR

And now for something really interesting: the results! The table below summarizes the RAM usage in kbytes (as obtained by running lua -e "print(collectgarbage'count')" from the eLua shell). OPT=0 is LTR's "compatibility mode" (basically this means that the patch is disabled, so you're running plain Lua) and OPT=2 is the patch in action.

|=============|============|============|
| PLATFORM | OPT=0 | OPT=2 |
|=============|============|============|
| AVR32 | 23.75 | 5.42 |
| AT91SAM7X | 25.16 | 5.42 |
| STR7 | 24.92 | 5.42 |
| STR9 | 22.23 | 5.42 |
| LPC2888 | 22.23 | 5.42 |
| I386 | 16.90 | 5.42 |
| LM3S | 27.14 | 5.42 |
| STM32 | ????? | ???? | (hint: 5.42:))
|=============|============|============|

Beautiful, no? :)

Mike , please test this on your STM3210 board (yes, I also modified your port) and let me know how it works.

- "cpu" module added to avr32, at91sam7x, str7

Pretty self-explanatory.

- "disp" module no longer generic (now stays in src/modules/lm3s). For this reason, the "disp" platform interface was also removed.

This concludes an older discussion on the list. The "disp" module was specific to the LM3S boards, and its platform interface wasn't generic enough, so it was moved to the LM3S backend and the platform interface was removed.

- the "modcommon" mechanism in STM32 (ROM loader) was depreciated in favour of the Lua Tiny RAM patch (and the "stm3210lcd" module from the STM32 backend now uses LTR).

Sorry, Mike :), but I really think this is the better way to go with this. Maybe we can still use parts of your mechanism in other parts of eLua.

- small bugfixes

I'll be coming back to our API-related threads after I manage to breathe a bit :) and after I send more information about LTR to the world.

Best,
Bogdan




_______________________________________________ Elua-dev mailing list Elua-dev at lists.berlios.de https://lists.berlios.de/mailman/listinfo/elua-dev 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090111/b6229038/attachment.html 

[Bittencourt]

My theory that Bogdan came from Valhalla to help us, pitful humans, with
eLua is getting validated.

--Pedro Bittencourt


On Mon, Jan 12, 2009 at 1:06 AM, James Snyder <jbsnyder at fanplastic.org>wrote:

> Wow, this is amazing :-)
>
> Based on a cursory look through, it sounds as if some tables and maybe
> functions are now classed as read-only? Is this so they don't have to be
> allocated in SRAM?  If the numbers below are accurate, it seems there's
> quite a benefit to doing so.  Is there much of a performance hit?
>
> -jsnyder
>
>
> ----- Original Message -----
> From: "Bogdan Marinescu" <bogdan.marinescu at gmail.com>
> To: "eLua Users and Development List" <elua-dev at lists.berlios.de>
> Sent: Sunday, January 11, 2009 2:57:19 PM GMT -06:00 US/Canada Central
> Subject: [eLua-dev] Lua Tiny RAM patch (finally) added to eLua
>
> Hi,
>
> If you've been wondering why I suddenly stopped replying in the threads
> about the platform specific modules/ADC platform interface/whatever else,
> here's your answer: in the last couple of days I've been working like a
> crazy maniac on integrating my Lua Tiny RAM (LTR) patch in eLua. I'll send a
> larger e-mail explaining what's LTR later, for now let's review the svn
> commit log:
>
> *- Lua Tiny RAM (LTR) patch is now integrated in eLua and is enabled by
> default
> - all eLua modules updated to work with LTR
>
> *And now for something really interesting: the results! The table below
> summarizes the RAM usage in kbytes (as obtained by running *lua -e
> "print(collectgarbage'count')" *from the eLua shell). OPT=0 is LTR's
> "compatibility mode" (basically this means that the patch is disabled, so
> you're running plain Lua) and OPT=2 is the patch in action.
>
> |=============|============|============|
> | PLATFORM    |   OPT=0    |   OPT=2    |
> |=============|============|============|
> | AVR32       |     23.75  |      5.42  |
> | AT91SAM7X   |     25.16  |      5.42  |
> | STR7        |     24.92  |      5.42  |
> | STR9        |     22.23  |      5.42  |
> | LPC2888     |     22.23  |      5.42  |
> | I386        |     16.90  |      5.42  |
> | LM3S        |     27.14  |      5.42  |
> | STM32       |     ?????  |      ????  | (hint: 5.42:))
> |=============|============|============|
>
> Beautiful, no? :)
>
> *Mike, *please test this on your STM3210 board (yes, I also modified your
> port) and let me know how it works.
>
> - *"cpu" module added to avr32, at91sam7x, str7*
>
> Pretty self-explanatory.
>
> * - "disp" module no longer generic (now stays in src/modules/lm3s). For
> this reason, the "disp" platform interface was also removed.*
>
> This concludes an older discussion on the list. The "disp" module was
> specific to the LM3S boards, and its platform interface wasn't generic
> enough, so it was moved to the LM3S backend and the platform interface was
> removed.
>
> *- the "modcommon" mechanism in STM32 (ROM loader) was depreciated in
> favour of the Lua Tiny RAM patch (and the "stm3210lcd" module from the STM32
> backend now uses LTR).*
>
> Sorry, Mike :), but I really think this is the better way to go with this.
> Maybe we can still use parts of your mechanism in other parts of eLua.
>
> * - small bugfixes*
>
> I'll be coming back to our API-related threads after I manage to breathe a
> bit :) and after I send more information about LTR to the world.
>
> Best,
> Bogdan
>
>
>
>
> _______________________________________________ Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
> _______________________________________________
> Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090112/4db7d7c2/attachment-0001.html 

[BogdanM]

OK people, it seems that this got your interested :), so these are the gory
details. First a few words about the patch itself ; the text below is a
modified version of the text I used to present the first version of the
patch in the Lua list, but now we're talking about the second version of the
patch, which is better, fixes some very nasty bugs from V1 and has more
features.

================================================================================

Audience
========

For people that use Lua on desktops, this patch is probably of no interest,
as its purpose is to minimize RAM footprint (at the expense of
speed/executable size). If, on the other hand, you're trying to run Lua on a
resource contrained system (like a microcontroller) or in a sandboxed
environment, and you're more concerned with memory usage than with speed,
you might find this patch useful.

Motivation
==========

The main thing that drove me to write this patch is Lua's memory consumption
at startup (obtained by running lua -e "print(collectgarbage'count')"). It's
about 17k for regular Lua 5.1.4, and more than 25k for some of eLua's
platforms. These figures are mainly a result of registering many different
modules to Lua. Each time you register a module (via luaL_register) you
create a new table and populate it with the module's methods. But a table is
a read/write datatype, so luaL_register is quite inefficient if you don't
plan to do any write operations on that table later (adding new elements or
manipulating existing ones). I found that I almost never have to do any such
operations on a module's table after it was created, I just query it for its
elements. So, from the perspective of someone worried about memory usage,
I'd rather have a different type of table in this case, one that wouldn't
need any RAM at all, since it would be read only, so it could reside
entirely in ROM.
And there's one more thing related to this context: Lua's functions. While
Lua does have the concept of C functions, they still require data structures
that need to be allocated (see lua_pushcclosure in lapi.c for details), as
they can have upvalues or environments. Once again, this isn't something I
use often with eLua. Most of the times my functions (especially the ones
exported by a C module) are very simple, and they don't need upvalues or
environments at all. In conclusion, having a "simpler" function type would
improve memory usage.

Details
=======

The patch adds two new data types to Lua. Both or them are based on the
lightuserdata type already found in Lua, and they share the same basic
attributes: they don't need to be dynamically allocated (as they're just
pointers on steroids) and they're compared in the same way lightuserdatas
are compared (by value). And of course, they are not collectable, so the
garbage collector won't have anything to do with them. The new types are:

1. lightfunctions: these are "simple" functions, in the sense that they
can't have upvalues or environments. They are just pointers to regular C
functions. Other than that, you can use them from Lua just as you'd use any
other function.

2. ro(m)tables: these are read-only tables, but unlike the read-only tables
that one can already implement in Lua with metamethods, they have a very
specific property: they don't need any RAM at all. They are fully constant,
so they can be read directly from ROM. They have a number of special
features and limitations when compared with a regular table:

- rotables can only contain values of type "lightfunction", lua_Number or
pointers to other rotables.
- you can't add/delete/modify elements from rotables (obviously).

*NEW IN V2: *

- you can use rotables as metatables for both "regular" tables and for Lua
types (via debug.setmetatable)
- rotabels can have another rotable (or itself) as a metatable
- rotabels can have both string and integer keys
- you can iterate over rotables with pairs/ipairs/next just as you do with
"regular" tables.

Just as with lightuserdata, you can only create lightfunctions and rotables
from C code, never from Lua itself.

How to use it
==========

The patch must be applied to an unmodified Lua 5.1.4 source tree. Once you
unpack the source tree and download the patch, do this:

$ cd lua-5.1.4
$ cp <location>/lua_tiny_ram.patch .
$ patch -p1 < lua_tiny_ram.patch

Then edit lua-5.1.4/src/Makefile to set the optimization level for the
patch:

CFLAGS= -O2 -Wall $(MYCFLAGS) -DLUA_OPTIMIZE_MEMORY=x, where x can be:

0: this doesn't actually optimize anything, in just tries to keep the
patched version as close as possible to the unpatched version in terms of
speed and functionality. This passes the full Lua 5.1 test suite (see
"Testing" below for details).

1: this applies a moderate optimization, the resulting code takes less
memory and still passes the full test suite. In this mode most of the
standard libraries are modified to use lightfunctions instead of "regular"
functions, and some of them are "converted" to use rotables instead of
regular tables.

2: aggresive optimizations, this modifies the standard libraries even more
than it does at level 1. The IO library is modified to use the registry
instead of environments, thus making it more resource-friendly (the side
effect being that this mode doesn't support pipes). It also leaves the _G
table with a single method (__index) and sets it as its own metatable, so
all the "standard" Lua functions are now hidden behind a __index access.
This mode passes almost all the tests in the Lua 5.1 test suite, except for
a few tests from "db.lua" (which fail because the functions in "io" are now
lightfunctions and they can't have upvalues anymore) and a few more  from
"nextvar.lua" (which fail because of reasons I don't fully understand, but
they have to do with the fact that _G's only method is now __index).

To use it in your own code, call "luaL_register_light" instead of
"luaL_register" to register your module with lightfunctions istead of
regular functions. Also, you can completeley forget about registering your
module if you use rotables. See linit.c, mathlib.c and oslib.c for an
example of how to turn your regular code into code that uses rotables.

Results
=======

All of the results come from running "lua -e "print(collectgarbage'count')""
from the command line.

*NOTE: these numbers are no longer perfectly accurate for version 2 of the
patch, but they're still pretty close to the real numbers.*

Regular Lua 5.1.4:           17.0966
Patched Lua 5.1.4, OPT = 0 : 17.0849 (the difference comes from a few small
changes in loadlib.c)
Patched Lua 5.1.4, OPT = 1 :  9.2011
Patched Lua 5.1.4, OPT = 2 :  5.8984

Before you ask: I don't have any information about how the patch alters the
execution speed, and I don't plan to do any soon, as this is not at all
something I'm concerned with at this point. Clearly the virtual machine
would be slower, but I have no idea how much slower. Of course, you're more
than welcomed to test this yourself and post your results.

Testing
=======

I tested my patch with the Lua 5.1 test suite (
http://lua-users.org/lists/lua-l/2006-03/msg00723.html). I had to modify it
a bit to take into account the new datatypes, the modified test suite is
available from ftp://ftp.berlios.de/pub/elua (anonymous). The test suite was
an excellent testing tool. I thought I had the patch ready until I found the
test suite and ran it. After another week of work, I had something that
could be called functional :)

I tested everything via "make generic", which is how I always build Lua for
my embedded environments. This means (among other things) that I didn't test
pipes and dynamic module loading, although I don't see why they wouldn't
work (except for pipes, which won't work at optmization level 2, as already
explained).

I never tested the patch in a multithreaded environment with more threads
running different lua_States. I never even used regular Lua like this, so I
can't make asumptions about how my patch would behave in a multithreaded
environment. It doesn't use any global or static variables, but you might
encounter other problems with it.

================================================================================

Coming up next: how to use the patch in eLua.

Best,
Bogdan


On Mon, Jan 12, 2009 at 5:46 AM, Pedro Bittencourt <
pedrobittencourt87 at gmail.com> wrote:

> My theory that Bogdan came from Valhalla to help us, pitful humans, with
> eLua is getting validated.
>
> --Pedro Bittencourt
>
>
> On Mon, Jan 12, 2009 at 1:06 AM, James Snyder <jbsnyder at fanplastic.org>wrote:
>
>> Wow, this is amazing :-)
>>
>> Based on a cursory look through, it sounds as if some tables and maybe
>> functions are now classed as read-only? Is this so they don't have to be
>> allocated in SRAM?  If the numbers below are accurate, it seems there's
>> quite a benefit to doing so.  Is there much of a performance hit?
>>
>> -jsnyder
>>
>>
>> ----- Original Message -----
>> From: "Bogdan Marinescu" <bogdan.marinescu at gmail.com>
>> To: "eLua Users and Development List" <elua-dev at lists.berlios.de>
>> Sent: Sunday, January 11, 2009 2:57:19 PM GMT -06:00 US/Canada Central
>> Subject: [eLua-dev] Lua Tiny RAM patch (finally) added to eLua
>>
>> Hi,
>>
>> If you've been wondering why I suddenly stopped replying in the threads
>> about the platform specific modules/ADC platform interface/whatever else,
>> here's your answer: in the last couple of days I've been working like a
>> crazy maniac on integrating my Lua Tiny RAM (LTR) patch in eLua. I'll send a
>> larger e-mail explaining what's LTR later, for now let's review the svn
>> commit log:
>>
>> *- Lua Tiny RAM (LTR) patch is now integrated in eLua and is enabled by
>> default
>> - all eLua modules updated to work with LTR
>>
>> *And now for something really interesting: the results! The table below
>> summarizes the RAM usage in kbytes (as obtained by running *lua -e
>> "print(collectgarbage'count')" *from the eLua shell). OPT=0 is LTR's
>> "compatibility mode" (basically this means that the patch is disabled, so
>> you're running plain Lua) and OPT=2 is the patch in action.
>>
>> |=============|============|============|
>> | PLATFORM    |   OPT=0    |   OPT=2    |
>> |=============|============|============|
>> | AVR32       |     23.75  |      5.42  |
>> | AT91SAM7X   |     25.16  |      5.42  |
>> | STR7        |     24.92  |      5.42  |
>> | STR9        |     22.23  |      5.42  |
>> | LPC2888     |     22.23  |      5.42  |
>> | I386        |     16.90  |      5.42  |
>> | LM3S        |     27.14  |      5.42  |
>> | STM32       |     ?????  |      ????  | (hint: 5.42:))
>> |=============|============|============|
>>
>> Beautiful, no? :)
>>
>> *Mike, *please test this on your STM3210 board (yes, I also modified your
>> port) and let me know how it works.
>>
>> - *"cpu" module added to avr32, at91sam7x, str7*
>>
>> Pretty self-explanatory.
>>
>> * - "disp" module no longer generic (now stays in src/modules/lm3s). For
>> this reason, the "disp" platform interface was also removed.*
>>
>> This concludes an older discussion on the list. The "disp" module was
>> specific to the LM3S boards, and its platform interface wasn't generic
>> enough, so it was moved to the LM3S backend and the platform interface was
>> removed.
>>
>> *- the "modcommon" mechanism in STM32 (ROM loader) was depreciated in
>> favour of the Lua Tiny RAM patch (and the "stm3210lcd" module from the STM32
>> backend now uses LTR).*
>>
>> Sorry, Mike :), but I really think this is the better way to go with this.
>> Maybe we can still use parts of your mechanism in other parts of eLua.
>>
>> * - small bugfixes*
>>
>> I'll be coming back to our API-related threads after I manage to breathe a
>> bit :) and after I send more information about LTR to the world.
>>
>> Best,
>> Bogdan
>>
>>
>>
>>
>> _______________________________________________ Elua-dev mailing list
>> Elua-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>
>> _______________________________________________
>> Elua-dev mailing list
>> Elua-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>
>>
>
> _______________________________________________
> Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090112/34f6008e/attachment-0001.html 

[Frédéric THOMAS-5]

Brilliant.

Thanks Bogdan!

Fred

On Jan 12, 2009, at 09:57 , Bogdan Marinescu wrote:

> - Lua Tiny RAM (LTR) patch is now integrated in eLua and is enabled  
> by default
> - all eLua modules updated to work with LTR

[BogdanM]

Part II: how to actually use the patch.

If you read my previous message carefully (just kidding, I know you did :)
), you noticed that the patch is controlled via the
*LUA_OPTIMIZE_MEMORY*macro, that can have 3 values: 0 (patch not
active), 1 (medium
optimizations) and 2 (aggressive optimizations). In eLua, however, we'll use
just 0 and 2, to keep things less confusing (and because it makes sense to
do so for the current targets of eLua). A new scons parameter was added to
control this optimization:

*scons <args> optram = 0* (disable optimization). In this mode
luaL_register_light will still use regular Lua tables and functions, just to
maintain full compatiblity with "regular" Lua. Also, in this mode, you can't
use "global" rotables ("global" rotables are explained below). You can use
this mode for higher compatibility with Lua, for higher speed (although I
don't think that the difference in speed is so large) or to test the patch
itself (if you have a Lua program that's not working with optimizations
enabled, but works with *optram=0*, chances are that LTR is screwing
something up). I might remove this mode as some point, but not anytime soon.
*scons <args> optram = 1* (enable optimization, which actually means that *
LUA_OPTIMIZE_MEMORY* is set to 2. *optram=1* is also the default, so you
don't need to specify it explicitly in the scons command line).

Now for the interesting part: how to actually use it. Let's assume that you
want to register a simple module called "*mod*" that has a single function
named "f". For regular Lua, you'd do something like this:

*static const luaL_reg mod_map[] =
{
  { "f", f_implementation },
  { NULL, NULL }
};

LUALIB_API int luaopen_mod( lua_State *L )
{
  luaL_register( L, "mod", mod_map );
  return 1;
}
*
For the rotables implementation, however, you'd need to define the same
thing like this:

*const luaR_entry mod_map[] = **// note: no static this time*
*{
  { LRO_STRKEY( "f" ), LRO_FUNCVAL( f_implementation ) },
  { LRO_NILKEY, LRO_NILVAL }
};

// note: in this case the "luaopen_mod" function isn't really needed anymore
LUALIB_API int luaopen_mod( lua_State *L )
{
  return 0;
}
*
A few points about the rotables example above:

- a rotable needs a "map" (*mod_map*) array much like a regular module, but
you need to define that array with special macros:
  - for keys: *LRO_STRKEY("str")* defines a string key,
*LRO_NUMKEY(n)*defines an integer key, and
*LRO_NILKEY* defines a NULL (empty) key
  - for values: *LRO_FUNCVAL(f)* defines a lightfunction value, *
LRO_NUMVAL(f)* defines a number value, *LRO_RO(p)* defines a rotable value
(p is the pointer to the rotable) and *LRO_NILVAL* defines a NULL (empty)
value.
- all the "global" rotables in the system (the ones that must be visible
from _G, like the rotables of all the modules exported to Lua) must be
included in a special array, called *lua_rotable *(defined in *linit.c*).
Simply including the rotable's definition array (*mod_map *in this case) in
the *lua_rotable *array makes it visible globally, thus you don't need to
call any kind of register function.

The two forms above (for regular tables and for rotables) are clearly
different, but we want to keep them both to be able to work at both optlevel
0 and optlevel 2. You can use #ifdefs to differentiate between the two cases
in different optimization levels, but this becomes really annoying after a
(short) while. This is why I added another file called *lrodefs.h* (src/lua)
that can be used to give an "universal" definition to our map arrays. Once
again, an example is the best way to describe this:

*#define MIN_OPT_LEVEL  2 // the minimum optimization level at which we use
rotables
#include "lrodefs.h"
**const LUA_REG_TYPE mod_map[] = // note: no more luaL_reg or luaR_entry
**{
  { LSTRKEY( "f" ), LFUNCVAL( f_implementation ) },
  { LNILKEY, LNILVAL }
};
// note: no more LRO_something, just Lsomething (for example LRO_STRKEY
becomes LSTRKEY)

**LUALIB_API int luaopen_mod( lua_State *L )
{
  LREGISTER( L, "mod", mod_map ); // note: no more luaL_register, no "return
1"
}
*
Now, if *LUA_OPTIMIZE_MEMORY* is less than *MIN_OPT_LEVEL* (thus 0 in the
case of eLua), the above definition will compile in its "regular table"
format. If *LUA_OPTIMIZE_MEMORY* is 2, it compiles to the rotables format.
Problem solved :) *"LREGISTER"* will also take care of calling
"luaL_register" and return 1 and optlevel 0 and do absolutely nothing at
optlevel 2. You can see more examples of this in any module from* *
src/modules.

To actually use all this stuff in eLua, simply replace your old definition
of platform modules from *platform_conf.h *like this:

*Old format:

#define LUA_PLATFORM_LIBS\
  { AUXLIB_PIO, luaopen_pio },\
  { AUXLIB_TMR, luaopen_tmr },\
  { AUXLIB_PD, luaopen_pd },\
  { AUXLIB_UART, luaopen_uart },\
  { AUXLIB_TERM, luaopen_term },\
  { AUXLIB_PWM, luaopen_pwm },\
  { AUXLIB_PACK, luaopen_pack },\
  { AUXLIB_BIT, luaopen_bit },\
  { LUA_MATHLIBNAME, luaopen_math }**

New format:

#define LUA_PLATFORM_LIBS_ROM\  // note: this time the macro is called
LUA_PLATFORM_LIBS_ROM
  _ROM( AUXLIB_PIO, luaopen_pio, pio_map )\
  _ROM( AUXLIB_TMR, luaopen_tmr, tmr_map )\
  _ROM( AUXLIB_PD, luaopen_pd, pd_map )\
  _ROM( AUXLIB_UART, luaopen_uart, uart_map )\
  _ROM( AUXLIB_TERM, luaopen_term, term_map )\
  _ROM( AUXLIB_PWM, luaopen_pwm, pwm_map )\
  _ROM( AUXLIB_PACK, luaopen_pack, pack_map )\
  _ROM( AUXLIB_BIT, luaopen_bit, bit_map )\
  _ROM( LUA_MATHLIBNAME, luaopen_math, math_map )

IMPORTANT: *the fact that there are no commas between two different _ROM
declaration (as seen above) is NOT an error; on the contrary, this is very
much intended. Try using commas and you'll get in trouble really soon :)

Note the addition of a 3rd parameter to the _ROM macro, which is the name of
the definition array for the (ro)table.
That's it. The code in linit.c will take care of everything else, including
initializing the list of modules in *LUA_PLATFORM_LIBS_ROM *with regular
tables instead of rotables at optlevel 0.
You can also have a list of modules that you to use regular tables no matter
what the optimization level is. In that case, list it in the
*LUA_PLATFORM_LIBS_REG
*macro via the *old *syntax for *LUA_PLATFORM_LIBS *(the regular Lua syntax
for defining a module to be registered with luaL_register). If you want this
module to use lightfunctions instead of regular functions (at optlevel 2),
use "*luaL_register_light*" instead of "*luaL_register*".

Other notes:

- for eLua, *MIN_OPT_LEVEL* should be always set to 2
- you need a C99-compatible compiler to use this (because of the
compile-time explicit union initialization that's needed to declare const
rotables). Fortunately this isn't a issue right now, as all current eLua
targets use GCC and GCC knows how to handle this.
- your linker command file should export two symbols: *stext *and *etext*.
They should be declared before and after the .rodata* section placement
(generally you'd declare *stext *at the beginning of *.text *definition and
*etext *and the end of *.text* definition, see for example
src/lua/at91sam7x256/flash256.lds for an example). These are needed by the
patch to differentiate between a regular table and a rotable (long story
here :), will get into details if you want me to).
- *remember to declare all you rotable's definition array as 'const'!!
*Forgetting
to do so will not only increase memory usage, it will also render the patch
not functional, because of the way it recognizes rotables (see above).
- rotables can have metatables. If a rotable must have a metatable, it needs
a *"__metatable"* field to point to its metatable (which is also a rotable,
not necessarily another rotable) and the usual metatable functions. For
example, let's make our *mod *rotable its own metatable and declare an
__index function. Moreover, let's do this for both optlevel 0 and optlevel
2.

*static int mod_mt_index( lua_State *L ) {
  return 0;
}*

*#define MIN_OPT_LEVEL  2 // the minimum optimization level at which we use
rotables
#include "lrodefs.h"
**const LUA_REG_TYPE mod_map[] =
**{
  { LSTRKEY( "f" ), LFUNCVAL( f_implementation ) },
#if LUA_OPTIMIZE_MEMORY > 0
  { LSTRKEY( "__metatable" ), LROVAL( mod_map ) },
#endif
**  { LSTRKEY( "__index" ), LFUNCVAL( mod_mt_index) },*
*  { LNILKEY, LNILVAL };*
*};

**LUALIB_API int luaopen_mod( lua_State *L )
{
#if LUA_OPTIMIZE_MEMORY > 0
  return 0;
#else
  luaL_register( L, "mod", mod_map );

  // Set "mod" as its own metatable
  lua_pushvalue( L, -1 );
  lua_setmetatable( L, -2 );

  return 1;
#endif
}*

That's it for now, I hope I was clear enough. I know there will be
questions, please feel free to ask anything.

Best,
Bogdan

On Mon, Jan 12, 2009 at 10:57 AM, Bogdan Marinescu <
bogdan.marinescu at gmail.com> wrote:

> OK people, it seems that this got your interested :), so these are the gory
> details. First a few words about the patch itself ; the text below is a
> modified version of the text I used to present the first version of the
> patch in the Lua list, but now we're talking about the second version of the
> patch, which is better, fixes some very nasty bugs from V1 and has more
> features.
>
>
> ================================================================================
>
> Audience
> ========
>
> For people that use Lua on desktops, this patch is probably of no interest,
> as its purpose is to minimize RAM footprint (at the expense of
> speed/executable size). If, on the other hand, you're trying to run Lua on a
> resource contrained system (like a microcontroller) or in a sandboxed
> environment, and you're more concerned with memory usage than with speed,
> you might find this patch useful.
>
> Motivation
> ==========
>
> The main thing that drove me to write this patch is Lua's memory
> consumption at startup (obtained by running lua -e
> "print(collectgarbage'count')"). It's about 17k for regular Lua 5.1.4, and
> more than 25k for some of eLua's platforms. These figures are mainly a
> result of registering many different modules to Lua. Each time you register
> a module (via luaL_register) you create a new table and populate it with the
> module's methods. But a table is a read/write datatype, so luaL_register is
> quite inefficient if you don't plan to do any write operations on that table
> later (adding new elements or manipulating existing ones). I found that I
> almost never have to do any such operations on a module's table after it was
> created, I just query it for its elements. So, from the perspective of
> someone worried about memory usage, I'd rather have a different type of
> table in this case, one that wouldn't need any RAM at all, since it would be
> read only, so it could reside entirely in ROM.
> And there's one more thing related to this context: Lua's functions. While
> Lua does have the concept of C functions, they still require data structures
> that need to be allocated (see lua_pushcclosure in lapi.c for details), as
> they can have upvalues or environments. Once again, this isn't something I
> use often with eLua. Most of the times my functions (especially the ones
> exported by a C module) are very simple, and they don't need upvalues or
> environments at all. In conclusion, having a "simpler" function type would
> improve memory usage.
>
> Details
> =======
>
> The patch adds two new data types to Lua. Both or them are based on the
> lightuserdata type already found in Lua, and they share the same basic
> attributes: they don't need to be dynamically allocated (as they're just
> pointers on steroids) and they're compared in the same way lightuserdatas
> are compared (by value). And of course, they are not collectable, so the
> garbage collector won't have anything to do with them. The new types are:
>
> 1. lightfunctions: these are "simple" functions, in the sense that they
> can't have upvalues or environments. They are just pointers to regular C
> functions. Other than that, you can use them from Lua just as you'd use any
> other function.
>
> 2. ro(m)tables: these are read-only tables, but unlike the read-only tables
> that one can already implement in Lua with metamethods, they have a very
> specific property: they don't need any RAM at all. They are fully constant,
> so they can be read directly from ROM. They have a number of special
> features and limitations when compared with a regular table:
>
> - rotables can only contain values of type "lightfunction", lua_Number or
> pointers to other rotables.
> - you can't add/delete/modify elements from rotables (obviously).
>
> *NEW IN V2: *
>
> - you can use rotables as metatables for both "regular" tables and for Lua
> types (via debug.setmetatable)
> - rotabels can have another rotable (or itself) as a metatable
> - rotabels can have both string and integer keys
> - you can iterate over rotables with pairs/ipairs/next just as you do with
> "regular" tables.
>
> Just as with lightuserdata, you can only create lightfunctions and rotables
> from C code, never from Lua itself.
>
> How to use it
> ==========
>
> The patch must be applied to an unmodified Lua 5.1.4 source tree. Once you
> unpack the source tree and download the patch, do this:
>
> $ cd lua-5.1.4
> $ cp <location>/lua_tiny_ram.patch .
> $ patch -p1 < lua_tiny_ram.patch
>
> Then edit lua-5.1.4/src/Makefile to set the optimization level for the
> patch:
>
> CFLAGS= -O2 -Wall $(MYCFLAGS) -DLUA_OPTIMIZE_MEMORY=x, where x can be:
>
> 0: this doesn't actually optimize anything, in just tries to keep the
> patched version as close as possible to the unpatched version in terms of
> speed and functionality. This passes the full Lua 5.1 test suite (see
> "Testing" below for details).
>
> 1: this applies a moderate optimization, the resulting code takes less
> memory and still passes the full test suite. In this mode most of the
> standard libraries are modified to use lightfunctions instead of "regular"
> functions, and some of them are "converted" to use rotables instead of
> regular tables.
>
> 2: aggresive optimizations, this modifies the standard libraries even more
> than it does at level 1. The IO library is modified to use the registry
> instead of environments, thus making it more resource-friendly (the side
> effect being that this mode doesn't support pipes). It also leaves the _G
> table with a single method (__index) and sets it as its own metatable, so
> all the "standard" Lua functions are now hidden behind a __index access.
> This mode passes almost all the tests in the Lua 5.1 test suite, except for
> a few tests from "db.lua" (which fail because the functions in "io" are now
> lightfunctions and they can't have upvalues anymore) and a few more  from
> "nextvar.lua" (which fail because of reasons I don't fully understand, but
> they have to do with the fact that _G's only method is now __index).
>
> To use it in your own code, call "luaL_register_light" instead of
> "luaL_register" to register your module with lightfunctions istead of
> regular functions. Also, you can completeley forget about registering your
> module if you use rotables. See linit.c, mathlib.c and oslib.c for an
> example of how to turn your regular code into code that uses rotables.
>
> Results
> =======
>
> All of the results come from running "lua -e
> "print(collectgarbage'count')"" from the command line.
>
> *NOTE: these numbers are no longer perfectly accurate for version 2 of the
> patch, but they're still pretty close to the real numbers.*
>
> Regular Lua 5.1.4:           17.0966
> Patched Lua 5.1.4, OPT = 0 : 17.0849 (the difference comes from a few small
> changes in loadlib.c)
> Patched Lua 5.1.4, OPT = 1 :  9.2011
> Patched Lua 5.1.4, OPT = 2 :  5.8984
>
> Before you ask: I don't have any information about how the patch alters the
> execution speed, and I don't plan to do any soon, as this is not at all
> something I'm concerned with at this point. Clearly the virtual machine
> would be slower, but I have no idea how much slower. Of course, you're more
> than welcomed to test this yourself and post your results.
>
> Testing
> =======
>
> I tested my patch with the Lua 5.1 test suite (
> http://lua-users.org/lists/lua-l/2006-03/msg00723.html). I had to modify
> it a bit to take into account the new datatypes, the modified test suite is
> available from ftp://ftp.berlios.de/pub/elua (anonymous). The test suite
> was an excellent testing tool. I thought I had the patch ready until I found
> the test suite and ran it. After another week of work, I had something that
> could be called functional :)
>
> I tested everything via "make generic", which is how I always build Lua for
> my embedded environments. This means (among other things) that I didn't test
> pipes and dynamic module loading, although I don't see why they wouldn't
> work (except for pipes, which won't work at optmization level 2, as already
> explained).
>
> I never tested the patch in a multithreaded environment with more threads
> running different lua_States. I never even used regular Lua like this, so I
> can't make asumptions about how my patch would behave in a multithreaded
> environment. It doesn't use any global or static variables, but you might
> encounter other problems with it.
>
>
> ================================================================================
>
> Coming up next: how to use the patch in eLua.
>
> Best,
> Bogdan
>
>
> On Mon, Jan 12, 2009 at 5:46 AM, Pedro Bittencourt <
> pedrobittencourt87 at gmail.com> wrote:
>
>> My theory that Bogdan came from Valhalla to help us, pitful humans, with
>> eLua is getting validated.
>>
>> --Pedro Bittencourt
>>
>>
>> On Mon, Jan 12, 2009 at 1:06 AM, James Snyder <jbsnyder at fanplastic.org>wrote:
>>
>>> Wow, this is amazing :-)
>>>
>>> Based on a cursory look through, it sounds as if some tables and maybe
>>> functions are now classed as read-only? Is this so they don't have to be
>>> allocated in SRAM?  If the numbers below are accurate, it seems there's
>>> quite a benefit to doing so.  Is there much of a performance hit?
>>>
>>> -jsnyder
>>>
>>>
>>> ----- Original Message -----
>>> From: "Bogdan Marinescu" <bogdan.marinescu at gmail.com>
>>> To: "eLua Users and Development List" <elua-dev at lists.berlios.de>
>>> Sent: Sunday, January 11, 2009 2:57:19 PM GMT -06:00 US/Canada Central
>>> Subject: [eLua-dev] Lua Tiny RAM patch (finally) added to eLua
>>>
>>> Hi,
>>>
>>> If you've been wondering why I suddenly stopped replying in the threads
>>> about the platform specific modules/ADC platform interface/whatever else,
>>> here's your answer: in the last couple of days I've been working like a
>>> crazy maniac on integrating my Lua Tiny RAM (LTR) patch in eLua. I'll send a
>>> larger e-mail explaining what's LTR later, for now let's review the svn
>>> commit log:
>>>
>>> *- Lua Tiny RAM (LTR) patch is now integrated in eLua and is enabled by
>>> default
>>> - all eLua modules updated to work with LTR
>>>
>>> *And now for something really interesting: the results! The table below
>>> summarizes the RAM usage in kbytes (as obtained by running *lua -e
>>> "print(collectgarbage'count')" *from the eLua shell). OPT=0 is LTR's
>>> "compatibility mode" (basically this means that the patch is disabled, so
>>> you're running plain Lua) and OPT=2 is the patch in action.
>>>
>>> |=============|============|============|
>>> | PLATFORM    |   OPT=0    |   OPT=2    |
>>> |=============|============|============|
>>> | AVR32       |     23.75  |      5.42  |
>>> | AT91SAM7X   |     25.16  |      5.42  |
>>> | STR7        |     24.92  |      5.42  |
>>> | STR9        |     22.23  |      5.42  |
>>> | LPC2888     |     22.23  |      5.42  |
>>> | I386        |     16.90  |      5.42  |
>>> | LM3S        |     27.14  |      5.42  |
>>> | STM32       |     ?????  |      ????  | (hint: 5.42:))
>>> |=============|============|============|
>>>
>>> Beautiful, no? :)
>>>
>>> *Mike, *please test this on your STM3210 board (yes, I also modified
>>> your port) and let me know how it works.
>>>
>>> - *"cpu" module added to avr32, at91sam7x, str7*
>>>
>>> Pretty self-explanatory.
>>>
>>> * - "disp" module no longer generic (now stays in src/modules/lm3s). For
>>> this reason, the "disp" platform interface was also removed.*
>>>
>>> This concludes an older discussion on the list. The "disp" module was
>>> specific to the LM3S boards, and its platform interface wasn't generic
>>> enough, so it was moved to the LM3S backend and the platform interface was
>>> removed.
>>>
>>> *- the "modcommon" mechanism in STM32 (ROM loader) was depreciated in
>>> favour of the Lua Tiny RAM patch (and the "stm3210lcd" module from the STM32
>>> backend now uses LTR).*
>>>
>>> Sorry, Mike :), but I really think this is the better way to go with
>>> this. Maybe we can still use parts of your mechanism in other parts of eLua.
>>>
>>> * - small bugfixes*
>>>
>>> I'll be coming back to our API-related threads after I manage to breathe
>>> a bit :) and after I send more information about LTR to the world.
>>>
>>> Best,
>>> Bogdan
>>>
>>>
>>>
>>>
>>> _______________________________________________ Elua-dev mailing list
>>> Elua-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>>
>>> _______________________________________________
>>> Elua-dev mailing list
>>> Elua-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>>
>>>
>>
>> _______________________________________________
>> Elua-dev mailing list
>> Elua-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090112/ae3f44e2/attachment-0001.html