eLua and yielding with lua_yield() function

[mpthompson]

OK, working with eLua I'm now starting to get into the more esoteric details of Lua (at least for me) that I can really use some help with.

Trying to start at a very simple level, I'm working on testing event driven aspects of Lua with the following C function which simply reports on a button press -- yielding with lua_yield() if it isn't pressed:

-----------------------
static int button_waitpressed( lua_State* L )
{
  unsigned id;
  u32 res;
 
  id = luaL_checkinteger( L, 1 );
  res = platform_button_pressed( id );
  if (!res)
    return lua_yield( L, 0 );
  lua_pushinteger( L, res );
  return 1;
}
-----------------------

My Lua program to test this is:

-----------------------
function buttonloop(x)
  local state
  repeat
    repeat
      state = button.waitpressed(x)
    until state == 1
    print "button pressed"
  until false
end

waitforbutton = coroutine.create(buttonloop(0))
while button.pressed(1) == 0 do
  coroutine.resume(waitforbutton)
end
-----------------------

Following the examples in chapter 30, I believe the code above should work and report the string "button pressed" whenever the button is pressed, but instead I get the following error:

attempt to yield across metamethod/C-call boundary

It seems this error should occur when you call from Lua a C function which in turn invokes Lua code which yields -- something I'm pretty sure I'm not doing in this example as I'm yielding on the return.

I suspect that there is something about the eLua environment that is different from running under an operating system.  I've found some references on the web that state Lua code called from pcall() trigger this error -- perhaps the lua interpreter under eLua is called with this.  I'll obviously have to do some digging.  However, I'm hoping that others more familiar with eLua might have some thoughts on where I can begin to understand this error further.

Thanks,

Mike Thompson

[mpthompson]

mpthompson wrote:
 > OK, working with eLua I'm now starting to get into the more
 > esoteric details of Lua (at least for me) that I can really
 > use some help with.

Ooops, sorry for the false alarm.  I wish I could recall silly emails...

The problem was my attempt to call the function when creating the
coroutine.  The altered code below works perfectly as expected.

-----------------------------
function buttonloop()
   local state
   repeat
     repeat
       state = button.waitpressed(0)
     until state == 1
     print "button pressed"
   until false
end

waitforbutton = coroutine.create(buttonloop)
while button.pressed(1) == 0 do
   coroutine.resume(waitforbutton)
end
-----------------------------

The button loop function yields correctly from C code to the main
routine until the button0 is pressed when it will start spitting out
"button pressed" until released.  Pressing button1 then stops the program.

I'm reason for doing this is in the kids programming IDE I'm creating
for the Lego NXT brick is "highly" event driven.  I'll likely have
scores, if not hundreds, of such routines waiting for events to happen
and snippets of code to be kicked off in response.  My interest will be
in making such event triggered constructs as efficient as possible so
that hundreds of events might be handled without too much overhead.

For efficiency I would like to keep all event handling code in C, but I
believe I'm limited by some architectural decisions behind Lua (which
are a good thing as it made things like eLua possible).  Getting Coco
running on eLua is probably asking too much, but it would allow true C
coroutine semantics so I can yield from a coroutine across a C call
boundary and resume back to it.

Mike Thompson
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Hmmmm. I don't think Coco on eLua would be that difficult. I'll give it a closer look these days, in-between two documents :)

Best,
Bogdan

On Thu, Apr 9, 2009 at 8:59 AM, Mike Thompson <[hidden email]> wrote:

mpthompson wrote:
 > OK, working with eLua I'm now starting to get into the more
 > esoteric details of Lua (at least for me) that I can really
 > use some help with.

Ooops, sorry for the false alarm.  I wish I could recall silly emails...

The problem was my attempt to call the function when creating the
coroutine.  The altered code below works perfectly as expected.

-----------------------------
function buttonloop()
  local state
  repeat
    repeat
      state = button.waitpressed(0)

    until state == 1
    print "button pressed"
  until false
end

waitforbutton = coroutine.create(buttonloop)

while button.pressed(1) == 0 do
  coroutine.resume(waitforbutton)
end

-----------------------------

The button loop function yields correctly from C code to the main
routine until the button0 is pressed when it will start spitting out
"button pressed" until released.  Pressing button1 then stops the program.

I'm reason for doing this is in the kids programming IDE I'm creating
for the Lego NXT brick is "highly" event driven.  I'll likely have
scores, if not hundreds, of such routines waiting for events to happen
and snippets of code to be kicked off in response.  My interest will be
in making such event triggered constructs as efficient as possible so
that hundreds of events might be handled without too much overhead.

For efficiency I would like to keep all event handling code in C, but I
believe I'm limited by some architectural decisions behind Lua (which
are a good thing as it made things like eLua possible).  Getting Coco
running on eLua is probably asking too much, but it would allow true C
coroutine semantics so I can yield from a coroutine across a C call
boundary and resume back to it.

Mike Thompson

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

I suppose this again brings up the whole lua interrupt concept, but I'm wondering how this currently could work with hundreds of events.

I assume that you still need some kind of dispatcher like the following: http://www.lua.org/pil/9.4.html

Is the advantage one might get from something like coco a reduction in overhead and bookkeeping, since you can resume something on the C side rather than resuming something on the Lua side that calls something in C?

I suppose you could replace the Lua dispatcher with some sort of event coroutine that runs on the C side, and gives you an event type or message if something has happened that is registered to be reported.  I can see that sort of thing being fairly useful with a few other bits to handle what to do with the messages in lua.

Am I off-track here?

----- "Mike Thompson" <[hidden email]> wrote:

> mpthompson wrote:
>  > OK, working with eLua I'm now starting to get into the more
>  > esoteric details of Lua (at least for me) that I can really
>  > use some help with.
>
> Ooops, sorry for the false alarm.  I wish I could recall silly
> emails...
>
> The problem was my attempt to call the function when creating the
> coroutine.  The altered code below works perfectly as expected.
>
> -----------------------------
> function buttonloop()
>    local state
>    repeat
>      repeat
>        state = button.waitpressed(0)
>      until state == 1
>      print "button pressed"
>    until false
> end
>
> waitforbutton = coroutine.create(buttonloop)
> while button.pressed(1) == 0 do
>    coroutine.resume(waitforbutton)
> end
> -----------------------------
>
> The button loop function yields correctly from C code to the main
> routine until the button0 is pressed when it will start spitting out
> "button pressed" until released.  Pressing button1 then stops the
> program.
>
> I'm reason for doing this is in the kids programming IDE I'm creating
>
> for the Lego NXT brick is "highly" event driven.  I'll likely have
> scores, if not hundreds, of such routines waiting for events to happen
>
> and snippets of code to be kicked off in response.  My interest will
> be
> in making such event triggered constructs as efficient as possible so
>
> that hundreds of events might be handled without too much overhead.
>
> For efficiency I would like to keep all event handling code in C, but
> I
> believe I'm limited by some architectural decisions behind Lua (which
>
> are a good thing as it made things like eLua possible).  Getting Coco
>
> running on eLua is probably asking too much, but it would allow true C
>
> coroutine semantics so I can yield from a coroutine across a C call
> boundary and resume back to it.
>
> Mike Thompson
> _______________________________________________
> Elua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[mpthompson]

James Snyder wrote:
> I suppose this again brings up the whole lua interrupt concept, but I'm wondering how this currently could work with hundreds of events.
>
> I assume that you still need some kind of dispatcher like the following: http://www.lua.org/pil/9.4.html
>
> Is the advantage one might get from something like coco a reduction in overhead and bookkeeping, since you can resume something on the C side rather than resuming something on the Lua side that calls something in C?
>
> I suppose you could replace the Lua dispatcher with some sort of event coroutine that runs on the C side, and gives you an event type or message if something has happened that is registered to be reported.  I can see that sort of thing being fairly useful with a few other bits to handle what to do with the messages in lua.
>
> Am I off-track here?

You're not off track at all.  I'm not yet familiar enough with Lua to
understand what exactly is practical, but my thinking is creating a
central scheduler/dispatcher written in C for efficiency and compactness
of implementation.  I'll then have a Lua component that initializes and
registers two pieces of code for each event handler within the application.

The first piece is a Lua/C function that defines a condition for when an
event should be handled.  Examples of event conditions could be a button
press, the distance sensor indicating an object at a specific distance,
the light sensor detecting light or dark, a timer going off, a packet of
data received over the USB port or virtual events generated from other
pieces of code.  Certain conditions can be handled entirely within C
such as system level events and timers while other will need the
snippets of Lua code to handle more complex conditions such those
involving math expressions being processed.

The second piece is the the actual event handler which is really just a
coroutine within Lua.  Once a condition for an event is detected (i.e.
the code above returns non-nil) the scheduler then schedules the
co-routine to run.  I'll make provisions so that loops created within
the code will yield appropriately so as not to drown out all other event
handling if a long loop is created.

The scheduler is in a loop waiting for event handling code to yield and
then serially checking the conditions to see if other event handlers are
ready to be resumed.

My ultimate goal is to create a children's programming IDE for Lego
robotics (and other robots) that is very much inspired by Scratch which
is created by the Life Long Kindergarten group at MIT.  Kids graphically
assemble their programs in the IDE (so they are really never exposed
directly to the underlying Lua language) and then the IDE converts the
GUI elements to Lua algorithmically, compiles it to bytecode and then
downloads the code to the robot for execution.  I'm fairly certain that
with some care and imagination, I can create something much more
friendly for kids than Lego's own NXT-G environment.  To achieve this I
need a bytecode interpreter running on the NXT and so far Lua seems to
fit the bill nicely.  My alternative is Java bytecode which seems
bloated compared to my experience so far with the lightness of Lua.

My challenge is preserving Scratch-like programming semantics as much as
possible (i.e. keeping it highly event driven) yet still produce code
that will execute well on a very low power Arm processor.

Because the kids aren't programming directly in Lua I'll have control
over the resulting code such as making sure that looping constructs
appropriately call yield operations to allow other code to run.  Some
aspects of this might be a challenge, but I have a feeling Lua's
cooperative threading will be MUCH easier to deal with than if I had
actual preemptive threads to deal with.

As much fun as I'm having with embedded Lua, it's really just a piece of
the overall application I'm hoping to create.

Mike
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Apr 9, 2009, at 11:42 AM, Mike Thompson wrote:

> James Snyder wrote:
>> I suppose this again brings up the whole lua interrupt concept, but  
>> I'm wondering how this currently could work with hundreds of events.
>>
>> I assume that you still need some kind of dispatcher like the  
>> following: http://www.lua.org/pil/9.4.html
>>
>> Is the advantage one might get from something like coco a reduction  
>> in overhead and bookkeeping, since you can resume something on the  
>> C side rather than resuming something on the Lua side that calls  
>> something in C?
>>
>> I suppose you could replace the Lua dispatcher with some sort of  
>> event coroutine that runs on the C side, and gives you an event  
>> type or message if something has happened that is registered to be  
>> reported.  I can see that sort of thing being fairly useful with a  
>> few other bits to handle what to do with the messages in lua.
>>
>> Am I off-track here?
>
> You're not off track at all.  I'm not yet familiar enough with Lua to
> understand what exactly is practical, but my thinking is creating a
> central scheduler/dispatcher written in C for efficiency and  
> compactness
> of implementation.  I'll then have a Lua component that initializes  
> and
> registers two pieces of code for each event handler within the  
> application.
>
> The first piece is a Lua/C function that defines a condition for  
> when an
> event should be handled.  Examples of event conditions could be a  
> button
> press, the distance sensor indicating an object at a specific  
> distance,
> the light sensor detecting light or dark, a timer going off, a  
> packet of
> data received over the USB port or virtual events generated from other
> pieces of code.  Certain conditions can be handled entirely within C
> such as system level events and timers while other will need the
> snippets of Lua code to handle more complex conditions such those
> involving math expressions being processed.
>
> The second piece is the the actual event handler which is really  
> just a
> coroutine within Lua.  Once a condition for an event is detected (i.e.
> the code above returns non-nil) the scheduler then schedules the
> co-routine to run.  I'll make provisions so that loops created within
> the code will yield appropriately so as not to drown out all other  
> event
> handling if a long loop is created.
>
> The scheduler is in a loop waiting for event handling code to yield  
> and
> then serially checking the conditions to see if other event handlers  
> are
> ready to be resumed.

I see you've thought this through :-)  I think this would be excellent  
to have as a general facility that's easy to hook into.  In recent  
weeks I've been trying to think about how to couple things like the  
analog conversions with coroutines to make things more efficient than  
just constantly polling from the lua side.

I think, also, that Bogdan may be right about it being not hugely  
painful to get Coco working.  While it can use POSIX facilities for  
context switching, it looks like there are a number of other ways to  
do that including inline assembler and setjmp, so that may be fairly  
doable too.

> My ultimate goal is to create a children's programming IDE for Lego
> robotics (and other robots) that is very much inspired by Scratch  
> which
> is created by the Life Long Kindergarten group at MIT.  Kids  
> graphically
> assemble their programs in the IDE (so they are really never exposed
> directly to the underlying Lua language) and then the IDE converts the
> GUI elements to Lua algorithmically, compiles it to bytecode and then
> downloads the code to the robot for execution.  I'm fairly certain  
> that
> with some care and imagination, I can create something much more
> friendly for kids than Lego's own NXT-G environment.  To achieve  
> this I
> need a bytecode interpreter running on the NXT and so far Lua seems to
> fit the bill nicely.  My alternative is Java bytecode which seems
> bloated compared to my experience so far with the lightness of Lua.
>
> My challenge is preserving Scratch-like programming semantics as  
> much as
> possible (i.e. keeping it highly event driven) yet still produce code
> that will execute well on a very low power Arm processor.
>
> Because the kids aren't programming directly in Lua I'll have control
> over the resulting code such as making sure that looping constructs
> appropriately call yield operations to allow other code to run.  Some
> aspects of this might be a challenge, but I have a feeling Lua's
> cooperative threading will be MUCH easier to deal with than if I had
> actual preemptive threads to deal with.

They certainly have downsides, but it does, by default, steer one away  
from a lot of potential issues with making things thread-safe.

It would be somewhat nice to have green threads in Lua, although I  
guess you can sort of do that with Coco or with the debug hooks.

Here are a few interesting discussions that are related:
http://lists.luaforge.net/pipermail/kepler-project/2008-October/003004.html
http://lua-users.org/lists/lua-l/2008-07/msg00666.html

>
> As much fun as I'm having with embedded Lua, it's really just a  
> piece of
> the overall application I'm hoping to create.

Very neat project.  I don't have any NXT devices, but I'll be  
interested in watching where this goes.

It should also give us some interesting information about how eLua  
works for these usage styles :-)

--
James Snyder
Biomedical Engineering
Northwestern University
[hidden email]
http://fanplastic.org/key.txt
ph: [phone]


_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[mpthompson]

James Snyder wrote:
> I see you've thought this through :-)  I think this would be excellent
> to have as a general facility that's easy to hook into.  In recent weeks
> I've been trying to think about how to couple things like the analog
> conversions with coroutines to make things more efficient than just
> constantly polling from the lua side.

Hmmm, I haven't thought of making it a general purpose facility.  I'll
need to think about that more as it would help evolve anything put
together better than me working on my lonesome.

> I think, also, that Bogdan may be right about it being not hugely
> painful to get Coco working.  While it can use POSIX facilities for
> context switching, it looks like there are a number of other ways to do
> that including inline assembler and setjmp, so that may be fairly doable
> too.

I only spent about 5 minutes looking at Coco so far, but it appeared to
me that it depended on OS features for implementation.  Not sure how
easy that would be to replicate in the raw eLua environment.  Based on
the Bogdan's comments I'll plan to spend more time looking into how Coco
works to see what is really involved.  It's ported to a number of
environments so hopefully there is already enough of a bread trail to
understand what the effort would be.

The JIT would be cool to :-), but not if it increases the ROM footprint.

> Here are a few interesting discussions that are related:
> http://lists.luaforge.net/pipermail/kepler-project/2008-October/003004.html
> http://lua-users.org/lists/lua-l/2008-07/msg00666.html

Thanks for the links.  I'll read through these over the weekend.

Mike
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Apr 9, 2009, at 4:30 PM, Mike Thompson wrote:

> James Snyder wrote:
>> I see you've thought this through :-)  I think this would be  
>> excellent
>> to have as a general facility that's easy to hook into.  In recent  
>> weeks
>> I've been trying to think about how to couple things like the analog
>> conversions with coroutines to make things more efficient than just
>> constantly polling from the lua side.
>
> Hmmm, I haven't thought of making it a general purpose facility.  I'll
> need to think about that more as it would help evolve anything put
> together better than me working on my lonesome.

I'd be happy to chip in here.

>
>> I think, also, that Bogdan may be right about it being not hugely
>> painful to get Coco working.  While it can use POSIX facilities for
>> context switching, it looks like there are a number of other ways  
>> to do
>> that including inline assembler and setjmp, so that may be fairly  
>> doable
>> too.
>
> I only spent about 5 minutes looking at Coco so far, but it appeared  
> to
> me that it depended on OS features for implementation.

Being curious, I've already taken the coco patch and applied it to a  
branch of eLua :-)

It basically looks like _setjmp and _longjmp which save signals needs  
to be set up or appropriately implemented.  I'm not sure if this would  
be something that could be implemented with a small amount of code, or  
if some sort of threading/ c-side coroutine library would be needed to  
make this work.

It's mentioned at one of the links below, but one of the alternatives  
is to use the the resumable VM patch, which doesn't require platform  
specific functionality.
http://lua-users.org/wiki/ResumableVmPatch

One unfortunate issue with that is the patch hasn't been updated for  
5.1.4, and, at least to me, the fixes for the patch appear non-obvious  
and would require some work...

--
James Snyder
Biomedical Engineering
Northwestern University
[hidden email]
http://fanplastic.org/key.txt
ph: [phone]


_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[mpthompson]

On Thu, Apr 9, 2009 at 5:34 PM, James Snyder <[hidden email]> wrote:
> Being curious, I've already taken the coco patch and applied it to a branch
> of eLua :-)
>
> It basically looks like _setjmp and _longjmp which save signals needs to be
> set up or appropriately implemented.  I'm not sure if this would be
> something that could be implemented with a small amount of code, or if some
> sort of threading/ c-side coroutine library would be needed to make this
> work.

Out of curiosity, how close is the source code in '/src/lua' to the
standard 5.1.4 distribution.  I've think I've seen it said that very
few changes were made to the standard Lua source code for eLua.

I'll try to take a look at this over the weekend.  It certainly seems
that Coco is worthy of some examination to see what is involved.

Mike
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

Applying the patch was mostly clean, it took me about a minute to fix the hunks that didn't apply.

1. I didn't bother with the Makefile patch, that didn't apply cleanly at all, but I don't think it's needed for normal builds using scons.
2. the layout of files is different from standard lua, so you'll likely have to point patch to the right files.
3. You'll need  to add lcoco.c to lua_files in SConstruct.

The main modifications will presumably be needed in lcoco.c.

Here's a bit of info on the associated _setjmp/_longjmp functions/macros:
http://opengroup.org/onlinepubs/007908799/xsh/_longjmp.html
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/_setjmp.3.html

The differences there in definition suggest to me why platform specific implementations are needed :-)  I assume, however that those, in conjunction with the auto-configuration done in lcoco.c should help in teasing out what's needed for the implementation.

I suppose we could also ping Mike Pall for suggestions.

Best.

-jsnyder

----- "Mike Thompson" <[hidden email]> wrote:

> On Thu, Apr 9, 2009 at 5:34 PM, James Snyder <[hidden email]>
> wrote:
> > Being curious, I've already taken the coco patch and applied it to a
> branch
> > of eLua :-)
> >
> > It basically looks like _setjmp and _longjmp which save signals
> needs to be
> > set up or appropriately implemented.  I'm not sure if this would be
> > something that could be implemented with a small amount of code, or
> if some
> > sort of threading/ c-side coroutine library would be needed to make
> this
> > work.
>
> Out of curiosity, how close is the source code in '/src/lua' to the
> standard 5.1.4 distribution.  I've think I've seen it said that very
> few changes were made to the standard Lua source code for eLua.
>
> I'll try to take a look at this over the weekend.  It certainly seems
> that Coco is worthy of some examination to see what is involved.
>
> Mike
> _______________________________________________
> Elua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Hmmm. We don't really have signals, at least not POSIX signals, so I'm not sure that we should worry about _setjmp/_longjmp. This is only a hunch though, I didn't actually check this.
About the JIT: I looked at that, and porting it to ARM is definitely not an easy task. Maybe the next version of LuaJIT will make things easier in this area, we have to wait and see.
My approach to interrupts was however much simpler that what you're suggesting here: keep a queue of interrupts, and set a Lua hook as long as this queue is not empty. The hook will then jump to the corresponding Lua interrupt handler. Just like lua.c does in order to handle CTRL+C (SIGINT).

Best,
Bogdan

On Fri, Apr 10, 2009 at 8:40 AM, James Snyder <[hidden email]> wrote:

Applying the patch was mostly clean, it took me about a minute to fix the hunks that didn't apply.

1. I didn't bother with the Makefile patch, that didn't apply cleanly at all, but I don't think it's needed for normal builds using scons.
2. the layout of files is different from standard lua, so you'll likely have to point patch to the right files.
3. You'll need  to add lcoco.c to lua_files in SConstruct.

The main modifications will presumably be needed in lcoco.c.

Here's a bit of info on the associated _setjmp/_longjmp functions/macros:
http://opengroup.org/onlinepubs/007908799/xsh/_longjmp.html
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/_setjmp.3.html

The differences there in definition suggest to me why platform specific implementations are needed :-)  I assume, however that those, in conjunction with the auto-configuration done in lcoco.c should help in teasing out what's needed for the implementation.

I suppose we could also ping Mike Pall for suggestions.

Best.

-jsnyder

----- "Mike Thompson" <[hidden email]> wrote:

> On Thu, Apr 9, 2009 at 5:34 PM, James Snyder <[hidden email]>
> wrote:
> > Being curious, I've already taken the coco patch and applied it to a
> branch
> > of eLua :-)
> >
> > It basically looks like _setjmp and _longjmp which save signals
> needs to be
> > set up or appropriately implemented.  I'm not sure if this would be
> > something that could be implemented with a small amount of code, or
> if some
> > sort of threading/ c-side coroutine library would be needed to make
> this
> > work.
>
> Out of curiosity, how close is the source code in '/src/lua' to the
> standard 5.1.4 distribution.  I've think I've seen it said that very
> few changes were made to the standard Lua source code for eLua.
>
> I'll try to take a look at this over the weekend.  It certainly seems
> that Coco is worthy of some examination to see what is involved.
>
> Mike
> _______________________________________________
> Elua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Apr 10, 2009, at 2:40 AM, Bogdan Marinescu wrote:

Hmmm. We don't really have signals, at least not POSIX signals, so I'm not sure that we should worry about _setjmp/_longjmp. This is only a hunch though, I didn't actually check this.

Right, and it looks like it doesn't actually need the signal stuff, just the stack/context switching.  I think I may be close to a working patch.  I'm trying to figure out what order stuff is stored in the jmp buffer since Coco wants to explicitly modify those.  I'll commit it on a branch if it works at all :-)

Also, in line with the discussion we had some time ago about Duff's device, someone has used a similar mechanism to do C coroutines (not sure if this would ever be useful for anything we're doing but it's interesting):

http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html

About the JIT: I looked at that, and porting it to ARM is definitely not an easy task. Maybe the next version of LuaJIT will make things easier in this area, we have to wait and see.

Isn't LuaJIT x86 specific?

My approach to interrupts was however much simpler that what you're suggesting here: keep a queue of interrupts, and set a Lua hook as long as this queue is not empty. The hook will then jump to the corresponding Lua interrupt handler. Just like lua.c does in order to handle CTRL+C (SIGINT).

I guess the main difference here is that the multitasking is explicit with the coroutine method vs debug hooks.  The debug hooks could also be used to do some sort of threading I think too.

Hmm.. I see no harm in trying both and seeing how well they work for different approaches in a branch.

Best,
Bogdan

On Fri, Apr 10, 2009 at 8:40 AM, James Snyder <[hidden email]> wrote:

Applying the patch was mostly clean, it took me about a minute to fix the hunks that didn't apply.

1. I didn't bother with the Makefile patch, that didn't apply cleanly at all, but I don't think it's needed for normal builds using scons.
2. the layout of files is different from standard lua, so you'll likely have to point patch to the right files.
3. You'll need  to add lcoco.c to lua_files in SConstruct.

The main modifications will presumably be needed in lcoco.c.

Here's a bit of info on the associated _setjmp/_longjmp functions/macros:
http://opengroup.org/onlinepubs/007908799/xsh/_longjmp.html
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/_setjmp.3.html

The differences there in definition suggest to me why platform specific implementations are needed :-)  I assume, however that those, in conjunction with the auto-configuration done in lcoco.c should help in teasing out what's needed for the implementation.

I suppose we could also ping Mike Pall for suggestions.

Best.

-jsnyder

----- "Mike Thompson" <[hidden email]> wrote:

> On Thu, Apr 9, 2009 at 5:34 PM, James Snyder <[hidden email]>
> wrote:
> > Being curious, I've already taken the coco patch and applied it to a
> branch
> > of eLua :-)
> >
> > It basically looks like _setjmp and _longjmp which save signals
> needs to be
> > set up or appropriately implemented.  I'm not sure if this would be
> > something that could be implemented with a small amount of code, or
> if some
> > sort of threading/ c-side coroutine library would be needed to make
> this
> > work.
>
> Out of curiosity, how close is the source code in '/src/lua' to the
> standard 5.1.4 distribution.  I've think I've seen it said that very
> few changes were made to the standard Lua source code for eLua.
>
> I'll try to take a look at this over the weekend.  It certainly seems
> that Coco is worthy of some examination to see what is involved.
>
> Mike
> _______________________________________________
> Elua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

--

James Snyder

Biomedical Engineering

Northwestern University

[hidden email]

http://fanplastic.org/key.txt

ph: [phone]

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

OK.  I made it work I think.  It runs the example lua scripts that come with coco without error so long as the counts are scaled back in a few cases (one example tries to make 100 coroutines.. since each one needs a stack.. that gets big :-).

There are definitely some areas where things could be tuned a bit better for low memory situations.  I'm experimentally trying to figure out what a reasonable stack size is. At the moment it's been pretty needy, getting bent out of shape if I give it less than 1K. Also, right now the whole thing seems to blow up if it runs out of memory, which could be improved :-)

I'll put it in a branch tonight or tomorrow in case anyone else wants to look at the current state.

-jsnyder

On Apr 10, 2009, at 5:39 PM, James Snyder wrote:

On Apr 10, 2009, at 2:40 AM, Bogdan Marinescu wrote:

Hmmm. We don't really have signals, at least not POSIX signals, so I'm not sure that we should worry about _setjmp/_longjmp. This is only a hunch though, I didn't actually check this.

Right, and it looks like it doesn't actually need the signal stuff, just the stack/context switching.  I think I may be close to a working patch.  I'm trying to figure out what order stuff is stored in the jmp buffer since Coco wants to explicitly modify those.  I'll commit it on a branch if it works at all :-)

Also, in line with the discussion we had some time ago about Duff's device, someone has used a similar mechanism to do C coroutines (not sure if this would ever be useful for anything we're doing but it's interesting):

http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html

About the JIT: I looked at that, and porting it to ARM is definitely not an easy task. Maybe the next version of LuaJIT will make things easier in this area, we have to wait and see.

Isn't LuaJIT x86 specific?

My approach to interrupts was however much simpler that what you're suggesting here: keep a queue of interrupts, and set a Lua hook as long as this queue is not empty. The hook will then jump to the corresponding Lua interrupt handler. Just like lua.c does in order to handle CTRL+C (SIGINT).

I guess the main difference here is that the multitasking is explicit with the coroutine method vs debug hooks.  The debug hooks could also be used to do some sort of threading I think too.

Hmm.. I see no harm in trying both and seeing how well they work for different approaches in a branch.

Best,
Bogdan

On Fri, Apr 10, 2009 at 8:40 AM, James Snyder <[hidden email]> wrote:

Applying the patch was mostly clean, it took me about a minute to fix the hunks that didn't apply.

1. I didn't bother with the Makefile patch, that didn't apply cleanly at all, but I don't think it's needed for normal builds using scons.
2. the layout of files is different from standard lua, so you'll likely have to point patch to the right files.
3. You'll need  to add lcoco.c to lua_files in SConstruct.

The main modifications will presumably be needed in lcoco.c.

Here's a bit of info on the associated _setjmp/_longjmp functions/macros:
http://opengroup.org/onlinepubs/007908799/xsh/_longjmp.html
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man3/_setjmp.3.html

The differences there in definition suggest to me why platform specific implementations are needed :-)  I assume, however that those, in conjunction with the auto-configuration done in lcoco.c should help in teasing out what's needed for the implementation.

I suppose we could also ping Mike Pall for suggestions.

Best.

-jsnyder

----- "Mike Thompson" <[hidden email]> wrote:

> On Thu, Apr 9, 2009 at 5:34 PM, James Snyder <[hidden email]>
> wrote:
> > Being curious, I've already taken the coco patch and applied it to a
> branch
> > of eLua :-)
> >
> > It basically looks like _setjmp and _longjmp which save signals
> needs to be
> > set up or appropriately implemented.  I'm not sure if this would be
> > something that could be implemented with a small amount of code, or
> if some
> > sort of threading/ c-side coroutine library would be needed to make
> this
> > work.
>
> Out of curiosity, how close is the source code in '/src/lua' to the
> standard 5.1.4 distribution.  I've think I've seen it said that very
> few changes were made to the standard Lua source code for eLua.
>
> I'll try to take a look at this over the weekend.  It certainly seems
> that Coco is worthy of some examination to see what is involved.
>
> Mike
> _______________________________________________
> Elua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

--

James Snyder

Biomedical Engineering

Northwestern University

[hidden email]

http://fanplastic.org/key.txt

ph: [phone]

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

--

James Snyder

Biomedical Engineering

Northwestern University

[hidden email]

http://fanplastic.org/key.txt

ph: [phone]

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[mpthompson]

James Snyder wrote:

> OK.  I made it work I think.  It runs the example lua scripts that come
> with coco without error so long as the counts are scaled back in a few
> cases (one example tries to make 100 coroutines.. since each one needs a
> stack.. that gets big :-).
>
> There are definitely some areas where things could be tuned a bit better
> for low memory situations.  I'm experimentally trying to figure out what
> a reasonable stack size is. At the moment it's been pretty needy,
> getting bent out of shape if I give it less than 1K. Also, right now the
> whole thing seems to blow up if it runs out of memory, which could be
> improved :-)
>
> I'll put it in a branch tonight or tomorrow in case anyone else wants to
> look at the current state.
>
> -jsnyder

James,

I would definitely be interested in taking a look at your branch.  If it
can be made to work well, it may impact the direction I take with the
project I'm working on.  However, it does seem that specifying a 1K
stack for each C coroutine imposes a fairly heavy price in a limited RAM
environment for the features that Coco offers.

Looking over the Coco documentation it looks that both regular
coroutines and Coco C coroutines can be mixed so perhaps it becomes a
matter of using the new Coco capabilities judiciously on coroutines that
really need the capability.

Unfortunately, a family emergency will be taking up a lot of my time
over the next few weeks so it may take me a while to get back with
comments on your work once a new branch is created.

Mike
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Apr 13, 2009, at 2:12 PM, Mike Thompson wrote:

> James,
>
> I would definitely be interested in taking a look at your branch.  
> If it
> can be made to work well, it may impact the direction I take with the
> project I'm working on.  However, it does seem that specifying a 1K
> stack for each C coroutine imposes a fairly heavy price in a limited  
> RAM
> environment for the features that Coco offers.

Yeah, I'm not sure why that was the case.  I didn't do that much  
testing though, so I may be missing something.

Sorry I didn't post it over the weekend into the main repository.  For  
some reason I was getting some weird issues with git-svn not wanting  
to create the branch for me.  I know why it's complaining, but I  
didn't get around to fixing the issue.  I'll do so in the next few days.

The code is up here if you don't mind browsing in git:
http://github.com/jsnyder/elua/tree/cocobranch

I did not include the examples that come with coco.  Many of them will  
probably just blow up if you run them straight without modification.  
Try reducing the number of coroutines created to 10 and then ramp it  
up for each case.

> Looking over the Coco documentation it looks that both regular
> coroutines and Coco C coroutines can be mixed so perhaps it becomes a
> matter of using the new Coco capabilities judiciously on coroutines  
> that
> really need the capability.

I'm not sure what the deal is there.  Perhaps there's some stack  
imparted by Lua that's increasing the amount of space needed?

I'll email Mike Pall, who wrote the Coco patch, about some of these  
issues sometime this week, and see if he has any suggestions/comments.

>
> Unfortunately, a family emergency will be taking up a lot of my time
> over the next few weeks so it may take me a while to get back with
> comments on your work once a new branch is created.

I'm sorry to hear that.  I hope all goes well.

-jsnyder

--
James Snyder
Biomedical Engineering
Northwestern University
[hidden email]
http://fanplastic.org/key.txt
ph: [phone]


_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[mpthompson]

James Snyder wrote:
> Sorry I didn't post it over the weekend into the main repository.  For
> some reason I was getting some weird issues with git-svn not wanting to
> create the branch for me.  I know why it's complaining, but I didn't get
> around to fixing the issue.  I'll do so in the next few days.

No big hurry at all, but I'm very happy you got something working.

Right now I'm just beginning to code up my scheduler in pure Lua to
fully understand the different approaches I might take and later on I'll
start figuring out how to pull different portions into C.

Mike
_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

James,

I would definitely be interested in taking a look at your branch.  If it
can be made to work well, it may impact the direction I take with the
project I'm working on.  However, it does seem that specifying a 1K
stack for each C coroutine imposes a fairly heavy price in a limited RAM
environment for the features that Coco offers.

For sure, but I don't know if you can do better than this.
 

Looking over the Coco documentation it looks that both regular
coroutines and Coco C coroutines can be mixed so perhaps it becomes a
matter of using the new Coco capabilities judiciously on coroutines that
really need the capability.

"Judiciously" is the right word, indeed. Overusing Coco is generally not a big deal on desktops, but might be quite different on embedded. Personally I never had to use Coco, but I'm not the best to judge here, as I use Lua rarely on desktops.
Unfortunately, mixing Lua coroutines and Coco coroutines might lead to problems, since after you apply Coco all the coroutines are created WITH a C stack, you need to give special arguments to coroutine.create if you want a regular coroutine. I'd rather have it the other way around to preserve compatibility with old code.
 

Unfortunately, a family emergency will be taking up a lot of my time
over the next few weeks so it may take me a while to get back with
comments on your work once a new branch is created.

Sorry to hear that, hope it will end soon and good :)
And thanks both for pushing Coco into eLua, it might appeal to some people. We can put it into trunk if we make it configurable at compile time.

Best,
Bogdan

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Apr 14, 2009, at 7:17 AM, Bogdan Marinescu wrote:

 

Looking over the Coco documentation it looks that both regular
coroutines and Coco C coroutines can be mixed so perhaps it becomes a
matter of using the new Coco capabilities judiciously on coroutines that
really need the capability.

"Judiciously" is the right word, indeed. Overusing Coco is generally not a big deal on desktops, but might be quite different on embedded. Personally I never had to use Coco, but I'm not the best to judge here, as I use Lua rarely on desktops.

The way the patch is originally set up each new coroutine gets 60k of stack space, which, obviously was way WAY too much for eLua.  I'd somewhat like to inspect what really is on the stack that needs to be preserved when you create a coroutine from Lua, but I haven't investigated yet.  Bogdan do you have a sense for this?

I also haven't done the next obvious thing and see how many coroutines I can create in stock eLua, so I have no idea whether the 1k overhead is really ridiculous or comparable.  I'll do some experiments today.  I'm TA'ing a lab where the students are using NXT bricks to make a tensile testing machine for biomaterials :-)  (unfortunately they're using MATLAB, but I'll certainly mention pbLua and eLua to the profs for future reference).

Unfortunately, mixing Lua coroutines and Coco coroutines might lead to problems, since after you apply Coco all the coroutines are created WITH a C stack, you need to give special arguments to coroutine.create if you want a regular coroutine. I'd rather have it the other way around to preserve compatibility with old code.

I'm pretty sure we could reverse that behavior at the cost of making it behave differently than the regular coco patch.  That might make a good way to have it be reasonable to turn coco on by default (when platform support is properly set up).

 

Unfortunately, a family emergency will be taking up a lot of my time
over the next few weeks so it may take me a while to get back with
comments on your work once a new branch is created.

Sorry to hear that, hope it will end soon and good :)
And thanks both for pushing Coco into eLua, it might appeal to some people. We can put it into trunk if we make it configurable at compile time.

Luckily it's a non-destructive patch.  You can restore default behavior with a define that disables coco.  I would suggest that if it gets merged that we make disabled the default state, and then it can be selectively enabled where desired.  If it turns out we can make the overhead really minimal, maybe they can be on by default at some point.

--

James Snyder

Biomedical Engineering

Northwestern University

[hidden email]

http://fanplastic.org/key.txt

ph: [phone]

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Jesus Alvarez]

What is the proper syntax to configure and read an 8-bit port (rather than a
pin) in eLua? I tried both

pio.configport(pio.DIR, pio.DIR_INPUT, "PA")
pio.configport(pio.PULL, pio.PULL_UP, "PA")
io.write("PA = ", pio.getport("PA"), "\n")

and

pio.dir["PA"] = pio.INPUT
pio.pull["PA"] = pio.PULLUP
io.write("PA = ", pio["PA"], "\n")

The first one results in syntax error and the second doesn't read the port.

BTW - configport() and getport() are listed in the documentation directory
(en/refman.html). Were those used in an older version?

Regards,
Jesus Alvarez

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Hi,

Currently the PIO module is in a highly experimental state, I'll probably restore the previous syntax soon. However, at this point, the pio module is called gpio :) Try your second example with gpio instead of pio, it should work.

Best,
Bogdan

On Wed, Apr 22, 2009 at 3:31 AM, Jesus Alvarez <[hidden email]> wrote:

What is the proper syntax to configure and read an 8-bit port (rather than a
pin) in eLua? I tried both

pio.configport(pio.DIR, pio.DIR_INPUT, "PA")
pio.configport(pio.PULL, pio.PULL_UP, "PA")
io.write("PA = ", pio.getport("PA"), "\n")

and

pio.dir["PA"] = pio.INPUT
pio.pull["PA"] = pio.PULLUP
io.write("PA = ", pio["PA"], "\n")

The first one results in syntax error and the second doesn't read the port.

BTW - configport() and getport() are listed in the documentation directory
(en/refman.html). Were those used in an older version?

Regards,
Jesus Alvarez

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev