What can I do?

[John Hind]

The things I need in the eLua mbed implementation for my own projects are as follows:

 

1.       Ethernet support (not in James’s mbed ‘preliminary’ port at the moment).

2.       Support for USB MSD (flash drives and hard disks). There does not seem to be any USB work in eLua to date, so this is maybe breaking completely new ground. However it looks relatively straightforward to slot this in under the current FAT filesystem support for SD/MMC. Later I will need serial over USB and MSD device emulation as I want to replace the auxiliary JTAG chip on the mbed with an on-chip implementation.

3.       Multi-threading support. Initially I thought of trying to put FreeRTOS or similar in under eLua, but on reflection that is likely to be too profligate with RAM as the Lua RAM overhead will then be per-thread. Then I read Patrick Rapin’s “Enhanced Coroutines in Lua” chapter in “Lua Programming Gems”. Basically this suggests a coroutine scheduler with an enhanced yield method which yields back to the scheduler until some event occurs. For example you could yield for 100ms and the scheduler could resume some other coroutine or run other functions in the main thread during this time. This seems an ideal approach for microcontrollers as it allows a non-blocking wait for interrupt and cooperative multi-threading.

4.       LIN networking support – this should be relatively easy to build on top of the current serial port support, perhaps could even be done in Lua.

 

Is anyone already working on any of the above? I’m guessing at least 1 is already in train – is it likely to deliver within the next quarter? I will be happy to contribute anything I have to do myself back to the project or to help anyone already working on something.

 

 

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Tue, Dec 7, 2010 at 7:21 PM, John Hind <[hidden email]> wrote:
> The things I need in the eLua mbed implementation for my own projects are as
> follows:
>
>
>
> 1.       Ethernet support (not in James’s mbed ‘preliminary’ port at the
> moment).

If you need TCP/IP, eLua needs more than just a port to mbed, a better
TCP/IP stack (or at the best a better integration of the current one)
is needed.

>
> 2.       Support for USB MSD (flash drives and hard disks). There does not
> seem to be any USB work in eLua to date, so this is maybe breaking
> completely new ground. However it looks relatively straightforward to slot
> this in under the current FAT filesystem support for SD/MMC. Later I will
> need serial over USB and MSD device emulation as I want to replace the
> auxiliary JTAG chip on the mbed with an on-chip implementation.
>
> 3.       Multi-threading support. Initially I thought of trying to put
> FreeRTOS or similar in under eLua, but on reflection that is likely to be
> too profligate with RAM as the Lua RAM overhead will then be per-thread.

I beleive you're right.

> Then I read Patrick Rapin’s “Enhanced Coroutines in Lua” chapter in “Lua
> Programming Gems”. Basically this suggests a coroutine scheduler with an
> enhanced yield method which yields back to the scheduler until some event
> occurs. For example you could yield for 100ms and the scheduler could resume
> some other coroutine or run other functions in the main thread during this
> time. This seems an ideal approach for microcontrollers as it allows a
> non-blocking wait for interrupt and cooperative multi-threading.

Man, I really have to get me that book :) However, I believe that an
event-based system would work on the current system too, without the
need to yield every 100ms. Just wait on the coroutine until some event
occurs, and if it happens to be the 100ms timeout event yield back to
the calling thread.

> 4.       LIN networking support – this should be relatively easy to build on
> top of the current serial port support, perhaps could even be done in Lua.

You're probably right about this.

> Is anyone already working on any of the above? I’m guessing at least 1 is
> already in train – is it likely to deliver within the next quarter? I will
> be happy to contribute anything I have to do myself back to the project or
> to help anyone already working on something.

Sorry, I have less and less free time to work on eLua, so I can't
really help here :( I can however provide support if you decide to
write it yourself.

Best,
Bogdan
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Tue, Dec 7, 2010 at 12:52 PM, Bogdan Marinescu
<[hidden email]> wrote:

> On Tue, Dec 7, 2010 at 7:21 PM, John Hind <[hidden email]> wrote:
>> The things I need in the eLua mbed implementation for my own projects are as
>> follows:
>>
>>
>>
>> 1.       Ethernet support (not in James’s mbed ‘preliminary’ port at the
>> moment).
>
> If you need TCP/IP, eLua needs more than just a port to mbed, a better
> TCP/IP stack (or at the best a better integration of the current one)
> is needed.

It could be made to work with the existing uIP module, but I think
what we'd like to do is either switch to lwIP or some combination of a
UDP stack and fuller TCP/IP stack such that in cases where minimal
resources are available the user could just use the UDP stack.

I personally haven't been using ethernet much on eLua so I'm not sure
if/when I'd get to adding the current ethernet module to mbed or if
I'd get around to the more extensive task of working on inserting a
replacement stack.

I'd definitely be willing to at least provide help to anyone who is
interested in improving this area and has less familiarity with eLua
and it's internals.

>>
>> 2.       Support for USB MSD (flash drives and hard disks). There does not
>> seem to be any USB work in eLua to date, so this is maybe breaking
>> completely new ground. However it looks relatively straightforward to slot
>> this in under the current FAT filesystem support for SD/MMC. Later I will
>> need serial over USB and MSD device emulation as I want to replace the
>> auxiliary JTAG chip on the mbed with an on-chip implementation.

I don't think this is will be too hard.  At least for Luminary devices
that provide hardware support for this, there's an example
(usb_host_msc in the StellarisWare distribution) of how to set this up
with FatFs (which we use).  I've thought about playing with this a bit
on the 9B92 that I have, but haven't gotten around to it.  For LPC1768
(what's in the mbed) there's also an example for this (called USBMem
in the lcp17xx cmsis driver lib).

I'd also like to try enabling a CDC (for LM3S there's also a code
example for this), but I haven't gotten around to it because most of
the luminary boards come with an integrated JTAG + UART device.

Adding support on a single platform for any of these would probably be
pretty easy.  Where we might have to put in more thought on would be
what we might want to make generic and part of the eLua core of any of
this and what shape that might take.

>>
>> 3.       Multi-threading support. Initially I thought of trying to put
>> FreeRTOS or similar in under eLua, but on reflection that is likely to be
>> too profligate with RAM as the Lua RAM overhead will then be per-thread.
>
> I beleive you're right.
>
>> Then I read Patrick Rapin’s “Enhanced Coroutines in Lua” chapter in “Lua
>> Programming Gems”. Basically this suggests a coroutine scheduler with an
>> enhanced yield method which yields back to the scheduler until some event
>> occurs. For example you could yield for 100ms and the scheduler could resume
>> some other coroutine or run other functions in the main thread during this
>> time. This seems an ideal approach for microcontrollers as it allows a
>> non-blocking wait for interrupt and cooperative multi-threading.
>
> Man, I really have to get me that book :) However, I believe that an
> event-based system would work on the current system too, without the
> need to yield every 100ms. Just wait on the coroutine until some event
> occurs, and if it happens to be the 100ms timeout event yield back to
> the calling thread.

I'll have to look at this example again. IIRC his example is designed
to work with pthreads and windows threads, but I suspect it wouldn't
be hard to adapt to another, simpler threading API.

Without having re-read the article, the idea here would be that
instead of writing one's own coroutine scheduler in Lua, you would
have a dedicated scheduler, written in C that is provided by the
system.  I suspect that it wouldn't be too complicated for us to do
something like this, especially if the scheduler is quite
simple/stupid like the PIL example: http://www.lua.org/pil/9.4.html

FYI, the lpg sources for this article are here:
http://www.lua.org/gems/lpg124.zip

>
>> 4.       LIN networking support – this should be relatively easy to build on
>> top of the current serial port support, perhaps could even be done in Lua.
>
> You're probably right about this.

Ditto.  I'm not very familiar with this besides quickly looking it up
on Wikipedia :-)  If it doesn't require anything more in terms of UART
configuration than what we already provide, you could probably do it
all in Lua and not even have to touch any C.  Depending on the
application it may make more sense to use the platform abstraction API
from the C side and have that handle receiving and decoding packets.
We've already implemented some protocols on top of serial with decent
success in the form of LuaRPC and RFS, both of which use the platform
API from C.

>
>> Is anyone already working on any of the above? I’m guessing at least 1 is
>> already in train – is it likely to deliver within the next quarter? I will
>> be happy to contribute anything I have to do myself back to the project or
>> to help anyone already working on something.
>
> Sorry, I have less and less free time to work on eLua, so I can't
> really help here :( I can however provide support if you decide to
> write it yourself.

I'm not directly working on any of these right now, but I certainly
have interest in experimenting with #2 & #3.  #1 is interesting, but I
don't have direct need for it on my current projects.

Also, ditto on providing support :-)

>
> Best,
> Bogdan
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>



--
James Snyder
Biomedical Engineering
Northwestern University
[hidden email]
PGP: http://fanplastic.org/key.txt
Phone: [phone]
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[John Hind]

> -----Original Message-----
> From: [hidden email] [mailto:elua-dev-
> [hidden email]] On Behalf Of James Snyder
> Sent: 07 December 2010 23:34

> >> 3.       Multi-threading support.
>
> I'll have to look at this example again. IIRC his example is designed
> to work with pthreads and windows threads, but I suspect it wouldn't
> be hard to adapt to another, simpler threading API.
>
> Without having re-read the article, the idea here would be that
> instead of writing one's own coroutine scheduler in Lua, you would
> have a dedicated scheduler, written in C that is provided by the
> system.  I suspect that it wouldn't be too complicated for us to do
> something like this, especially if the scheduler is quite
> simple/stupid like the PIL example: http://www.lua.org/pil/9.4.html

If we are satisfied with coroutine (cooperative) multitasking, I do not
think any additional threading support would be needed. The thread type
which supports coroutines in standard Lua should be all we need, along with
interrupt handlers (IH) written in C. The IH would set event flags and the
scheduler would use these flags to determine which coroutines are candidates
to be resumed. Rapin's basic idea is to enhance the yield method so it
specifies resume criteria (on serial reception, after 100ms, when a pin
changes state etc.). The scheduler (which as you suggest could be
generalised and written in C) starts all coroutines, so coroutines always
yield back to it. The scheduler could also be able to run Lua functions on
the primary thread, so combining coroutine scheduling with a simple event
handling paradigm (for example to hook up an LED to follow changing states
on an input pin). So the event abstraction comprises a boolean flag and a
list of functions and threads (coroutines). The flag can (optionally) be
connected to a specified "interrupt port" in the runtime. When the flag
becomes set (explicitly from Lua or by an IH), the scheduler executes the
functions and resumes the coroutines sequentially.
Very ambitiously, this scheme could perhaps even be extended to do hard
real-time. All the code segments would be profiled for minimum and maximum
execution time and the scheduler would use real-time critical path analysis
to guarantee specified event response time requirements (or raise an
exception if impossible). For this, the scheduler would have to take over
managing the garbage collector as well.

- John



_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Hi,

> If we are satisfied with coroutine (cooperative) multitasking, I do not
> think any additional threading support would be needed.

I am generally satisfied with that, but of course this varies a lot
with different application types.

> The thread type
> which supports coroutines in standard Lua should be all we need, along with
> interrupt handlers (IH) written in C.

There is generic interrupt support in eLua now for both C and Lua (in
trunk, will pe part of version 0.8 which is to be released soon).
Besides interrupt support, a mechanism for polling different interrupt
flags can be also implemented (think about a "select"-like call that
works on interrupt flags instead of file descriptors). This might be
just what you need in your coroutine. In would block in such a
"select" call waiting for a number of interrupts, one of them being a
timer expired interrupt (and it would yield if it receives this). This
can be extended to support generic events, not only interrupts. I
wouldn't do this "event layer" in C though, sounds more like a job for
the Lua side.

> The IH would set event flags and the
> scheduler would use these flags to determine which coroutines are candidates
> to be resumed. Rapin's basic idea is to enhance the yield method so it
> specifies resume criteria (on serial reception, after 100ms, when a pin
> changes state etc.). The scheduler (which as you suggest could be
> generalised and written in C) starts all coroutines, so coroutines always
> yield back to it. The scheduler could also be able to run Lua functions on
> the primary thread, so combining coroutine scheduling with a simple event
> handling paradigm (for example to hook up an LED to follow changing states
> on an input pin). So the event abstraction comprises a boolean flag and a
> list of functions and threads (coroutines). The flag can (optionally) be
> connected to a specified "interrupt port" in the runtime. When the flag
> becomes set (explicitly from Lua or by an IH), the scheduler executes the
> functions and resumes the coroutines sequentially.

I believe this can be implemented in eLua even now with a bit of
coroutine/metamethod magic :) Except, of course, it will still be
cooperative.

> Very ambitiously, this scheme could perhaps even be extended to do hard
> real-time. All the code segments would be profiled for minimum and maximum
> execution time and the scheduler would use real-time critical path analysis
> to guarantee specified event response time requirements (or raise an
> exception if impossible). For this, the scheduler would have to take over
> managing the garbage collector as well.

Sure, everything is possible :) I genereally don't believe in hard
real-time systems that are written in an interpreted language, most of
the time they prove to be more or less soft real-time in my
experience. That's not to say that it can't be done though.

Best,
Bogdan
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[John Hind]

> -----Original Message-----
> From: [hidden email] [mailto:elua-dev-
> [hidden email]] On Behalf Of Bogdan Marinescu
> Sent: 08 December 2010 11:11
>
> > If we are satisfied with coroutine (cooperative) multitasking, I do
> not
> > think any additional threading support would be needed.
>
> I am generally satisfied with that, but of course this varies a lot
> with different application types.
>

I'm not really, but when we're measuring RAM in KB I think we have to be!
Even Apple runs into RAM problems with pre-emptive multitasking!

> There is generic interrupt support in eLua now for both C and Lua (in
> trunk, will pe part of version 0.8 which is to be released soon).

I did spot a brief reference to that in the User Labs WiKi, but could not
find anything more about it - I'll have a poke around in the source code.
I'll not ask you to define "soon" - I've found that can be a dangerous
question to ask open source developers!

> Besides interrupt support, a mechanism for polling different interrupt
> flags can be also implemented (think about a "select"-like call that
> works on interrupt flags instead of file descriptors). This might be
> just what you need in your coroutine. In would block in such a
> "select" call waiting for a number of interrupts, one of them being a
> timer expired interrupt (and it would yield if it receives this). This
> can be extended to support generic events, not only interrupts. I
> wouldn't do this "event layer" in C though, sounds more like a job for
> the Lua side.
>

I do not think this will work - even if the "select" call is non-blocking,
unless the coroutine yields no more Lua code can execute. We need to arrange
it such that the only code that may block is the scheduler - when it has no
more code ready to run it can block or busy-wait for a new event. The timer
expired call I mentioned in the original message was only an example of
using an enhanced coroutine yield to implement a non-blocking event wait -
it could have been any interrupt event.

Most of what I am suggesting could be done either in C or in Lua, but ISRs
have to be C and there has to be a way for ISR code to set a flag which can
be asynchronously read in Lua, on the fly without blocking. Asynchronous
events must be generated by an ISR (and therefore in C code), though they
would not be limited to the interrupt flags specified by the hardware (for
example in serial comms data can be buffered by an ISR until a specified
packet termination criterion is met and then a flag is set in RAM to trigger
the scheduler to run some Lua code). It might be useful to allow events to
be raised in Lua, but these would necessarily be synchronous events
conceptually equivalent to simple Boolean globals.

- Best regards,
John.


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

> I did spot a brief reference to that in the User Labs WiKi, but could not
> find anything more about it - I'll have a poke around in the source code.
> I'll not ask you to define "soon" - I've found that can be a dangerous
> question to ask open source developers!

The full docs are already in trunk (doc/en/inthandlers.txt). And yes,
that is a dangerous question to ask, especially with the holidays
around the corner :) We plan to have something around Christmas, but
this might have been already delayed with me being busier than usual
at work and the other guys having their own set of time consuming
tasks.

> I do not think this will work - even if the "select" call is non-blocking,
> unless the coroutine yields no more Lua code can execute.

Ah OK, we were talking about different things then. My idea was that
the coroutine was supposed to block for 100ms (for example) even if it
didn't have anything to do, then yield (thus partially scheduling
itself). Cooperative multitasking with time slots. You're right that
in this time no more Lua code can execute (except for Lua interrupt
handlers).

> We need to arrange
> it such that the only code that may block is the scheduler - when it has no
> more code ready to run it can block or busy-wait for a new event. The timer
> expired call I mentioned in the original message was only an example of
> using an enhanced coroutine yield to implement a non-blocking event wait -
> it could have been any interrupt event.

I think I understood now. You are indeed looking for something that
should be implemented on the C side.

> Most of what I am suggesting could be done either in C or in Lua, but ISRs
> have to be C and there has to be a way for ISR code to set a flag which can
> be asynchronously read in Lua, on the fly without blocking.

If you need a flag in Lua wouldn't it be easier to have the interrupt
handler directly in Lua? Currently interrupt support in Lua is done
using standard Lua hooks, so as long as the VM is running you will get
to the interrupt handler no matter what. Of course the actual
interrupt handlers are written in C and the C code puts them in a
queue that is later examined by the Lua hook function :) but you might
not need to be disturbed with such details and use the Lua handlers
directly.

> Asynchronous
> events must be generated by an ISR (and therefore in C code), though they
> would not be limited to the interrupt flags specified by the hardware (for
> example in serial comms data can be buffered by an ISR until a specified
> packet termination criterion is met and then a flag is set in RAM to trigger
> the scheduler to run some Lua code).

Yes, a software interrupt-on-UART-char-match (which is an instance of
what you described above) is already planned. Very useful for tasks
like reading sentences from a GPS receiver when an interrupt could be
generated when NL is received and the code could simply read the
sentence and parse it. There's a ton of cool stuff that can be done
with these "software interrupts" (which are actually semantically
similar to events).

> It might be useful to allow events to
> be raised in Lua, but these would necessarily be synchronous events
> conceptually equivalent to simple Boolean globals.

Yes, they are already asynchronous, as explained. The only real
limitation right now is the "VM must run" one. I don't know how to fix
this without multithreading support (actual threads, not Lua threads),
I'm not even sure it can be done. If the current model proves to be
too simple for practical uses I'll probably have to add a simple
threading library to eLua. But this is something I'll do only when I'm
convinced that there aren't other practical solutions to try.

Best,
Bogdan
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[John Hind]

> -----Original Message-----
> From: [hidden email] [mailto:elua-dev-
> [hidden email]] On Behalf Of Bogdan Marinescu
> Sent: 08 December 2010 12:43


> Yes, they are already asynchronous, as explained. The only real
> limitation right now is the "VM must run" one. I don't know how to fix
> this without multithreading support (actual threads, not Lua threads),
> I'm not even sure it can be done. If the current model proves to be
> too simple for practical uses I'll probably have to add a simple
> threading library to eLua. But this is something I'll do only when I'm
> convinced that there aren't other practical solutions to try.

Ah! I understand now - you have already done something similar to what I am
suggesting. However I would not describe this as asynchronous on the Lua
side. It is only asynchronous if the Lua interpreter is idle when the
interrupt occurs. Otherwise the queue effectively serialises the code to run
synchronously after the current code terminates. My scheme is similar except
it can also have coroutines as well as functions linked to ISRs and the
scheduler/serialiser/queue is differently implemented. Coroutines would be
useful for implementing state machines and DSP algorithims for processing
data streams for example.

What I got from Patrick Rapin was the insight that coroutine yields could
replace blocking waits and polls. This allows a very familiar coding style
were the blocking waits/polls of naive code are simply replaced by
non-blocking yields. You could do the same thing with functions hooked to
ISRs using global variables to maintain state between function calls, but
that demands a very different, more sophisticated, coding style.

You could regard this as an "other practical solution to try"!

- John


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

> Ah! I understand now - you have already done something similar to what I am
> suggesting. However I would not describe this as asynchronous on the Lua
> side.

They are asynchronous in the sense that they can interrupt the regular
Lua program flow at any time and execute another function (the
interrupt handler) instead.

> It is only asynchronous if the Lua interpreter is idle when the
> interrupt occurs. Otherwise the queue effectively serialises the code to run
> synchronously after the current code terminates. My scheme is similar except
> it can also have coroutines as well as functions linked to ISRs and the
> scheduler/serialiser/queue is differently implemented. Coroutines would be
> useful for implementing state machines and DSP algorithims for processing
> data streams for example.
>
> What I got from Patrick Rapin was the insight that coroutine yields could
> replace blocking waits and polls. This allows a very familiar coding style
> were the blocking waits/polls of naive code are simply replaced by
> non-blocking yields. You could do the same thing with functions hooked to
> ISRs using global variables to maintain state between function calls, but
> that demands a very different, more sophisticated, coding style.
>
> You could regard this as an "other practical solution to try"!

I must confess that I don't fully understand this model yet. Oh well,
another reason for me to order that book :) I should've done it a long
time ago anyway.

Best,
Bogdan
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[John Hind]

> -----Original Message-----
> From: [hidden email] [mailto:elua-dev-
> [hidden email]] On Behalf Of Bogdan Marinescu
> Sent: 08 December 2010 14:22


> They are asynchronous in the sense that they can interrupt the regular
> Lua program flow at any time and execute another function (the
> interrupt handler) instead.

Surely not? Can you interrupt Lua code at any point, run a different Lua
function and then return to the original Lua code where it left off? All in
the same Lua State? I understand if you mean run a C interrupt handler
asynchronously and then resume the interrupted Lua code. Point is that the
Lua programmer cannot write a true asynchronous interrupt handler, only poll
"pre-cooked" ones provided in the runtime. Any Lua code run as a result of
an interrupt has to be serialised and can only run when the currently
running code terminates. Or have I missed a trick?

> I must confess that I don't fully understand this model yet. Oh well,
> another reason for me to order that book :) I should've done it a long
> time ago anyway.

Check your gmail - there should be a nice surprise! Rupin's approach is
designed for operation in a multi-tasking operating system using native
threads to run pre-cooked C routines. I'm suggesting these routines could be
substituted by ISR code on a bare metal microcontroller implementation. The
key point is that an enhanced "yield" method specifies to the scheduler the
conditions under which the coroutine should be resumed.

- John


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Wed, Dec 8, 2010 at 6:54 PM, John Hind <[hidden email]> wrote:

>> -----Original Message-----
>> From: [hidden email] [mailto:elua-dev-
>> [hidden email]] On Behalf Of Bogdan Marinescu
>> Sent: 08 December 2010 14:22
>
>
>> They are asynchronous in the sense that they can interrupt the regular
>> Lua program flow at any time and execute another function (the
>> interrupt handler) instead.
>
> Surely not? Can you interrupt Lua code at any point, run a different Lua
> function and then return to the original Lua code where it left off? All in
> the same Lua State?

You can interrupt it at almost any point using lua_sethook:

http://pgl.yoyo.org/luai/i/lua_sethook

This is what the Lua interrupt support is currently using: a count
hook (LUA_MASKCOUNT) with count = 2 (to give the "regular" Lua code a
chance to execute even if a "storm" of interrupts is received). So
every 2 Lua VM instructions a hook function is executed, which in turn
checks if any interrupt is queued and calls the corresponding handler
if it is (src/elua_int.c:elua_int_hook).

> I understand if you mean run a C interrupt handler
> asynchronously and then resume the interrupted Lua code. Point is that the
> Lua programmer cannot write a true asynchronous interrupt handler, only poll
> "pre-cooked" ones provided in the runtime. Any Lua code run as a result of
> an interrupt has to be serialised and can only run when the currently
> running code terminates. Or have I missed a trick?

The one above probably :)

>> I must confess that I don't fully understand this model yet. Oh well,
>> another reason for me to order that book :) I should've done it a long
>> time ago anyway.
>
> Check your gmail - there should be a nice surprise!

Thank you very much for the surprise!

> Rupin's approach is
> designed for operation in a multi-tasking operating system using native
> threads to run pre-cooked C routines. I'm suggesting these routines could be
> substituted by ISR code on a bare metal microcontroller implementation. The
> key point is that an enhanced "yield" method specifies to the scheduler the
> conditions under which the coroutine should be resumed.

The problem here is that we don't have native threads yet. I was
trying to avoid adding an OS to eLua, even a minimal one, but I'm
currently reconsidering that position.

Best,
Bogdan
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Wed, Dec 8, 2010 at 4:02 PM, Bogdan Marinescu
<[hidden email]> wrote:

> On Wed, Dec 8, 2010 at 6:54 PM, John Hind <[hidden email]> wrote:
>>> -----Original Message-----
>>> From: [hidden email] [mailto:elua-dev-
>>> [hidden email]] On Behalf Of Bogdan Marinescu
>>> Sent: 08 December 2010 14:22
>>
>>
>>> They are asynchronous in the sense that they can interrupt the regular
>>> Lua program flow at any time and execute another function (the
>>> interrupt handler) instead.
>>
>> Surely not? Can you interrupt Lua code at any point, run a different Lua
>> function and then return to the original Lua code where it left off? All in
>> the same Lua State?
>
> You can interrupt it at almost any point using lua_sethook:
>
> http://pgl.yoyo.org/luai/i/lua_sethook
>
> This is what the Lua interrupt support is currently using: a count
> hook (LUA_MASKCOUNT) with count = 2 (to give the "regular" Lua code a
> chance to execute even if a "storm" of interrupts is received). So
> every 2 Lua VM instructions a hook function is executed, which in turn
> checks if any interrupt is queued and calls the corresponding handler
> if it is (src/elua_int.c:elua_int_hook).
>
>> I understand if you mean run a C interrupt handler
>> asynchronously and then resume the interrupted Lua code. Point is that the
>> Lua programmer cannot write a true asynchronous interrupt handler, only poll
>> "pre-cooked" ones provided in the runtime. Any Lua code run as a result of
>> an interrupt has to be serialised and can only run when the currently
>> running code terminates. Or have I missed a trick?
>
> The one above probably :)
>
>>> I must confess that I don't fully understand this model yet. Oh well,
>>> another reason for me to order that book :) I should've done it a long
>>> time ago anyway.
>>
>> Check your gmail - there should be a nice surprise!
>
> Thank you very much for the surprise!
>
>> Rupin's approach is
>> designed for operation in a multi-tasking operating system using native
>> threads to run pre-cooked C routines. I'm suggesting these routines could be
>> substituted by ISR code on a bare metal microcontroller implementation. The
>> key point is that an enhanced "yield" method specifies to the scheduler the
>> conditions under which the coroutine should be resumed.
>
> The problem here is that we don't have native threads yet. I was
> trying to avoid adding an OS to eLua, even a minimal one, but I'm
> currently reconsidering that position.

I think we can get away with not using real stack-needing threads for
things like this.  I'm jumping back in to this conversation more than
a few steps later than when I last chimed in, but I think there might
be a way to meld these two concepts together.

With eLua ISRs there isn't an explicit scheduler, just a queue that
gets examined every two VM instructions.  This provides a coarse
grained pre-emptive model for handling ISRs. This could be adapted to
provide some of the functionality that John is describing by allowing
for different types of events to trigger Lua ISR calls.  There is also
the difference that John/Rupin's version makes use of coroutines
instead of functions to allow for a given "thread" of execution to
have points where it yields control, but expects to be resumed at a
later time.  This has an advantage over pure function handlers in that
when yielding the coroutine has its scope saved, which will come back
when it gets resumed.  I presume you could get some of this by using
closures with the current model, but there is something to be said for
mixing these concepts together together so that if one knows a given
thread won't be able to do something until new data comes in, it would
be wasteful to have it sit in a loop polling for something and have
that be pre-empted in order for anything else to happen, when instead
you could have the ISR facility allow a coroutine to "yield" or
"select" by asking the ISR facility to resume it when a given event
comes in.

Unless I've completely glossed over things, this would seem to provide
the benefits of pre-emptive ISRs and select-like behavior under a
single model?  Of course the Lua ISR system would need to be adapted
to not only handle functions, but do the right thing with coroutines.

>
> Best,
> Bogdan
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>



--
James Snyder
Biomedical Engineering
Northwestern University
[hidden email]
PGP: http://fanplastic.org/key.txt
Phone: [phone]
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[John Hind]

> -----Original Message-----
> From: [hidden email] [mailto:elua-dev-
> [hidden email]] On Behalf Of Bogdan Marinescu
> Sent: 08 December 2010 22:03

> You can interrupt it at almost any point using lua_sethook:
>
> http://pgl.yoyo.org/luai/i/lua_sethook
>
> This is what the Lua interrupt support is currently using: a count
> hook (LUA_MASKCOUNT) with count = 2 (to give the "regular" Lua code a
> chance to execute even if a "storm" of interrupts is received). So
> every 2 Lua VM instructions a hook function is executed, which in turn
> checks if any interrupt is queued and calls the corresponding handler
> if it is (src/elua_int.c:elua_int_hook).
>

Wow! I have used the debug hook function before, but never thought to use it
as an interrupt mechanism in normal program execution. This suggests that a
coroutine could be asynchronously forced to yield by an external event (just
test a flag in the hook function and execute a yield if it is set). You have
officially blown my mind: I will do some experimenting with this in my
Windows runtime so I can get a decent 'C' debugger on the job.




_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[John Hind]

> -----Original Message-----
> From: [hidden email] [mailto:elua-dev-
> [hidden email]] On Behalf Of James Snyder
> Sent: 09 December 2010 00:48
>
> I think we can get away with not using real stack-needing threads for
> things like this.  I'm jumping back in to this conversation more than
> a few steps later than when I last chimed in, but I think there might
> be a way to meld these two concepts together.
>
> With eLua ISRs there isn't an explicit scheduler, just a queue that
> gets examined every two VM instructions.  This provides a coarse
> grained pre-emptive model for handling ISRs. This could be adapted to
> provide some of the functionality that John is describing by allowing
> for different types of events to trigger Lua ISR calls.  There is also
> the difference that John/Rupin's version makes use of coroutines
> instead of functions to allow for a given "thread" of execution to
> have points where it yields control, but expects to be resumed at a
> later time.  This has an advantage over pure function handlers in that
> when yielding the coroutine has its scope saved, which will come back
> when it gets resumed.  I presume you could get some of this by using
> closures with the current model, but there is something to be said for
> mixing these concepts together together so that if one knows a given
> thread won't be able to do something until new data comes in, it would
> be wasteful to have it sit in a loop polling for something and have
> that be pre-empted in order for anything else to happen, when instead
> you could have the ISR facility allow a coroutine to "yield" or
> "select" by asking the ISR facility to resume it when a given event
> comes in.
>
> Unless I've completely glossed over things, this would seem to provide
> the benefits of pre-emptive ISRs and select-like behavior under a
> single model?  Of course the Lua ISR system would need to be adapted
> to not only handle functions, but do the right thing with coroutines.

I think you are right - plus I just realised that this mechanism could be
used to force a coroutine to yield asynchronously so we could use it for
full pre-emptive multitasking within a single Lua State! (Albeit as you say
"course grained".) Given that the Lua VM allows you to take this kind of
liberty using debug hooks, it ought to be feasible to patch it to do this
in-line without the performance overhead.

I will do some experimenting with this on my Visual Studio compiled Windows
runtime.



_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Thu, Dec 9, 2010 at 2:52 PM, John Hind <[hidden email]> wrote:

>> -----Original Message-----
>> From: [hidden email] [mailto:elua-dev-
>> [hidden email]] On Behalf Of James Snyder
>> Sent: 09 December 2010 00:48
>>
>> I think we can get away with not using real stack-needing threads for
>> things like this.  I'm jumping back in to this conversation more than
>> a few steps later than when I last chimed in, but I think there might
>> be a way to meld these two concepts together.
>>
>> With eLua ISRs there isn't an explicit scheduler, just a queue that
>> gets examined every two VM instructions.  This provides a coarse
>> grained pre-emptive model for handling ISRs. This could be adapted to
>> provide some of the functionality that John is describing by allowing
>> for different types of events to trigger Lua ISR calls.  There is also
>> the difference that John/Rupin's version makes use of coroutines
>> instead of functions to allow for a given "thread" of execution to
>> have points where it yields control, but expects to be resumed at a
>> later time.  This has an advantage over pure function handlers in that
>> when yielding the coroutine has its scope saved, which will come back
>> when it gets resumed.  I presume you could get some of this by using
>> closures with the current model, but there is something to be said for
>> mixing these concepts together together so that if one knows a given
>> thread won't be able to do something until new data comes in, it would
>> be wasteful to have it sit in a loop polling for something and have
>> that be pre-empted in order for anything else to happen, when instead
>> you could have the ISR facility allow a coroutine to "yield" or
>> "select" by asking the ISR facility to resume it when a given event
>> comes in.
>>
>> Unless I've completely glossed over things, this would seem to provide
>> the benefits of pre-emptive ISRs and select-like behavior under a
>> single model?  Of course the Lua ISR system would need to be adapted
>> to not only handle functions, but do the right thing with coroutines.
>
> I think you are right - plus I just realised that this mechanism could be
> used to force a coroutine to yield asynchronously so we could use it for
> full pre-emptive multitasking within a single Lua State!

I'm not 100% sure but this might require the coco patch. Not a big
issue, just increased RAM consumption.

> (Albeit as you say
> "course grained".) Given that the Lua VM allows you to take this kind of
> liberty using debug hooks, it ought to be feasible to patch it to do this
> in-line without the performance overhead.
>
> I will do some experimenting with this on my Visual Studio compiled Windows
> runtime.

Excellent! Please let us know your results, I'm also interested in this.

Best,
Bogdan
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On Thu, Dec 9, 2010 at 7:01 AM, Bogdan Marinescu
<[hidden email]> wrote:

> On Thu, Dec 9, 2010 at 2:52 PM, John Hind <[hidden email]> wrote:
>>> -----Original Message-----
>>> From: [hidden email] [mailto:elua-dev-
>>> [hidden email]] On Behalf Of James Snyder
>>> Sent: 09 December 2010 00:48
>>>
>>> I think we can get away with not using real stack-needing threads for
>>> things like this.  I'm jumping back in to this conversation more than
>>> a few steps later than when I last chimed in, but I think there might
>>> be a way to meld these two concepts together.
>>>
>>> With eLua ISRs there isn't an explicit scheduler, just a queue that
>>> gets examined every two VM instructions.  This provides a coarse
>>> grained pre-emptive model for handling ISRs. This could be adapted to
>>> provide some of the functionality that John is describing by allowing
>>> for different types of events to trigger Lua ISR calls.  There is also
>>> the difference that John/Rupin's version makes use of coroutines
>>> instead of functions to allow for a given "thread" of execution to
>>> have points where it yields control, but expects to be resumed at a
>>> later time.  This has an advantage over pure function handlers in that
>>> when yielding the coroutine has its scope saved, which will come back
>>> when it gets resumed.  I presume you could get some of this by using
>>> closures with the current model, but there is something to be said for
>>> mixing these concepts together together so that if one knows a given
>>> thread won't be able to do something until new data comes in, it would
>>> be wasteful to have it sit in a loop polling for something and have
>>> that be pre-empted in order for anything else to happen, when instead
>>> you could have the ISR facility allow a coroutine to "yield" or
>>> "select" by asking the ISR facility to resume it when a given event
>>> comes in.
>>>
>>> Unless I've completely glossed over things, this would seem to provide
>>> the benefits of pre-emptive ISRs and select-like behavior under a
>>> single model?  Of course the Lua ISR system would need to be adapted
>>> to not only handle functions, but do the right thing with coroutines.
>>
>> I think you are right - plus I just realised that this mechanism could be
>> used to force a coroutine to yield asynchronously so we could use it for
>> full pre-emptive multitasking within a single Lua State!
>
> I'm not 100% sure but this might require the coco patch. Not a big
> issue, just increased RAM consumption.

I'd have to think this through a bit further to be sure whether coco
is needed.  That patch adds full stackful coroutines but I suspect
what you're referring to here are the added abilities to yield in
situations where the stock Lua won't. 5.2 adds some of this
functionality as well, perhaps without the overhead of needing the
extra stack spaces? .  This might present a good reason for us to look
at 5.2 once there is a stable release :-)

In any case, coco does work just fine on eLua, there's a branch which
is a bit old but it definitely worked on the ARM targets I tested.
Each platform needs a little added support for patching the setjmp
buffer (the branch has this for at least newlib/thumb2), which usually
involves looking at the platform implementation of setjmp.

>
>> (Albeit as you say
>> "course grained".) Given that the Lua VM allows you to take this kind of
>> liberty using debug hooks, it ought to be feasible to patch it to do this
>> in-line without the performance overhead.
>>
>> I will do some experimenting with this on my Visual Studio compiled Windows
>> runtime.
>
> Excellent! Please let us know your results, I'm also interested in this.
>
> Best,
> Bogdan
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>

--
James Snyder
Biomedical Engineering
Northwestern University
[hidden email]
PGP: http://fanplastic.org/key.txt
Phone: [phone]
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[John Hind]

> -----Original Message-----
> From: [hidden email] [mailto:elua-dev-
> [hidden email]] On Behalf Of James Snyder
> Sent: 09 December 2010 19:06


> I'd have to think this through a bit further to be sure whether coco
> is needed.  That patch adds full stackful coroutines but I suspect
> what you're referring to here are the added abilities to yield in
> situations where the stock Lua won't. 5.2 adds some of this
> functionality as well, perhaps without the overhead of needing the
> extra stack spaces? .  This might present a good reason for us to look
> at 5.2 once there is a stable release :-)

I've investigated some more and I think the way to proceed is with an
interpreter patch similar to Mark Tomczak's "Interpreter Bailout Flag".
There will be a flag in lua_State and when this is set, the interpreter will
check if we are yieldable before each op code and if so, call lua_yield. I
can copy the not-yieldable test (which is very simple) that normally raises
an exception in lua_yield. Probably best to assign the flag in a byte and
include Tomczak's bailout functionality (which excecutes RETURN opcodes
repeatedly to cleanly terminate the Lua chunk) as another flag in the same
byte. This way coco would not be needed and yes, the changes in 5.2 do look
very interesting in this respect - also there are hints that (Lua) threads
may be lighter on RAM as well!

The idea is that ISR's will call a scheduler routine which will evaluate
whether the currently running coroutine is still highest priority, if not
the flag is set to force the running coroutine to yield ASAP. Rupin's method
will also be available so a coroutine can voluntarily yield to wait on an
interrupt event, and the next highest priority coroutine or function which
is in a runnable state is then resumed or called. Interrupt-linked functions
will be lighter in RAM than coroutines, but will not be preemptable - once
started they will run to completion (as in the present system - I think?) so
they need to be relatively short.

-John


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev