Multitasking Proposal
Locked
|
Hi Everybody, I have just published a proposal paper here: http://tracker.eluaproject.net/projects/elua/wiki/PreemptiveCoroutinesProposal outlining my ideas (standing on the shoulders of giants) on how preemptive multitasking could be provided in eLua. Please read and comment and particularly criticise any flaws or weaknesses I may have missed. I also believe this provides an alternative approach which ought to be considered before the INT module (on which my proposal builds) is set in stone within eLua. Best regards, John Hind. _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
On Mon, Jan 31, 2011 at 6:01 PM, John Hind <[hidden email]> wrote:
> Hi Everybody, > > > > I have just published a proposal paper here: > > > > http://tracker.eluaproject.net/projects/elua/wiki/PreemptiveCoroutinesProposal > > > > outlining my ideas (standing on the shoulders of giants) on how preemptive > multitasking could be provided in eLua. Please read and comment and > particularly criticise any flaws or weaknesses I may have missed. I also > believe this provides an alternative approach which ought to be considered > before the INT module (on which my proposal builds) is set in stone within > eLua. Thank you very much for your effort and interesnt John I promise I'll take a good look at it and come back with feedback. But right now we're quite busy with launching eLua 0.8 so it might take a bit more until I get a chance to do that. Best, Bogdan _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by John Hind
I would have added this to the Discussion Page if I knew how to.
This seems like a good idea.
When I first red the proposal, I didn’t understand the problem. The “big loop” approach seemed fine until I realized this is really about interrupt handling. Of course, it could be used in a broader sense. At first, I was thinking why add a level of indirection on a microcontroller platform. I was stuck on the word multitasking. I understand now. I haven’t followed all your implementation details as of yet. Will the locking scheme prevent deadlocks? Is it possible to just use one “big” lock that protects global state? From: [hidden email] [[hidden email]] On Behalf Of John Hind [[hidden email]] Sent: Monday, January 31, 2011 11:01 AM To: [hidden email] Subject: [eLua-dev] Multitasking Proposal Hi Everybody,
I have just published a proposal paper here:
http://tracker.eluaproject.net/projects/elua/wiki/PreemptiveCoroutinesProposal
outlining my ideas (standing on the shoulders of giants) on how preemptive multitasking could be provided in eLua. Please read and comment and particularly criticise any flaws or weaknesses I may have missed. I also believe this provides an alternative approach which ought to be considered before the INT module (on which my proposal builds) is set in stone within eLua.
Best regards,
John Hind.
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is confidential or proprietary to K&L Microwave, Inc. Any unauthorized review, use, disclosure or distribution is prohibited. If this communication has been transmitted from a U.S. location it may also contain data subject to the International Traffic in Arms Regulations or U.S. Export Administration Regulations and, if so, cannot be disseminated, distributed or copied to foreign nationals, residing in the U.S. or abroad, without the prior approval of the U.S. Department of State or appropriate export licensing authority. If you have received this communication in error, please notify the sender by reply e-mail and delete or destroy all copies of this e-mail message and/or any file attachment(s). _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
Thanks for your comments and time, Mike. You need to register and then ask one of the admins to add you to the project before you can edit the WiKi. I had to ask - the system we are using is not very helpful. You also have to be a bit sneaky to work out how to add a page as the index pages are not editable even when you are authorised! The locking scheme *should* be mostly automatic and deadlock free. I included an explicit, Lua level lock only for 'unusual' situations where there are application level data dependencies in Lua (such as a linked list or ring buffer with pointers held to entries in a table). If one coroutine takes the lock (it is one global lock) and does not release it, the asynchronous yield will remain blocked. Probably, the lock should be forced off each time context is switched and perhaps an exception raised if it is on when a coroutine explicitly yields. There is also a separate, fully automatic, lock to protect Lua's internal data structures and it should be impossible for that to deadlock (if I have coded it right). From: [hidden email] [mailto:[hidden email]] On Behalf Of King, Mike I would have added this to the Discussion Page if I knew how to. This seems like a good idea.
From: [hidden email] [[hidden email]] On Behalf Of John Hind [[hidden email]] Hi Everybody, I have just published a proposal paper here: http://tracker.eluaproject.net/projects/elua/wiki/PreemptiveCoroutinesProposal outlining my ideas (standing on the shoulders of giants) on how preemptive multitasking could be provided in eLua. Please read and comment and particularly criticise any flaws or weaknesses I may have missed. I also believe this provides an alternative approach which ought to be considered before the INT module (on which my proposal builds) is set in stone within eLua. Best regards, John Hind. CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is confidential or proprietary to K&L Microwave, Inc. Any unauthorized review, use, disclosure or distribution is prohibited. If this communication has been transmitted from a U.S. location it may also contain data subject to the International Traffic in Arms Regulations or U.S. Export Administration Regulations and, if so, cannot be disseminated, distributed or copied to foreign nationals, residing in the U.S. or abroad, without the prior approval of the U.S. Department of State or appropriate export licensing authority. If you have received this communication in error, please notify the sender by reply e-mail and delete or destroy all copies of this e-mail message and/or any file attachment(s). _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by John Hind
Hi,
we (Sierra Wireless) have quite some experience running a customized Lua VM on embedded devices, although our hardware is slightly more powerful than the typical eLua target (several hundreds KB of RAM). We enjoy the development speed, the flexibility, and ease of porting between different OSes.
Among others, we use Lua-level multitasking extensively, to get concurrency without any assumption about OS-level threading. We take the collaborative approach; in practice, we almost never have any issue with threads not yielding fast enough: all our I/O operations yield, and embedded code spends most of its time doing I/Os anyway.
Given the maintenance issues associated with task preemption (to paraphrase Roberto, you shouldn't trust a language whose interpretation of `n=n+1' isn't deterministic), and the lack of pathologically uncollaborative tasks we've experienced in the real life, I wonder how common the need for coroutine preemption might be. I know it's clearly not worth it at all for us.
Unsurprisingly, our API is similar to yours: - sched.run (func, args...) to start a thread - sched.signal (emitter, event, args...) to notify a noteworthy event - sched.wait (emitters, events) to block on signals
- sched.kill (task) - sched.sighook (emitters, events, hookfunction) to react immediately to a signal (no blocking calls allowed in the hook). The generic Lua part of our code is MIT-licenced (https://svn.anyware-tech.com/wavecom/openat-lua/branches/oatlua4), although some important lower level bricks aren't (TCP, SSL etc.).
-- Fabien _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by BogdanM
Hello,
On Mon, Jan 31, 2011 at 15:05, Bogdan Marinescu <[hidden email]> wrote:
Hi John,
x2 :) And thank you very much for important and well organized work on this. We should be able to handle it sometime soon now, after (finally !:) releasing v0.8 and taking a little time to breath again after that :) Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Fabien
I kind of have sympathy with this. I agree that any asynchronous yield should be an optional feature. Roberto's hellgate of non-determinism has already kind of been opened in eLua with the INT module and, it could be argued, by the debug hooks in baseline Lua. Personally I have never had a problem with shared memory multitasking, but it is true you can easily get in a subtle mess if you do not follow the rules. I have not entirely convinced myself that we are 'following the rules' with this kind of VM code splicing either in my asynchronous yield or (much more so) the system used in the INT module. Your system, which is essentially Patrick Rapin's system as I referenced, and therefore my suggestion without the asynchronous yield, is the important part. However you cut it, we need non-blocking, interrupt based I/O and particularly a non-blocking time delay. Your ability to hook a Lua function to an interrupt generated event so it gets queued for execution synchronously during the next non-blocking wait is also a good additional feature since it avoids the RAM overhead of a coroutine in some simpler cases. From: [hidden email] [mailto:[hidden email]] On Behalf Of Fabien Hi, we (Sierra Wireless) have quite some experience running a customized Lua VM on embedded devices, although our hardware is slightly more powerful than the typical eLua target (several hundreds KB of RAM). We enjoy the development speed, the flexibility, and ease of porting between different OSes. Among others, we use Lua-level multitasking extensively, to get concurrency without any assumption about OS-level threading. We take the collaborative approach; in practice, we almost never have any issue with threads not yielding fast enough: all our I/O operations yield, and embedded code spends most of its time doing I/Os anyway. Given the maintenance issues associated with task preemption (to paraphrase Roberto, you shouldn't trust a language whose interpretation of `n=n+1' isn't deterministic), and the lack of pathologically uncollaborative tasks we've experienced in the real life, I wonder how common the need for coroutine preemption might be. I know it's clearly not worth it at all for us. Unsurprisingly, our API is similar to yours: - sched.run (func, args...) to start a thread - sched.signal (emitter, event, args...) to notify a noteworthy event - sched.wait (emitters, events) to block on signals - sched.kill (task) - sched.sighook (emitters, events, hookfunction) to react immediately to a signal (no blocking calls allowed in the hook). The generic Lua part of our code is MIT-licenced (https://svn.anyware-tech.com/wavecom/openat-lua/branches/oatlua4), although some important lower level bricks aren't (TCP, SSL etc.). -- Fabien _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by King, Mike
Hello list,
On Tue, Feb 1, 2011 at 00:44, King, Mike <[hidden email]> wrote:
Mike, pls register on our tracker at http://tracker.eluaproject.net, send me your LoginID (in pvt if you prefer) and I'll add you as a developer with full access to editions.
The final INT handling (C & Lua) is part of the discussions here too, you're right, because it will be used for the implementation but the main goal of the discussions are preemptive multitasking.
Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by King, Mike
On Tue, Feb 1, 2011 at 07:43, John Hind <[hidden email]> wrote:
Of course I'll be glad to change anything (roles, status, ...) you need to have full access to edit those pages. Again, we can also move this to our MoinMoin user-wiki at http://wiki.eluaproject.net, which is much easier and powerful to edit, although we don't need fancy features for this brainstorming (and the reason why John chosed to keep it in the tracker was probably because it is much more related to the core development than to user or community oriented "project".)
Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Dado Sutter
Thanks Dado - I realised my timing was poor when I became exposed to the torrent of traffic from the tracker project. The engines of issue clearance are certainly churning impressively right now! With all this hard work, 0.8 cannot be too far off! From: [hidden email] [mailto:[hidden email]] On Behalf Of Dado Sutter Hello, On Mon, Jan 31, 2011 at 15:05, Bogdan Marinescu <[hidden email]> wrote: On Mon, Jan 31, 2011 at 6:01 PM, John Hind <[hidden email]> wrote: > Hi Everybody, > > > > I have just published a proposal paper here: > > > > http://tracker.eluaproject.net/projects/elua/wiki/PreemptiveCoroutinesProposal > > > > outlining my ideas (standing on the shoulders of giants) on how preemptive > multitasking could be provided in eLua. Please read and comment and > particularly criticise any flaws or weaknesses I may have missed. I also > believe this provides an alternative approach which ought to be considered > before the INT module (on which my proposal builds) is set in stone within > eLua. Hi John,
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Fabien
Hello,
On Tue, Feb 1, 2011 at 08:34, Fabien <[hidden email]> wrote: Hi, Hi Fabien. It is great to have you in this thread too, as we all know you've been playing with a (proprietary) RTOS embedded in your GPRS Modems where you embedded Lua and they are successful commercial products already. For those who didn't know, Fabien designed the sw for previous Wavecom modems, before they were bought by the giant Sierra Wireless.
Yes, I've mentioned this in an early message. While having preemptive multitasking can be great for the growth of our project and for the maturity of many products, cooperative cooroutines will always be easier to use, are natively implemented (by Lua), are infinitely easier to debug and, if used correctly, are enough for a lot of problems.
Which means Lua VM bytecode execution is atomic for this as we were wondering :)
Again, for those who don't know, Fabien came to Rio in 2009 and, together with me and Bogdan (who came too !:) made a talk on his uses of Lua on Wavecom products and their futures. Among other things, he announced (and it was a pleasure to hear !) that future Wavecom/Sierra Wireless products with Lua embedded would be "eLua API compatible" too. As they are ahead of us on this and strongly pushed by commercial demands, with a nice base of products on the streets already, I think it is great to have him here and it is interesting to check their API (resumed above) too.
Too bad :( But I know and must mention here Fabien's personal and continuous efforts to convince the big bosses up there to open those too. Thank you again for this Fabien.
Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Fabien
On Tue, Feb 1, 2011 at 09:59, John Hind <[hidden email]> wrote:
Oh, you are very lucky (and of course wise :)
Again, I hope to hear Roberto's thoughts on this, as soon as he's back from a few days off. And I'm sure Bogdan will do some deeper tests too, as soon as we launch v0.8
... and the C part of the INT handlers also allow us to add filters or to create interesting "virtual" events, like the GPS parsing sentence that Bogdan has mentioned and other more complex ones. Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Dado Sutter
On Tue, Feb 1, 2011 at 10:12, John Hind <[hidden email]> wrote:
I'm sorry for the flooding but this is only due to our (frenetic :) final sprint on v0.8 closing and polishing (unfortunately, we don't have a large team for testing yet but we'll do fine). You can also disable your notifications on your account settings, at least during this period.
Definitely yes but we shouldn't be mentioning this here :) I try to avoid creating expectations. As you all know, both me, Bogdan and James (unfortunately :) have other "main" jobs (yet :) but we are really close to the release of eLua v0.8 now. Thank you all very much again for the support and for helping eLua to grow faster and faster. Best Dado
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by Fabien
On Tue, Feb 1, 2011 at 12:59 PM, John Hind <[hidden email]> wrote:
Indeed, but the debug module is quite openly advertised as the official Lua hellgate, the only module which can break Lua's safety principles. It is of notoriously bad taste to have production code relying on debug APIs, so if it breaks your code, you had it coming.
I am not very confident in my ability to always follow the rules, and totally unconfident in other people's ability to do so. Since I sometimes have to maintain code written by others, I'd rather have the rules strictly enforced by default!
It's worse than that actually, it prevents from loosing events together. Imagine that you have a signal('ping', 'ping', <packetnumber>) arriving every second, and you try to catch it in a loop:
function catch_pings() while true
local emitter, event, arg = sched.wait('ping', 'ping') printf("received packet #%d", arg)
do_something_complicated_with(arg) end
end sched.run (catch_pings) if do_something_complicated_with(arg) takes more than one second, the sched.wait() in the next iteration of the loop will miss one packet. You need some form of synchronous hook to handle this. It is very common to use a hook simply to buffer some data into a pipe, and maybe we should integrate a simplified support for such signal piping in the API.
_______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
Wow, this is getting a lot of respones which is nice. A few random
thoughts on the subject: 1. while the current interrupt model in eLua might change a bit more in the future I expect that the cetral idea will remain the same. The reason is quite simple: most of the expected audience of eLua (people with little or no programming experience at all) might not like to hear about multitasking (be it preemptive or not). An interrupt-based system is easier to understand than a non-preemptive multitasking system and much easier to understand than a preemprive multitasking system. Also, I'm trying to keep the original Lua usage model which offers both easy to understand concepts and more advanced (sometimes almost philosophycal :) ) ones which suits lots of users. So John, don't worry that the current interrupt system might prevent your work from entering eLua. In the end they'll most likely work in parallel. 2. most of the time I',m a big fan of what Fabien mentione ealier: cooperative systems are powerful enough to fulfill the needs of most applications out there. Preemptive multitasking is a very powerful tool, but with great power comes great responsability, as always :) In practice this translates into increased development and maintenance effort on preemptive systems (I used them quite a bit and I know how tricky they can be ever for seemingly innocent tasks). And I always try to use the simplest tool for the job I have to do. 3. I managed to take a quick look at John's proposal and at the interpreter bailout patch. Thanks for the idea, it's very good and I loved the concept of forcefully inserting "yield" calls into the Lua VM instruction stream to achieve preemptive multitasking. I'll have more to say on the subject, but until then an observation: this still isn't "real" preemptive multitasking. All you have to do is call a blocking C function from the Lua VM and the multitasking is gone. And there are quite a few such blocking functions in eLua (tmr.delay or uart.read with an infinite timeout are the usual examples). They don't even have to be blocking actually, if they take a long time to execute (take the same uart.read function but this time with a timeout of one second or more) the net effect is that preemptive multitasking isn't really preemptive anymore. This limits the applicability of the idea as the user must be specifically instructed to avoid these calls. In the light of this I don't know how much actual advantage you'd gain from this kind of pseudo-preemptive multitasking (and I mean that BTW, I really don't know :) I'd like to hear people's oppinions on this). I thought long and hard about true multitasking capabilities in eLua and my conclusion is that this can't be achieved without a C level threads library and all the complexity that comes with this. Best, Bogdan On Tue, Feb 1, 2011 at 5:39 PM, Fabien <[hidden email]> wrote: > On Tue, Feb 1, 2011 at 12:59 PM, John Hind <[hidden email]> wrote: >> >> Roberto's hellgate of non-determinism has already been opened by the debug >> hooks > > Indeed, but the debug module is quite openly advertised as the official Lua > hellgate, the only module which can break Lua's safety principles. It is of > notoriously bad taste to have production code relying on debug APIs, so if > it breaks your code, you had it coming. >> >> Personally I have never had a problem with shared memory multitasking, but >> it is true you can easily get in a subtle mess if you do not follow the >> rules. > > I am not very confident in my ability to always follow the rules, and > totally unconfident in other people's ability to do so. Since I sometimes > have to maintain code written by others, I'd rather have the rules strictly > enforced by default! > >> >> Your ability to hook a Lua function to an interrupt generated event so it >> gets queued for execution synchronously during the next non-blocking wait is >> also a good additional feature since it avoids the RAM overhead of a >> coroutine in some simpler cases. > > It's worse than that actually, it prevents from loosing events together. > Imagine that you have a signal('ping', 'ping', <packetnumber>) arriving > every second, and you try to catch it in a loop: > function catch_pings() > while true > local emitter, event, arg = sched.wait('ping', 'ping') > printf("received packet #%d", arg) > do_something_complicated_with(arg) > end > end > sched.run (catch_pings) > if do_something_complicated_with(arg) takes more than one second, the > sched.wait() in the next iteration of the loop will miss one packet. You > need some form of synchronous hook to handle this. It is very common to use > a hook simply to buffer some data into a pipe, and maybe we should integrate > a simplified support for such signal piping in the API. > > _______________________________________________ > eLua-dev mailing list > [hidden email] > https://lists.berlios.de/mailman/listinfo/elua-dev > > eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
You are right, Bogdan, about not being able to force a context switch within
C code called from Lua. I had not thought about that. But I guess my answer would be: let's get rid of (or just refrain from using) those blocking C functions! After all, the intention here is to provide a more flexible way of handling the same programming situations. Replace them either with the "yield until" syntax, or functions hooked to events or both. Both these solutions can be done with cooperative or preemptive coroutines. The key point we need to think about is that the current INT system already exposes us to the objections to shared memory multitasking: spliced INT code can (indeed to be useful, must) operate on the same variables that the code it is injected into operates on. > -----Original Message----- > From: [hidden email] [mailto:elua-dev- > [hidden email]] On Behalf Of Bogdan Marinescu > Sent: 01 February 2011 16:15 > To: eLua Users and Development List (www.eluaproject.net) > Subject: Re: [eLua-dev] Multitasking Proposal > > Wow, this is getting a lot of respones which is nice. A few random > thoughts on the subject: > > 1. while the current interrupt model in eLua might change a bit more > in the future I expect that the cetral idea will remain the same. The > reason is quite simple: most of the expected audience of eLua (people > with little or no programming experience at all) might not like to > hear about multitasking (be it preemptive or not). An interrupt-based > system is easier to understand than a non-preemptive multitasking > system and much easier to understand than a preemprive multitasking > system. Also, I'm trying to keep the original Lua usage model which > offers both easy to understand concepts and more advanced (sometimes > almost philosophycal :) ) ones which suits lots of users. So John, > don't worry that the current interrupt system might prevent your work > from entering eLua. In the end they'll most likely work in parallel. > 2. most of the time I',m a big fan of what Fabien mentione ealier: > cooperative systems are powerful enough to fulfill the needs of most > applications out there. Preemptive multitasking is a very powerful > tool, but with great power comes great responsability, as always :) In > practice this translates into increased development and maintenance > effort on preemptive systems (I used them quite a bit and I know how > tricky they can be ever for seemingly innocent tasks). And I always > try to use the simplest tool for the job I have to do. > 3. I managed to take a quick look at John's proposal and at the > interpreter bailout patch. Thanks for the idea, it's very good and I > loved the concept of forcefully inserting "yield" calls into the Lua > VM instruction stream to achieve preemptive multitasking. I'll have > more to say on the subject, but until then an observation: this still > isn't "real" preemptive multitasking. All you have to do is call a > blocking C function from the Lua VM and the multitasking is gone. And > there are quite a few such blocking functions in eLua (tmr.delay or > uart.read with an infinite timeout are the usual examples). They don't > even have to be blocking actually, if they take a long time to execute > (take the same uart.read function but this time with a timeout of one > second or more) the net effect is that preemptive multitasking isn't > really preemptive anymore. This limits the applicability of the idea > as the user must be specifically instructed to avoid these calls. In > the light of this I don't know how much actual advantage you'd gain > from this kind of pseudo-preemptive multitasking (and I mean that BTW, > I really don't know :) I'd like to hear people's oppinions on this). I > thought long and hard about true multitasking capabilities in eLua and > my conclusion is that this can't be achieved without a C level threads > library and all the complexity that comes with this. > > Best, > Bogdan > > On Tue, Feb 1, 2011 at 5:39 PM, Fabien <[hidden email]> wrote: > > On Tue, Feb 1, 2011 at 12:59 PM, John Hind <[hidden email]> > wrote: > >> > >> Roberto's hellgate of non-determinism has already been opened by the > debug > >> hooks > > > > Indeed, but the debug module is quite openly advertised as the > official Lua > > hellgate, the only module which can break Lua's safety principles. It > is of > > notoriously bad taste to have production code relying on debug APIs, > so if > > it breaks your code, you had it coming. > >> > >> Personally I have never had a problem with shared memory > multitasking, but > >> it is true you can easily get in a subtle mess if you do not follow > the > >> rules. > > > > I am not very confident in my ability to always follow the rules, and > > totally unconfident in other people's ability to do so. Since I > sometimes > > have to maintain code written by others, I'd rather have the rules > strictly > > enforced by default! > > > >> > >> Your ability to hook a Lua function to an interrupt generated event > so it > >> gets queued for execution synchronously during the next non-blocking > wait is > >> also a good additional feature since it avoids the RAM overhead of a > >> coroutine in some simpler cases. > > > > It's worse than that actually, it prevents from loosing events > together. > > Imagine that you have a signal('ping', 'ping', <packetnumber>) > arriving > > every second, and you try to catch it in a loop: > > function catch_pings() > > while true > > local emitter, event, arg = sched.wait('ping', 'ping') > > printf("received packet #%d", arg) > > do_something_complicated_with(arg) > > end > > end > > sched.run (catch_pings) > > if do_something_complicated_with(arg) takes more than one second, the > > sched.wait() in the next iteration of the loop will miss one packet. > You > > need some form of synchronous hook to handle this. It is very common > to use > > a hook simply to buffer some data into a pipe, and maybe we should > integrate > > a simplified support for such signal piping in the API. > > > > _______________________________________________ > > eLua-dev mailing list > > [hidden email] > > https://lists.berlios.de/mailman/listinfo/elua-dev > > > > > _______________________________________________ > eLua-dev mailing list > [hidden email] > https://lists.berlios.de/mailman/listinfo/elua-dev _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
In reply to this post by BogdanM
On Tue, Feb 1, 2011 at 8:22 PM, John Hind <[hidden email]> wrote:
> You are right, Bogdan, about not being able to force a context switch within > C code called from Lua. I had not thought about that. But I guess my answer > would be: let's get rid of (or just refrain from using) those blocking C > functions! I couldn't agree with you more! Good luck to us (and to others probably) in convincing our users to do so :) > After all, the intention here is to provide a more flexible way > of handling the same programming situations. Replace them either with the > "yield until" syntax, or functions hooked to events or both. Both these > solutions can be done with cooperative or preemptive coroutines. Definitely, and eLua already has means to replace most of (if not all) the blocking functions with non-blocking alternatives. > The key point we need to think about is that the current INT system already > exposes us to the objections to shared memory multitasking: spliced INT code > can (indeed to be useful, must) operate on the same variables that the code > it is injected into operates on. Sure. However, operation granularity is very different in Lua. An operation like "set the key in a hash map" couldn't be atomic in C; on the other hand Lua has the SETTABLE instrunction in the VM [1] which makes this an atomic operations. At the same time, since Lua does not use an optimizing compiler, I'm pretty sure you could look at all variables in a program as the C equivalent of "volatile". Of course all this doesn't eliminate synchronization issues (it couldn't do that) but I believe it makes them much easier to handle. I guess only time will tell :) Nobody (the core developers included) has enough practical experience with eLua interrupt support at this point to adopt a strong position on the matter. Best, Bogdan [1] http://luaforge.net/docman/view.php/83/98/ANoFrillsIntroToLua51VMInstructions.pdf > >> -----Original Message----- >> From: [hidden email] [mailto:elua-dev- >> [hidden email]] On Behalf Of Bogdan Marinescu >> Sent: 01 February 2011 16:15 >> To: eLua Users and Development List (www.eluaproject.net) >> Subject: Re: [eLua-dev] Multitasking Proposal >> >> Wow, this is getting a lot of respones which is nice. A few random >> thoughts on the subject: >> >> 1. while the current interrupt model in eLua might change a bit more >> in the future I expect that the cetral idea will remain the same. The >> reason is quite simple: most of the expected audience of eLua (people >> with little or no programming experience at all) might not like to >> hear about multitasking (be it preemptive or not). An interrupt-based >> system is easier to understand than a non-preemptive multitasking >> system and much easier to understand than a preemprive multitasking >> system. Also, I'm trying to keep the original Lua usage model which >> offers both easy to understand concepts and more advanced (sometimes >> almost philosophycal :) ) ones which suits lots of users. So John, >> don't worry that the current interrupt system might prevent your work >> from entering eLua. In the end they'll most likely work in parallel. >> 2. most of the time I',m a big fan of what Fabien mentione ealier: >> cooperative systems are powerful enough to fulfill the needs of most >> applications out there. Preemptive multitasking is a very powerful >> tool, but with great power comes great responsability, as always :) In >> practice this translates into increased development and maintenance >> effort on preemptive systems (I used them quite a bit and I know how >> tricky they can be ever for seemingly innocent tasks). And I always >> try to use the simplest tool for the job I have to do. >> 3. I managed to take a quick look at John's proposal and at the >> interpreter bailout patch. Thanks for the idea, it's very good and I >> loved the concept of forcefully inserting "yield" calls into the Lua >> VM instruction stream to achieve preemptive multitasking. I'll have >> more to say on the subject, but until then an observation: this still >> isn't "real" preemptive multitasking. All you have to do is call a >> blocking C function from the Lua VM and the multitasking is gone. And >> there are quite a few such blocking functions in eLua (tmr.delay or >> uart.read with an infinite timeout are the usual examples). They don't >> even have to be blocking actually, if they take a long time to execute >> (take the same uart.read function but this time with a timeout of one >> second or more) the net effect is that preemptive multitasking isn't >> really preemptive anymore. This limits the applicability of the idea >> as the user must be specifically instructed to avoid these calls. In >> the light of this I don't know how much actual advantage you'd gain >> from this kind of pseudo-preemptive multitasking (and I mean that BTW, >> I really don't know :) I'd like to hear people's oppinions on this). I >> thought long and hard about true multitasking capabilities in eLua and >> my conclusion is that this can't be achieved without a C level threads >> library and all the complexity that comes with this. >> >> Best, >> Bogdan >> >> On Tue, Feb 1, 2011 at 5:39 PM, Fabien <[hidden email]> wrote: >> > On Tue, Feb 1, 2011 at 12:59 PM, John Hind <[hidden email]> >> wrote: >> >> >> >> Roberto's hellgate of non-determinism has already been opened by the >> debug >> >> hooks >> > >> > Indeed, but the debug module is quite openly advertised as the >> official Lua >> > hellgate, the only module which can break Lua's safety principles. It >> is of >> > notoriously bad taste to have production code relying on debug APIs, >> so if >> > it breaks your code, you had it coming. >> >> >> >> Personally I have never had a problem with shared memory >> multitasking, but >> >> it is true you can easily get in a subtle mess if you do not follow >> the >> >> rules. >> > >> > I am not very confident in my ability to always follow the rules, and >> > totally unconfident in other people's ability to do so. Since I >> sometimes >> > have to maintain code written by others, I'd rather have the rules >> strictly >> > enforced by default! >> > >> >> >> >> Your ability to hook a Lua function to an interrupt generated event >> so it >> >> gets queued for execution synchronously during the next non-blocking >> wait is >> >> also a good additional feature since it avoids the RAM overhead of a >> >> coroutine in some simpler cases. >> > >> > It's worse than that actually, it prevents from loosing events >> together. >> > Imagine that you have a signal('ping', 'ping', <packetnumber>) >> arriving >> > every second, and you try to catch it in a loop: >> > function catch_pings() >> > while true >> > local emitter, event, arg = sched.wait('ping', 'ping') >> > printf("received packet #%d", arg) >> > do_something_complicated_with(arg) >> > end >> > end >> > sched.run (catch_pings) >> > if do_something_complicated_with(arg) takes more than one second, the >> > sched.wait() in the next iteration of the loop will miss one packet. >> You >> > need some form of synchronous hook to handle this. It is very common >> to use >> > a hook simply to buffer some data into a pipe, and maybe we should >> integrate >> > a simplified support for such signal piping in the API. >> > >> > _______________________________________________ >> > eLua-dev mailing list >> > [hidden email] >> > https://lists.berlios.de/mailman/listinfo/elua-dev >> > >> > >> _______________________________________________ >> eLua-dev mailing list >> [hidden email] >> https://lists.berlios.de/mailman/listinfo/elua-dev > > > _______________________________________________ > eLua-dev mailing list > [hidden email] > https://lists.berlios.de/mailman/listinfo/elua-dev > eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
 
|
Robert G. Jakabosky |
Re: Multitasking Proposal
|
|
In reply to this post by Dado Sutter
On Tuesday 01, Dado Sutter wrote:
> > On Tue, Feb 1, 2011 at 08:34, Fabien > <[hidden email]<fleutot%[hidden email]> > > wrote: > > > > Roberto, you shouldn't trust a language whose interpretation of `n=n+1' > > isn't deterministic) > > Which means Lua VM bytecode execution is atomic for this as we were > wondering :) Only if 'n' is a local variable. If you forgot to make it local, then that operation may no longer be atomic. Local variable: local n = 1 n = n + 1 LOADK 0 -1 ; local n = 1 ADD 0 0 -1 ; n = n + 1 Global variable: n = 1 n = n + 1 LOADK 0 -1 SETGLOBAL 0 -1 ; set global n to 1 GETGLOBAL 0 -1 ; load global n into register <------------ Global n could change here ADD 0 0 -2 ; add 1 to value in register <------------ or here SETGLOBAL 0 -1 ; set global n to value in register -- Robert G. Jakabosky _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
|
|
We use lua for a "wiring" components together. One issue is having "threads" calling back into Lua, i.e. get component information, post requests etc. So, right now, we post the event to a queue then wait the proper coroutine for that event. This might be to simplistic for processing ISRs..
For example... the main loop is running a scheduler, waiting for events to be posted. Once a event is posted the coroutine the is awaken to execute. The main issue with this everything now is non-blocking. Thus everything must have non-blocking calls. This is a little hard to do... but it really is just one big state machine type of execution.. To get long running coroutines, they schedule a simple event to get rescheduled. This is like the old WIN3.1 threading model... This way, there is not that much changes to the Lua core... Just a main even loop/scheduler... Just an idea... |
|
|
> We use lua for a "wiring" components together. One issue is having "threads"
> calling back into Lua, i.e. get component information, post requests etc. > So, right now, we post the event to a queue then wait the proper coroutine > for that event. This might be to simplistic for processing ISRs.. It's not simplistic. This is event-driven programming, usually implemented with message queues, and amongst other things it's nice because the queue serializes the events which are processed one by one, in turn. This is a form of self-synchronization. It's a widely used model and a very powerful one. > For example... the main loop is running a scheduler, waiting for events to > be posted. Once a event is posted the coroutine the is awaken to execute. > The main issue with this everything now is non-blocking. I wouldn't say "everything". A coroutine can still spend as much time as it wants before relinquishing control to the event scheduler (the main loop in your example). > Thus everything must have non-blocking calls. Ah sorry, I see your point now. Yes, everything must have non-blocking calls. > This is a little hard to > do... but it really is just one big state machine type of execution.. To > get long running coroutines, they schedule a simple event to get > rescheduled. This is like the old WIN3.1 threading model... This way, > there is not that much changes to the Lua core... Just a main even > loop/scheduler... > Just an idea... A very good idea. It's in the same class with what John's originally proposed (without the preemptive part) and what Fabien suggested (although an event-driven system is not what they suggested it fits the cooperative model very well). Actually one thing that keeps my faith in collaborative multitasking is "if it worked for Windows 3.1 it's probably good enough". Simplistic, I know, but also true :) Best, Bogdan > -- > View this message in context: http://elua-development.2368040.n2.nabble.com/Multitasking-Proposal-tp5977583p5982350.html > Sent from the eLua Development mailing list archive at Nabble.com. > _______________________________________________ > eLua-dev mailing list > [hidden email] > https://lists.berlios.de/mailman/listinfo/elua-dev > _______________________________________________ eLua-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/elua-dev |
| Free forum by Nabble | Edit this page |