Device Manager

[jbsnyder]

I have a couple questions about the device manager and usage thereof:

1.
In order to make LuaRPC fairly compatible with various transport  
layers we might use, should I be making the transport layer use the  
device manager?  I've not really delved into how all of that is set  
up, but it looks like this may be a useful fundamental way to hook  
into the transport layer?

Or should I have it work on a different level?  I think that the  
device manager functions should over the bases for what the RPC  
library needs (pretty much just sending and receiving bytes).  It  
might be nice for it to "automatically" work if a new new transport  
comes in so long as it conforms to an interface standard.

2.
Are the devman and stubs the main things that are newlib specific?

If one were thinking about working on top of another libc, where  
should one be focusing to need to find equivalent facilities in  
another C library?  Other C libraries, including the ARM ones do  
provide some similar hooks to the stubs, and for dealing with input/
output, so it should be possible without huge amounts of effort, I  
think.

--
James Snyder
Biomedical Engineering
Northwestern University
[hidden email]
http://fanplastic.org/key.txt
ph: [phone]


_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Thu, May 7, 2009 at 9:00 PM, James Snyder <[hidden email]> wrote:

I have a couple questions about the device manager and usage thereof:

1.
In order to make LuaRPC fairly compatible with various transport layers we might use, should I be making the transport layer use the device manager?  I've not really delved into how all of that is set up, but it looks like this may be a useful fundamental way to hook into the transport layer?

There isn't really a "transport layer" there, the device manager only deals with files in the traditional sense at the moment, so you can't have something like "/dev/tty0" (like in Linux) and treat it just like another file. Well, you could, but the interface is just too weak now I think. I actually give some thought to the idea of having something like this in eLua, but came to the conclusion that it isn't really worth the complexity of the implementation. What I'd personally do is add another "interface" to the platform interface. 
I have to think about this more though. Not for the near future, but for the long run. Having "/dev/spix" and "/dev/uartx" and "talking" to them via read/write/ioctl like a regular file has a lot of advantages, as well as some disadvantages. It sure seems tempting :)

Or should I have it work on a different level?  I think that the device manager functions should over the bases for what the RPC library needs (pretty much just sending and receiving bytes).  It might be nice for it to "automatically" work if a new new transport comes in so long as it conforms to an interface standard.

Something like an automount daemon, if I got it right. eLua is getting some file systems, so it should also get a "mount" command, and maybe even an automount, but again, I'm not sure that the current implementation allows for something more than just operations on regular file. It might though. It definitely works on consoles :), but a RPC transpor is probably too much for it. Even with desktop systems we have to use very different functions when writing to a console and writing to sockets, for example. That's why I said that I'd rather have some specialized functions in the platform interface for this, rather than making the device manager (which is a pretentious name for a very simple piece of code) much smarter than it is.

2.
Are the devman and stubs the main things that are newlib specific?

Yes. And because of this it will change a lot when we give eLua its own libc.

If one were thinking about working on top of another libc, where should one be focusing to need to find equivalent facilities in another C library?

There is already a branch in SVN on which I started to work on eLua with its own libc, which is a modified (well, almost _unmodified_ at this point) version of the Minix project's libc. Small, optimized for size, quite easy to understand and released under BSD, that's all we need :) There is _a lot_ of work to be done there, some trivial, some not quite so (like deciding if we also use the math emulation library from Minix (which I don't fully understand yet) ), but I think we can go there. And once we do, the advantages are huge IMO: complete portability, complete control over the library and how it works (so we can implement our stubs however we wish), reduced code size, (obviously) no more Newlib ... a dream come true :)

Best,
Bogdan

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On May 9, 2009, at 2:57 PM, Bogdan Marinescu <[hidden email]> wrote:

On Thu, May 7, 2009 at 9:00 PM, James Snyder <[hidden email][hidden email]> wrote:

I have a couple questions about the device manager and usage thereof:

1.
In order to make LuaRPC fairly compatible with various transport layers we might use, should I be making the transport layer use the device manager?  I've not really delved into how all of that is set up, but it looks like this may be a useful fundamental way to hook into the transport layer?

There isn't really a "transport layer" there, the device manager only deals with files in the traditional sense at the moment, so you can't have something like "/dev/tty0" (like in Linux) and treat it just like another file. Well, you could, but the interface is just too weak now I think. I actually give some thought to the idea of having something like this in eLua, but came to the conclusion that it isn't really worth the complexity of the implementation. What I'd personally do is add another "interface" to the platform interface. 
I have to think about this more though. Not for the near future, but for the long run. Having "/dev/spix" and "/dev/uartx" and "talking" to them via read/write/ioctl like a regular file has a lot of advantages, as well as some disadvantages. It sure seems tempting :)

OK. That's basically what I was wanting to know. On this front it would be nice. I also thought about trying to support rpc or exposing resources through plan 9's 9p. The Styx on a brick project (http://www.vitanuova.com/inferno/rcx_paper.html) from a few years back implemented this resource sharing/filesystem protocol on a lego mindstorms brick.  There's source for it on a google code project (http://code.google.com/p/inferno-os/) but I'm not sure if I'll get around to trying this or something similar. Part of the idea in plan 9 was to expose pretty much every resource through the filesystem, 9p is the protocol used. 

Or should I have it work on a different level?  I think that the device manager functions should over the bases for what the RPC library needs (pretty much just sending and receiving bytes).  It might be nice for it to "automatically" work if a new new transport comes in so long as it conforms to an interface standard.

Something like an automount daemon, if I got it right. eLua is getting some file systems, so it should also get a "mount" command, and maybe even an automount, but again, I'm not sure that the current implementation allows for something more than just operations on regular file. It might though. It definitely works on consoles :), but a RPC transpor is probably too much for it. Even with desktop systems we have to use very different functions when writing to a console and writing to sockets, for example. That's why I said that I'd rather have some specialized functions in the platform interface for this, rather than making the device manager (which is a pretentious name for a very simple piece of code) much smarter than it is.

Not necessarily automount, I was more just thinking of a common interface so that if one say added a new wireless stack and conformed with the interface one could just use this transport layer for console, rpc, etc without needing to do something custom for each usage modality. I realize to some extent that this modality might beak down somewhere but as with 9p one it seems like there should be a possibility for a common filelike interface for not just files but communication, direct control of non-file resources etc...

2.
Are the devman and stubs the main things that are newlib specific?

Yes. And because of this it will change a lot when we give eLua its own libc.

If one were thinking about working on top of another libc, where should one be focusing to need to find equivalent facilities in another C library?

There is already a branch in SVN on which I started to work on eLua with its own libc, which is a modified (well, almost _unmodified_ at this point) version of the Minix project's libc. Small, optimized for size, quite easy to understand and released under BSD, that's all we need :) There is _a lot_ of work to be done there, some trivial, some not quite so (like deciding if we also use the math emulation library from Minix (which I don't fully understand yet) ), but I think we can go there. And once we do, the advantages are huge IMO: complete portability, complete control over the library and how it works (so we can implement our stubs however we wish), reduced code size, (obviously) no more Newlib ... a dream come true :)

This would be nice to have. Have you read Ralph Hempel's article in lua gems?  I juat got the book this week.  He doesn't provide all the details but he does discuss some of the decisions he made about the c library as well as some of the math library choices. In particular his "flong" implementation, and I think he got his single precision math library from sdcc (small device c compiler). I somewhat wish there were more detail (or sources :) but he does provide some interesting pointers.  

Best,
Bogdan

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

OK. That's basically what I was wanting to know. On this front it would be nice. I also thought about trying to support rpc or exposing resources through plan 9's 9p. The Styx on a brick project (http://www.vitanuova.com/inferno/rcx_paper.html) from a few years back implemented this resource sharing/filesystem protocol on a lego mindstorms brick.  There's source for it on a google code project (http://code.google.com/p/inferno-os/) but I'm not sure if I'll get around to trying this or something similar. Part of the idea in plan 9 was to expose pretty much every resource through the filesystem, 9p is the protocol used. 

Very interesting, I wasn't aware of the existence of plan 9. I took a quick look at the protocol, and it sounds fine. Couldn't find the source code for Styx-on-a-Brick (http://www.vitanuova.com/inferno/rcx_paper.html) after a quick googling, but I'll keep on trying :) Still not sure how to interface with this from Linux (although I'm sure there's a way), not to mention Windows.
 

Not necessarily automount, I was more just thinking of a common interface so that if one say added a new wireless stack and conformed with the interface one could just use this transport layer for console, rpc, etc without needing to do something custom for each usage modality. I realize to some extent that this modality might beak down somewhere but as with 9p one it seems like there should be a possibility for a common filelike interface for not just files but communication, direct control of non-file resources etc...

Interesting, and it makes sense. And, as I said, this idea haunted me too :) Maybe 9p is simple/robust enough to make this possible in eLua. Not sure yet.

This would be nice to have. Have you read Ralph Hempel's article in lua gems?  I juat got the book this week.  He doesn't provide all the details but he does discuss some of the decisions he made about the c library as well as some of the math library

choices. In particular his "flong" implementation, and I think he got his single precision math library from sdcc (small device c compiler). I somewhat wish there were more detail (or sources :) but he does provide some interesting pointers.  

I talked to Ralph, and it turns out that he chose the Minix libc too for pbLua :)  So we're on the right track here. About the floating point emulation library, I'm still looking for alternatives. The sdcc idea is nice, thanks for that. But this is one area in which eLua and pbLua are different. If I'm not mistaken, pbLua can't even be compiled with the number type set to double, it's integer by default and uses a separate library for floating point numbers (which is a nice idea and we'll have it too one day). So our choice of a floating point emulation library might have to be different. I worked on three floating point emulation libraries myself a while ago, and I know how very tricky they can get, so we need to be careful.

Best,
Bogdan

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On May 11, 2009, at 3:12 AM, Bogdan Marinescu wrote:

OK. That's basically what I was wanting to know. On this front it would be nice. I also thought about trying to support rpc or exposing resources through plan 9's 9p. The Styx on a brick project (http://www.vitanuova.com/inferno/rcx_paper.html) from a few years back implemented this resource sharing/filesystem protocol on a lego mindstorms brick.  There's source for it on a google code project (http://code.google.com/p/inferno-os/) but I'm not sure if I'll get around to trying this or something similar. Part of the idea in plan 9 was to expose pretty much every resource through the filesystem, 9p is the protocol used. 

Very interesting, I wasn't aware of the existence of plan 9. I took a quick look at the protocol, and it sounds fine. Couldn't find the source code for Styx-on-a-Brick (http://www.vitanuova.com/inferno/rcx_paper.html) after a quick googling, but I'll keep on trying :) Still not sure how to interface with this from Linux (although I'm sure there's a way), not to mention Windows.

I had the same questions the first time I read this article and dug around.  The sources are indeed in that inferno-os project on google code:

http://code.google.com/p/inferno-os/source/browse/#svn/trunk/appl/demo/lego

http://code.google.com/p/inferno-os/source/browse/#svn/trunk/lib/lego

etc..  haven't dug around to try and figure out exactly what all is needed from those sources to have a working chunk.

This might be useful too:

http://code.google.com/p/inferno-ds/

Now, on to interfacing from Linux, Windows, etc..:

Linux has had v9fs included for some time: http://en.wikipedia.org/wiki/V9fs

There are also numerous implementations of the server/client in various languages:

http://9p.cat-v.org/implementations

I can't imagine it would be too hard to take a C implementation and hook it in with Lua.

I wouldn't necessarily expect that it show up as a real filesystem on all the platforms that would interface with it, I think in most cases they would just be used through a library that speaks the protocol.

 

Not necessarily automount, I was more just thinking of a common interface so that if one say added a new wireless stack and conformed with the interface one could just use this transport layer for console, rpc, etc without needing to do something custom for each usage modality. I realize to some extent that this modality might beak down somewhere but as with 9p one it seems like there should be a possibility for a common filelike interface for not just files but communication, direct control of non-file resources etc...

Interesting, and it makes sense. And, as I said, this idea haunted me too :) Maybe 9p is simple/robust enough to make this possible in eLua. Not sure yet.

I'm not sure either.  I also don't know what the overhead looks like for the implementations that are out there.  Conceptually it's quite powerful, but I'm not convinced whether it is a good or bad fit.  The Styx-on-a-brick project was particularly of interest because it suggests that it can be used on the class of devices eLua considers home :-)

This would be nice to have. Have you read Ralph Hempel's article in lua gems?  I juat got the book this week.  He doesn't provide all the details but he does discuss some of the decisions he made about the c library as well as some of the math library

choices. In particular his "flong" implementation, and I think he got his single precision math library from sdcc (small device c compiler). I somewhat wish there were more detail (or sources :) but he does provide some interesting pointers.  

I talked to Ralph, and it turns out that he chose the Minix libc too for pbLua :)  So we're on the right track here. About the floating point emulation library, I'm still looking for alternatives. The sdcc idea is nice, thanks for that. But this is one area in which eLua and pbLua are different. If I'm not mistaken, pbLua can't even be compiled with the number type set to double, it's integer by default and uses a separate library for floating point numbers (which is a nice idea and we'll have it too one day). So our choice of a floating point emulation library might have to be different. I worked on three floating point emulation libraries myself a while ago, and I know how very tricky they can get, so we need to be careful.

Yeah, I don't think it supports double.  He uses the "flong" (http://lua-users.org/lists/lua-l/2007-05/msg00306.html) to handle things as longs if there's no decimal point and floats if they have one.  It's an interesting tradeoff, but I don't think we'd want to give up support for higher precision (when wanted/needed).  I would tend towards whatever solution requires minimal maintenance and code.  I suspect any transition could be done gradually rather than having to wholesale make a switch?

Does the branch you've put together work as-is or is it more of a work-in-progress?

--

James Snyder

Biomedical Engineering

Northwestern University

[hidden email]

http://fanplastic.org/key.txt

ph: [phone]

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Linux has had v9fs included for some time: http://en.wikipedia.org/wiki/V9fs

Somehow, I'm not surprised :) Even if it wasn't, there's always FUSE in Linux.
 

There are also numerous implementations of the server/client in various languages:

http://9p.cat-v.org/implementations

I can't imagine it would be too hard to take a C implementation and hook it in with Lua.

Apparently there's at least a FUSE-like software for Windows and MAC too : http://yukoba.accelart.jp/2008/05/windows-fuse-dokan-become-open-source.html. Then again, if we have FUSE, we can implement pretty much everything we like :), as simple or complex as we want it.
 

I wouldn't necessarily expect that it show up as a real filesystem on all the platforms that would interface with it, I think in most cases they would just be used through a library that speaks the protocol.

Nice too, but having it as another file system would be simply beautiful.
 

I'm not sure either.  I also don't know what the overhead looks like for the implementations that are out there.  Conceptually it's quite powerful, but I'm not convinced whether it is a good or bad fit.  The Styx-on-a-brick project was particularly of interest because it suggests that it can be used on the class of devices eLua considers home :-)

That's what caught my attention too. Their MCU is far less limited than any of eLua's current targets, so it should be more than OK for us.

Yeah, I don't think it supports double.  He uses the "flong" (http://lua-users.org/lists/lua-l/2007-05/msg00306.html) to handle things as longs if there's no decimal point and floats if they have one.  It's an interesting tradeoff, but I don't think we'd want to give up support for higher precision (when wanted/needed).  

Nope, we don't want that.
 

I would tend towards whatever solution requires minimal maintenance and code.  I suspect any transition could be done gradually rather than having to wholesale make a switch?

Sure. Whatever is missing will be linked in from Newlib, and eventually Newlib will vanish completely. Hopefully before the mayan calendar ends ... :) Mind you, it's not only a question of libraries, we also have to (almost?) completely replace the header files without upsetting the toolchain. And do this in a platform-independent manner. All in all, not an easy task.
 

Does the branch you've put together work as-is or is it more of a work-in-progress?

It is 100% WIP. Although I'm pretty sure that the basic stuff (like string-related functions for example) are already working. But I didn't test them yet.

Best,
Bogdan

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[jbsnyder]

On May 11, 2009, at 4:56 PM, Bogdan Marinescu wrote:

There are also numerous implementations of the server/client in various languages:

http://9p.cat-v.org/implementations

I can't imagine it would be too hard to take a C implementation and hook it in with Lua.

Apparently there's at least a FUSE-like software for Windows and MAC too : http://yukoba.accelart.jp/2008/05/windows-fuse-dokan-become-open-source.html. Then again, if we have FUSE, we can implement pretty much everything we like :), as simple or complex as we want it.

There's at least one port of 9P to FUSE here:

http://swtch.com/plan9port/man/man4/9pfuse.html

(I'm not sure if that's the latest, in terms of sources.)

FUSE on Mac OS X is incredibly solid and well tested: http://code.google.com/p/macfuse/ (Port is maintained by Amit Singh who wrote the 1600 page Mac OS X Internals book)

I hadn't heard about the windows one, though, that's very interesting.

--

James Snyder

Biomedical Engineering

Northwestern University

[hidden email]

http://fanplastic.org/key.txt

ph: [phone]

_______________________________________________
Elua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev