Lua vs Python on Microcontrollers

[jbsnyder]

When I started looking around at interpreted languages on  
microcontrollers I ran across not only eLua, but also PyMite (http://pymite.python-hosting.com/ 
), the author of which (Dean) is now currently on this list.

I thought I would take this as an opportunity to strike up a little  
discussion on the merits and difficulties associated with implementing  
one language or the other (or any other interpreted languages, if it  
might be useful) on these devices with rather limited resources.

Lua seems naturally suited to limitations of an MCU, being pretty  
friendly to low resource situations out of the box.  I'm sure there  
are some modifications that were necessary to make eLua work in these  
environments in the first place, but sounds as if it didn't require  
much shoe-horning :-)

Python, on the other hand, being a "batteries included" language is  
quite large if you include all the standard modules and whatnot.  I  
certainly wouldn't expect that all of these would fit in an  
environment like the ones eLua or PyMite run on.  I would assume  
however, that some subset of libraries, and maybe core features of the  
language itself would fit reasonably (something like the array of  
modules that come with Lua?).

I'm wondering if those who have worked on implementing these  
environments could speak more intelligently about what is or is not  
reasonable, and maybe what some of the challenges are?  From my  
understanding PyMite implements a stripped down (maybe re-implemented  
from scratch?) version of Python's virtual machine.  I'm not sure if  
some of the opcodes are stripped out, or not, but I'm pretty sure that  
Python has way more opcodes than Lua does.  Also, lua makes use of a  
register-based VM rather than a more traditional stack-based VM.  Do  
these issues play a major role in how well these languages run and/or  
fit on a microcontroller?  Any thoughts on how these might relate to  
RAM, storage or CPU requirements?  On the python side, there is also  
TinyPy, which is supposedly modeled after lua in terms of the VM (not  
sure if it is register or stack based).  I've seen discussions on the  
TinyPy list about the current situation on getting it to run on micros  
(including needing to deal with filesystem ops for loading modules),  
but I'm not sure if it is reasonable or not to think that it might be  
a good solution.

I come from a background in a variety of languages, but have been  
using Python the most lately for desktop programming tasks.  I like it  
quite a bit as a language, and it has replaced MATLAB, C and other  
languages that I would have normally used otherwise to get something  
done.  I don't imagine writing much Lua for the types of things I use  
Python for on the desktop, but it sounds like it is better suited to  
MCUs without having to carve out any large portions of functionality.  
In particular, this means that the standard Lua documentation applies  
broadly to eLua, and that one can pretty much copy and paste a  
standard Lua script into eLua and have it work (so long as it doesn't  
use external modules not provided, or if the MCU doesn't have enough  
RAM).

Thanks!


--
James Snyder
Biomedical Engineering
Northwestern University
jbsnyder at fanplastic.org
http://fanplastic.org/key.txt
ph: [phone]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090123/cc7bc5d2/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : https://lists.berlios.de/pipermail/elua-dev/attachments/20090123/cc7bc5d2/attachment-0001.pgp 

[Dean Hall]

Good questions and discussion points, James.  I'd be happy to  
elaborate on PyMite.

First, PyMite's first targets were 8-bit AVR microcontrollers with 64  
KB Flash and 8 KB RAM.  So the resources were *much* smaller than even  
eLua is targeting.  However, I have found that 8 KB of RAM is too  
small to do anything useful.  32 KB seems to be enough to do useful  
things according to 2 different reports from researchers in Germany  
and MIT.  But this means attaching external RAM to the AVR which is  
beyond the scope of the hobbyist (at whom I am targeting PyMite).  
Since PyMite's first incarnation in 2002, the variety of 32-bit  
microcontrollers has increased and their prices have fallen; now, 32  
and even 64 KB of SRAM is available on-board both ARM7 and AVR32  
devices.  These new devices are on par with what eLua runs on.  Right  
now PyMite (and a few developers) are begging me to develop PyMite to  
run on these devices with greater resources.

In my early background research, I found previous attempts to reduce  
Python (Pippy, PythonCE) required around 256 KB of RAM and 1 M of  
program memory... so a total rewrite was necessary.  I rewrote the  
internal datastructures to reduce runtime memory requirements.  I  
rewrote the core interpreter, picking one at a time which bytecodes  
would make the cut to be included.  And I wrote the tools needed to  
take a .pyc compiled on a PC and compress it to a smaller format for  
inclusion into the program memory.  Literally, EVERY design decision  
was to reduce size.  Python was a very hard language to make small...  
and I'd say I still don't have it quite right.

One thing I added to PyMite was a pretty slick and easy way to call C  
functions from Python.  I call them native functions.  Briefly stated,  
you write a Python module with function defs, but put the C code in  
the docstring of the function and "pass" for the function body.  An  
included, automated tool finds these native functions and builds them  
into the executable and makes the lookup tables so it just works.  
This method was a lot better for smaller targets than CPython's  
method.  Using SWIG was never an option.  Lua has an easy way to write  
C functions and call them.

Lua was designed from the very start to be embedded (into other  
programs as well as into smaller systems).  Python was designed for  
the desktop and is rather big.  Even if I grow PyMite to use 256 KB  
Flash and 64 KB SRAM, I don't think I'll be able to fit every Python  
feature into PyMite.  Whereas, just recently, I showed that the eLua  
interpreter (minus its parser/lexer) will fit in 64 KB Flash and 16 KB  
SRAM.  So, in this respect, I believe Lua has the edge for size.  
However, I believe that due to my focus on internal data structure  
size, a PyMite program will use less RAM during runtime (but I admit  
this is difficult to quantify).

PyMite is stack based, like CPython.  tinypy is register based, like  
Lua.  I joined the tinypy maillist specifically because it's approach  
(follow the Lua design, but use Python syntax) was what I thought I  
wanted to do for the next version of PyMite.  However, tinypy's  
developer is focused on the desktop target and speed for gaming.  
tinypy would need a fairly significant rewrite for it to run in a low-
ram environment.

At this point, I believe eLua is in a great position: it offers the  
Lua language and useful modules.  I believe with further development  
PyMite could achieve a comparable level of ability as eLua on the same  
devices, but still not offer all of CPython's features (decorators,  
multiple inheritance, etc).  As a final note, I believe that since  
PyMite is completely re-implemented from scratch, it stands to  
introduce more design defects into the language.  So that is a  
drawback.  I haven't diffed eLua's src/lua directory against the  
standard Lua source tree, but I don't believe them to be vastly  
different.  So eLua has a high fidelity to the  Lua language and a  
common code base (fewer defects).

!!Dean


On Jan 23, 2009, at 12:36 , James Snyder wrote:

> When I started looking around at interpreted languages on  
> microcontrollers I ran across not only eLua, but also PyMite (http://pymite.python-hosting.com/ 
> ), the author of which (Dean) is now currently on this list.
>
> I thought I would take this as an opportunity to strike up a little  
> discussion on the merits and difficulties associated with  
> implementing one language or the other (or any other interpreted  
> languages, if it might be useful) on these devices with rather  
> limited resources.
>
> Lua seems naturally suited to limitations of an MCU, being pretty  
> friendly to low resource situations out of the box.  I'm sure there  
> are some modifications that were necessary to make eLua work in  
> these environments in the first place, but sounds as if it didn't  
> require much shoe-horning :-)
>
> Python, on the other hand, being a "batteries included" language is  
> quite large if you include all the standard modules and whatnot.  I  
> certainly wouldn't expect that all of these would fit in an  
> environment like the ones eLua or PyMite run on.  I would assume  
> however, that some subset of libraries, and maybe core features of  
> the language itself would fit reasonably (something like the array  
> of modules that come with Lua?).
>
> I'm wondering if those who have worked on implementing these  
> environments could speak more intelligently about what is or is not  
> reasonable, and maybe what some of the challenges are?  From my  
> understanding PyMite implements a stripped down (maybe re-
> implemented from scratch?) version of Python's virtual machine.  I'm  
> not sure if some of the opcodes are stripped out, or not, but I'm  
> pretty sure that Python has way more opcodes than Lua does.  Also,  
> lua makes use of a register-based VM rather than a more traditional  
> stack-based VM.  Do these issues play a major role in how well these  
> languages run and/or fit on a microcontroller?  Any thoughts on how  
> these might relate to RAM, storage or CPU requirements?  On the  
> python side, there is also TinyPy, which is supposedly modeled after  
> lua in terms of the VM (not sure if it is register or stack based).  
> I've seen discussions on the TinyPy list about the current situation  
> on getting it to run on micros (including needing to deal with  
> filesystem ops for loading modules), but I'm not sure if it is  
> reasonable or not to think that it might be a good solution.
>
> I come from a background in a variety of languages, but have been  
> using Python the most lately for desktop programming tasks.  I like  
> it quite a bit as a language, and it has replaced MATLAB, C and  
> other languages that I would have normally used otherwise to get  
> something done.  I don't imagine writing much Lua for the types of  
> things I use Python for on the desktop, but it sounds like it is  
> better suited to MCUs without having to carve out any large portions  
> of functionality.  In particular, this means that the standard Lua  
> documentation applies broadly to eLua, and that one can pretty much  
> copy and paste a standard Lua script into eLua and have it work (so  
> long as it doesn't use external modules not provided, or if the MCU  
> doesn't have enough RAM).
>
> Thanks!
>
>
> --
> James Snyder
> Biomedical Engineering
> Northwestern University
> jbsnyder at fanplastic.org
> http://fanplastic.org/key.txt
> ph: [phone]
>
> _______________________________________________
> Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev

[Jesus Alvarez]

Hi to all.

I am with Micromint, a developer of embedded controllers in Florida. I
started following the eLua project recently and joined the eLua-dev list a
few days ago. The eLua roadmap offers functionality that is attractive in
many application environments. Congratulations to Bogdan and the eLua
community on the achievements so far!

Throughout the years, we have seen many interpreters in use with our
embedded controllers.

For a long time, Basic was a very popular choice in the microcontroller
world. In some application domains, it still is popular. One big advantage
is that the language can be implemented with very low resource requirements.
8051 Basic takes only 8 KB and allows access to most of the functionality of
that 8-bit MCU. Basic allowed users without a computer science background to
implement a fairly large amount of functionality in a very short time.

Specialized C interpreters also led to significant functionality and low
resource requirements. You can build a C interpreter for an MCU with all the
APIs required by a specialized application in a 32 KB binary. That is still
an alternative popular with users familiar with C. Of course, making the
interpreter general purpose or adding networking makes the interpreter
binary grow.

Python has become very popular, gradually replacing Basic and Java for many
applications. Even AI researchers now consider Python an acceptable
alternative to Lisp (see Peter Norvig's comments at
http://norvig.com/python-lisp.html), something Java was never able to do.
The standard library implements a lot of functionality that leads to fast
code development but also results in relatively large resource requirements.
An old version of Python (1.5) was ported to the Palm some years ago
(Pippy). The binary plus library were around 400K and it had memory issues
in systems with less than 1 MB of RAM. Using popular Python libraries such
as PyLab was not feasible.

It may be difficult to implement the full Python standard library in
environments with 128 KB of flash and 64 KB of RAM, particularly if you also
need to include networking in that footprint. Yet a Python subset would
still be useful to many applications. For example, a chemist could code the
data acquisition for an experiment using a Python subset in an embedded
board that handles the sensors and actuators. The embedded Python
application could then periodically transfer the data via TCP/IP to a
workstation where it can be processed using a full Python implementation.
One big advantage there is that the chemist does not need to master two
languages and can apply his/her Python skills to the data acquisition
system.

Embedded Linux does offer an alternative for Python in embedded controllers.
Many embedded Linux boards now offer 64 MB of RAM and 1 GB of flash for a
relatively low cost. Besides more resources, many projects can take
advantage of a reliable multi-tasking operating system and many popular open
source applications. For many cases, embedded Linux is a good choice to run
Python applications in microcontrollers.

Lua has been popular as a scripting language to extend applications. It is
not as popular as Python so in many scenarios it would require users to
learn a new language. Yet its syntax is contemporary and its footprint is
relatively small. With eLua many practical applications are feasible even
within 128 KB of flash and 64 KB of RAM. The language is also simple to
extend. Applications that require specialized APIs for their hardware could
get a new eLua module working fairly quickly.

With interpreters you can implement a lot of functionality in a relatively
short time. The performance and capabilities are not as high as coding in C
or Assembler, but they can meet the requirements of many applications with
significantly less coding.

Regards,

Jesus Alvarez
Micromint USA
Tel [phone]
Fax [phone]

[Jesus Alvarez]

Dean,

Thanks for sharing your comments about PyMite and tinipy.

Coding a Python parser from scratch to target microcontrollers will
certainly generate incompatibilities (or defects as you call them) versus
the mainstream CPython implementation. Yet a Python subset could still be
very useful in many applications, especially if language functionality is
selected carefully. In most cases, people would not be porting a desktop
application to the embedded system but rather developing new code for a
specific embedded application. Besides, if a developer is proficient with
Python it is much easier for he/she to code their embedded application with
a Python subset than to learn a new language. The core of the language would
need to be there; developers may be willing to work around a lack of
multiple inheritance but they will resist changes to data structures, flow
control, iterators and functions.

Lack of an interpreter mode would be a drawback. From your comments, it
appears PyMite required compiling the code on the desktop and didn't
implement an interpreter mode. Although most production code runs compiled,
significant debugging and testing occurs in the interpreter. A possible
workaround to avoid the parser/ compiler overhead would be to have a special
"Python terminal" on the desktop acting as a remote interpreter console to
the embedded system, recompiling code as the user enters it.

For applications that can't work around the limitations of a Python subset,
using CPython under embedded Linux would be an alternative. The interpreter
resource issues are less critical with 64 MB of RAM than with 64 KB.

Regards,
Jesus Alvarez

[Dean Hall]

On Jan 23, 2009, at 17:25 , Jesus Alvarez wrote:

> Dean,
>
> Thanks for sharing your comments about PyMite and tinipy.
>
> Coding a Python parser from scratch to target microcontrollers will
> certainly generate incompatibilities (or defects as you call them)  
> versus
> the mainstream CPython implementation. Yet a Python subset could  
> still be
> very useful in many applications, especially if language  
> functionality is
> selected carefully. In most cases, people would not be porting a  
> desktop
> application to the embedded system but rather developing new code  
> for a
> specific embedded application. Besides, if a developer is proficient  
> with
> Python it is much easier for he/she to code their embedded  
> application with
> a Python subset than to learn a new language. The core of the  
> language would
> need to be there; developers may be willing to work around a lack of
> multiple inheritance but they will resist changes to data  
> structures, flow
> control, iterators and functions.

That is roughly the state PyMite is in right now.  It can do all the  
basic things you'd expect: assignment, math and logic, control flow,  
datatype operations, functions and modules (no iterators, though).

> Lack of an interpreter mode would be a drawback. From your comments,  
> it
> appears PyMite required compiling the code on the desktop and didn't
> implement an interpreter mode. Although most production code runs  
> compiled,
> significant debugging and testing occurs in the interpreter. A  
> possible
> workaround to avoid the parser/ compiler overhead would be to have a  
> special
> "Python terminal" on the desktop acting as a remote interpreter  
> console to
> the embedded system, recompiling code as the user enters it.

I call this an interactive mode.  PyMite has an interactive mode where  
the desktop is the host and the target, but it hasn't been adapted to  
a microcontroller target yet (not enough RAM on my ATmega103).

!!Dean

[BogdanM]

Hi all,

A personal comment first: I _love_ it when people talk about different
programming languages and their relative advantages and disadvantages using
logical arguments and a civilized tone, without getting into religious wars
like "my language will kick the living **** out of your language. Why? Cause
it DOES". Thanks for this, I really appreciate it.

And now my thoughts: I've been looking for a good way to embedded an
intepreted language into a microcontroller for a very long time now. For a
long while I've been the C-only type of programmer, that annoying guy who
thinks that everything but C (and maybe C++) is useless. Stupid, I know, but
so was me :) Once I started to widen my perspective a bit, the very first
thing I tried was Python, and it was a very pleasant experience for me. The
kind of productivity (and fun) I got from using Python was mind blowing for
someone that ate, dreamed and breathed C. I wrote a few large applications
in Python, and for desktop programming Python is still my favorite language
(and will probably remain so for a while). I explored others (Lua included),
but Python is the kind of complete package that most programmers are looking
for. I don't really care for the differences at the language level (while
Lua is simpler than Python, it's still a complete language), so I'm looking
at the libraries, and Python simply shines here.

However, I never considered Python for embedding into a MCU, because I
always though it was too 'heavy' for this task (this is why I find Dean's
work extraordinary). I started looking at Forth, and the conclusion was that
a "regular" user (maybe a non-programmer) wouldn't be crazy about the idea
of manipulating a stack directly, or writing his programs in postfix. The
same went for LISP. I find it interesting and challenging (although I only
had minimal exposure to it), but most people probably wouldn't. I also
considered different variants of BASIC (I was a big fan on BasicStamp when I
first met with micros), but BASIC just didn't do it for me the end, probably
because of my strong programming background that thought me a thing or two
about programming languages in general. This is why I knew I hit gold when I
found Lua. It was perfect for my needs in almost every way I could think of.


> I'm sure there are some modifications that were necessary to make eLua
work in
> these environments in the first place, but sounds as if it didn't require
> much shoe-horning :-)
You'll laugh, but the only thing I really needed to do in order to run the
interpreter for the first time was to rename Lua' "main" function to
"lua_main" and call it from my program. I had to take care of Newlib
instead, providing it the proper stubs for the MCU. Lua uses only ANSI
functions (and some POSIX in the OS library, which is normal) so it was
extremely easy to port it. I did some tests (before starting the project)
with some different memory allocators (dlmalloc, TLSF, chained versions) to
see what are best for eLua. dlmalloc won the initial round :), so I was
ready to go.

>Right now PyMite (and a few developers) are begging me to develop PyMite to
run
>on these devices with greater resources.
Well then, you give me no choice but to join their group :) I'd LOVE to have
Python run on my ARMs.

> Lua was designed from the very start to be embedded (into other
> programs as well as into smaller systems).  Python was designed for
> the desktop and is rather big.  Even if I grow PyMite to use 256 KB
> Flash and 64 KB SRAM, I don't think I'll be able to fit every Python
> feature into PyMite.  Whereas, just recently, I showed that the eLua
> interpreter (minus its parser/lexer) will fit in 64 KB Flash and 16 KB
> SRAM.  So, in this respect, I believe Lua has the edge for size.
> However, I believe that due to my focus on internal data structure
> size, a PyMite program will use less RAM during runtime (but I admit
> this is difficult to quantify).

You might be right, I didn't try that. One thing to keep in mind here is
that eLua starts with all the standard Lua libraries included by default. It
could take less RAM if you don't use the io library and loadlib (require and
friends), they are the only standard libraries that I couldn't make fully
ROMable. I'm thinking about reimplement at least the io library so it can
become completely ROMable (at this point it uses userdata, which in turn
uses the Lua registry, thus it takes some RAM). And loadlib might not be
required at all in the application (although if this idea that keeps popping
in my head is right, I'll be able to provide eLua with loadable C modules
one day, just like on desktop). Many things to think about here ...
I'm actually still trying to get the Lua memory usage right. The parser is
recursive, so it will eat quite a bit of stack for non-trivial programs.
I've tried to adress this by moving all the large parser structures from the
stack to the heap and increasing the stack size to 2k (which is only an
experimental value), and it seems to work fine for now, but I definitely
need to get back to this with a more "scientific" approach.

> I haven't diffed eLua's src/lua directory against the
> standard Lua source tree, but I don't believe them to be vastly
> different.  So eLua has a high fidelity to the  Lua language and a
> common code base (fewer defects).

These are the mods currently applied to Lua:

- lualong, a patch that changes Lua's number type from double to integer.
Quite safe, makes little to no modifications, unlikely to cause any
problems.
- my LTR. By far the most risky (and instrunsive) one, and likely to
introduce incompatibilities (but not serious ones; incompatibilities like
"math is now a rotable, so you can't add new members to it". True, but you
probably wouldn't do that in a real life program anyway).
- some changes to the parser that move large structures from the stack to
the heap (as specified above). A very safe change IMO.
- some small tweaks in the standard luaconf.h files (maximum number of
upvalues, maximum size of an input buffer, things like this) neede to
decrease the memory consumption a bit.

Best,
Bogdan

On Sat, Jan 24, 2009 at 1:56 AM, Dean Hall <dwhall256 at gmail.com> wrote:

>
> On Jan 23, 2009, at 17:25 , Jesus Alvarez wrote:
>
> > Dean,
> >
> > Thanks for sharing your comments about PyMite and tinipy.
> >
> > Coding a Python parser from scratch to target microcontrollers will
> > certainly generate incompatibilities (or defects as you call them)
> > versus
> > the mainstream CPython implementation. Yet a Python subset could
> > still be
> > very useful in many applications, especially if language
> > functionality is
> > selected carefully. In most cases, people would not be porting a
> > desktop
> > application to the embedded system but rather developing new code
> > for a
> > specific embedded application. Besides, if a developer is proficient
> > with
> > Python it is much easier for he/she to code their embedded
> > application with
> > a Python subset than to learn a new language. The core of the
> > language would
> > need to be there; developers may be willing to work around a lack of
> > multiple inheritance but they will resist changes to data
> > structures, flow
> > control, iterators and functions.
>
> That is roughly the state PyMite is in right now.  It can do all the
> basic things you'd expect: assignment, math and logic, control flow,
> datatype operations, functions and modules (no iterators, though).
>
> > Lack of an interpreter mode would be a drawback. From your comments,
> > it
> > appears PyMite required compiling the code on the desktop and didn't
> > implement an interpreter mode. Although most production code runs
> > compiled,
> > significant debugging and testing occurs in the interpreter. A
> > possible
> > workaround to avoid the parser/ compiler overhead would be to have a
> > special
> > "Python terminal" on the desktop acting as a remote interpreter
> > console to
> > the embedded system, recompiling code as the user enters it.
>
> I call this an interactive mode.  PyMite has an interactive mode where
> the desktop is the host and the target, but it hasn't been adapted to
> a microcontroller target yet (not enough RAM on my ATmega103).
>
> !!Dean
> _______________________________________________
> Elua-dev mailing list
> Elua-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/elua-dev
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090124/49e88ab5/attachment-0001.html 

[Dean Hall]

Bogdan's talk of the stack reminded me of one more little thing in  
PyMite: instead of declaring a big stack, I allocate execution frames  
on-demand.  This is a performance penalty, but I cache expired frames  
for cheap re-use.  A dynamic stack means no un-used space, but more  
importantly it allows having threads w/o multiple large stacks  
swallowing RAM.  The thread design for PyMite exists, but  
implementation is not complete to the best of my recollection.

!!Dean

[jbsnyder]

Thanks for the detailed reply Dean.  This certainly makes a lot of  
things more clear, and I think we can have a good discussion around  
this.

On Jan 23, 2009, at 1:46 PM, Dean Hall wrote:

> Good questions and discussion points, James.  I'd be happy to
> elaborate on PyMite.
>
> First, PyMite's first targets were 8-bit AVR microcontrollers with 64
> KB Flash and 8 KB RAM.  So the resources were *much* smaller than even
> eLua is targeting.  However, I have found that 8 KB of RAM is too
> small to do anything useful.  32 KB seems to be enough to do useful
> things according to 2 different reports from researchers in Germany
> and MIT.  But this means attaching external RAM to the AVR which is
> beyond the scope of the hobbyist (at whom I am targeting PyMite).
> Since PyMite's first incarnation in 2002, the variety of 32-bit
> microcontrollers has increased and their prices have fallen; now, 32
> and even 64 KB of SRAM is available on-board both ARM7 and AVR32
> devices.  These new devices are on par with what eLua runs on.  Right
> now PyMite (and a few developers) are begging me to develop PyMite to
> run on these devices with greater resources.

Yeah, the 1-8Kb space is reasonable for many purposes when one is  
writing in ASM or C, so long as either the code or data aren't too  
large.  If the project is very specialized, it's great, but I can't  
imagine trying to squeeze a very large VM or compiler into that kind  
of ram and flash space, so I'm quite impressed that you've managed to  
get a significant portion of the VM to fit in such a small space.

I think this is also why Arduino does well on this platform.  While it  
is easy to outgrow the available flash on the ATMega168, adding  
functionality is modular, and since one is still basically writing C  
code, it's fairly efficient space-wise.  I think if you were to take  
all of the optional libraries for interfacing with devices that come  
with the default support software, and include them with minimal  
additional code, you would likely be outgrowing the available flash.  
When one wants to have a VM and/or compiler, plus all of the  
peripheral libraries, the overhead is even greater, and therefore more  
difficult.

Also, external RAM could be an option, but I would guess there isn't  
any dedicated pathway for that, requiring some sort of serial  
interface to some SRAM or something similar, which would be somewhat  
slow.

Do you happen to have references for the papers you mention?  I'm  
curious about what issues they were studying.  Were they looking at  
interpreted languages or virtual machines on MCUs, or something  
different?

Also, there is at least one other VM on ATMega project that exists out  
there, NanoVM, which I believe implements some portion of the Java VM: http://www.harbaum.org/till/nanovm/index.shtml

> In my early background research, I found previous attempts to reduce
> Python (Pippy, PythonCE) required around 256 KB of RAM and 1 M of
> program memory... so a total rewrite was necessary.  I rewrote the
> internal datastructures to reduce runtime memory requirements.  I
> rewrote the core interpreter, picking one at a time which bytecodes
> would make the cut to be included.  And I wrote the tools needed to
> take a .pyc compiled on a PC and compress it to a smaller format for
> inclusion into the program memory.  Literally, EVERY design decision
> was to reduce size.  Python was a very hard language to make small...
> and I'd say I still don't have it quite right.

OK, so is the VM by itself already consuming quite a bit of available  
resources?  i.e. if I write a simple LED blinking application, am I  
already consuming a majority of the available RAM?

> One thing I added to PyMite was a pretty slick and easy way to call C
> functions from Python.  I call them native functions.  Briefly stated,
> you write a Python module with function defs, but put the C code in
> the docstring of the function and "pass" for the function body.  An
> included, automated tool finds these native functions and builds them
> into the executable and makes the lookup tables so it just works.
> This method was a lot better for smaller targets than CPython's
> method.  Using SWIG was never an option.  Lua has an easy way to write
> C functions and call them.

I've been working with building extensions to Lua in C for putting  
together the ADC module, and I'm finding that it's fairly easy and  
lightweight.  CPython does make it fairly easy to do quite a few  
things interfacing with C libraries, but I can see where this would be  
overkill on a microcontroller.

> Lua was designed from the very start to be embedded (into other
> programs as well as into smaller systems).  Python was designed for
> the desktop and is rather big.  Even if I grow PyMite to use 256 KB
> Flash and 64 KB SRAM, I don't think I'll be able to fit every Python
> feature into PyMite.  Whereas, just recently, I showed that the eLua
> interpreter (minus its parser/lexer) will fit in 64 KB Flash and 16 KB
> SRAM.  So, in this respect, I believe Lua has the edge for size.
> However, I believe that due to my focus on internal data structure
> size, a PyMite program will use less RAM during runtime (but I admit
> this is difficult to quantify).
>
> PyMite is stack based, like CPython.  tinypy is register based, like
> Lua.  I joined the tinypy maillist specifically because it's approach
> (follow the Lua design, but use Python syntax) was what I thought I
> wanted to do for the next version of PyMite.  However, tinypy's
> developer is focused on the desktop target and speed for gaming.
> tinypy would need a fairly significant rewrite for it to run in a low-
> ram environment.

That's too bad about TinyPy.  It is an interesting project, but I also  
have to admit that I was a bit less than happy when I started playing  
with it and realized that there were a decent number of language  
features that weren't implemented yet.  I'm guessing that some of the  
"significant rewrite" would be slimming things down for a  
microcontroller environment.  I might further guess that bringing  
TinyPy closer to CPython (in language features) might conflict with  
the goals of running in a microcontroller.

> At this point, I believe eLua is in a great position: it offers the
> Lua language and useful modules.  I believe with further development
> PyMite could achieve a comparable level of ability as eLua on the same
> devices, but still not offer all of CPython's features (decorators,
> multiple inheritance, etc).

Right, especially if PyMite were targeting a 32-bit MCU with a bit  
more RAM and flash.  I also think that some Python language features  
might not be all that necessary on a microcontroller, and are better  
suited towards desktop-type programming anyways.  Personally I might  
not miss decorators or multiple inheritance, but would miss things  
like iterators.

Do you think that a compiler might fit in this slightly larger  
environment, enabling a self-hosted interpreter?

>  As a final note, I believe that since
> PyMite is completely re-implemented from scratch, it stands to
> introduce more design defects into the language.  So that is a
> drawback.  I haven't diffed eLua's src/lua directory against the
> standard Lua source tree, but I don't believe them to be vastly
> different.  So eLua has a high fidelity to the  Lua language and a
> common code base (fewer defects).

Right, especially given that Python doesn't really seem to have a spec  
for implementation, aside from CPython.  That said, I think the  
microcontroller environment makes this sort of issue better in some  
ways. The projects implemented on top of it are likely simpler, and  
might not have as many complex failure modes.

I suppose, however, that debugging a VM can be fairly complicated  
depending on the type of issue encountered.

--
James Snyder
Biomedical Engineering
Northwestern University
jbsnyder at fanplastic.org
http://fanplastic.org/key.txt
ph: [phone]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090125/668f9df2/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : https://lists.berlios.de/pipermail/elua-dev/attachments/20090125/668f9df2/attachment.pgp 

[jbsnyder]

>
>> Lack of an interpreter mode would be a drawback. From your comments,
>> it
>> appears PyMite required compiling the code on the desktop and didn't
>> implement an interpreter mode. Although most production code runs
>> compiled,
>> significant debugging and testing occurs in the interpreter. A
>> possible
>> workaround to avoid the parser/ compiler overhead would be to have a
>> special
>> "Python terminal" on the desktop acting as a remote interpreter
>> console to
>> the embedded system, recompiling code as the user enters it.
>
> I call this an interactive mode.  PyMite has an interactive mode where
> the desktop is the host and the target, but it hasn't been adapted to
> a microcontroller target yet (not enough RAM on my ATmega103).

I've said this before but, to me, an interactive mode is probably one  
of the biggest features that I'm after in working with interpreted  
languages on microcontrollers.   There are pros and cons to having the  
parser/compiler on the microcontroller itself. One of which is whether  
one has to host much of a toolset on a desktop to be able to adjust  
code on the microcontroller.  That said, I think I'd probably be happy  
with either way of doing things (fully hosted, or PyMite's style of  
interactive mode).  Either one of these could make debugging and  
experimentation much easier.

Either one of these, in conjunction with a nice RPC mechanism, could  
make a great interactive tool where you could debug, update code, and  
ship data back and forth between the microcontroller and a desktop  
environment.  This would also make it fairly easy to work with the MCU  
from other languages like Python.

--
James Snyder
Biomedical Engineering
Northwestern University
jbsnyder at fanplastic.org
http://fanplastic.org/key.txt
ph: [phone]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/elua-dev/attachments/20090125/64133d21/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : https://lists.berlios.de/pipermail/elua-dev/attachments/20090125/64133d21/attachment-0001.pgp