C library

classic Classic list List threaded Threaded
12 messages Options
Matt Wilbur Matt Wilbur
Reply | Threaded
Open this post in threaded view
|

C library

Hi All,

I was horrified this week to see that the newlib implementation of scanf ( well, the lower level grunt called by vsfscanf ) was adding 100k to my binary size.  

I think I may experiment with another c library ( as Bogdan once suggested ) such as the Minix libc, or pdlibc.

Anyone else interested?  If yes, I'll post my progress to the list in order to elicit feedback and to ease integration into the trunk or a branch.  If no, I'll just work on it myself and merge the eLua trunk into my own stuff.

Matt
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
BogdanM BogdanM
Reply | Threaded
Open this post in threaded view
|

Re: C library

Hi,

On Sun, Jun 19, 2011 at 4:01 PM, Matt Wilbur <[hidden email]> wrote:
Hi All,

I was horrified this week to see that the newlib implementation of scanf ( well, the lower level grunt called by vsfscanf ) was adding 100k to my binary size.

I think I may experiment with another c library ( as Bogdan once suggested ) such as the Minix libc, or pdlibc.

Anyone else interested?  If yes, I'll post my progress to the list in order to elicit feedback and to ease integration into the trunk or a branch.  If no, I'll just work on it myself and merge the 
eLua trunk into my own stuff.

Totally interested. This is something I've been meaning to do for quite a while now, but never got the time (and things don't look like they're about to improve soon). 
I've heard about pdlibc, but right now (after a quick Google search, that is) it seems lost somewhere in the great bit bucket, can't find any relevant links. The Minix libc looked like a very good place to start when I checked it. Whatever you choose, please consider the licensing. If we are to include your work into the eLua tree we have to be sure that we use a libc with a compatible licensing. That includes MIT, BSD and (probably) Apache. It also completely excludes GNU and its derivates (so you can say goodbye to "dietlibc" for example) 
Good luck, and by all means, please keep me informed. I'll do my best to help.

Best,
Bogdan


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
Dean Hall Dean Hall
Reply | Threaded
Open this post in threaded view
|

Re: C library

avr-libc provides vfscanf and friends, but not vsfscanf.  The license  
seems compatible.  Might be something to learn from or adapt.

http://svn.savannah.nongnu.org/viewvc/*checkout*/trunk/avr-libc/libc/stdio/vfscanf.c?root=avr-libc&revision=2191&content-type=text%2Fplain

!!Dean


On Jun 19, 2011, at 09:27 , Bogdan Marinescu wrote:

> Hi,
>
> On Sun, Jun 19, 2011 at 4:01 PM, Matt Wilbur <[hidden email]>  
> wrote:
> Hi All,
>
> I was horrified this week to see that the newlib implementation of  
> scanf ( well, the lower level grunt called by vsfscanf ) was adding  
> 100k to my binary size.
>
> I think I may experiment with another c library ( as Bogdan once  
> suggested ) such as the Minix libc, or pdlibc.
>
> Anyone else interested?  If yes, I'll post my progress to the list  
> in order to elicit feedback and to ease integration into the trunk  
> or a branch.  If no, I'll just work on it myself and merge the  eLua  
> trunk into my own stuff.
>
> Totally interested. This is something I've been meaning to do for  
> quite a while now, but never got the time (and things don't look  
> like they're about to improve soon).
> I've heard about pdlibc, but right now (after a quick Google search,  
> that is) it seems lost somewhere in the great bit bucket, can't find  
> any relevant links. The Minix libc looked like a very good place to  
> start when I checked it. Whatever you choose, please consider the  
> licensing. If we are to include your work into the eLua tree we have  
> to be sure that we use a libc with a compatible licensing. That  
> includes MIT, BSD and (probably) Apache. It also completely excludes  
> GNU and its derivates (so you can say goodbye to "dietlibc" for  
> example)
> Good luck, and by all means, please keep me informed. I'll do my  
> best to help.
>
> Best,
> Bogdan
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
BogdanM BogdanM
Reply | Threaded
Open this post in threaded view
|

Re: C library

On Sun, Jun 19, 2011 at 5:44 PM, Dean Hall <[hidden email]> wrote:
avr-libc provides vfscanf and friends, but not vsfscanf.  The license seems compatible.  Might be something to learn from or adapt.

http://svn.savannah.nongnu.org/viewvc/*checkout*/trunk/avr-libc/libc/stdio/vfscanf.c?root=avr-libc&revision=2191&content-type=text%2Fplain

Thanks! Until a few minutes ago I was sure that avr libc is GPL. With the new information I got from Dean I can also recommend avr libc as a good model.

Best,
Bogdan


On Jun 19, 2011, at 09:27 , Bogdan Marinescu wrote:

Hi,

On Sun, Jun 19, 2011 at 4:01 PM, Matt Wilbur <[hidden email]> wrote:
Hi All,

I was horrified this week to see that the newlib implementation of scanf ( well, the lower level grunt called by vsfscanf ) was adding 100k to my binary size.

I think I may experiment with another c library ( as Bogdan once suggested ) such as the Minix libc, or pdlibc.

Anyone else interested?  If yes, I'll post my progress to the list in order to elicit feedback and to ease integration into the trunk or a branch.  If no, I'll just work on it myself and merge the  eLua trunk into my own stuff.

Totally interested. This is something I've been meaning to do for quite a while now, but never got the time (and things don't look like they're about to improve soon).
I've heard about pdlibc, but right now (after a quick Google search, that is) it seems lost somewhere in the great bit bucket, can't find any relevant links. The Minix libc looked like a very good place to start when I checked it. Whatever you choose, please consider the licensing. If we are to include your work into the eLua tree we have to be sure that we use a libc with a compatible licensing. That includes MIT, BSD and (probably) Apache. It also completely excludes GNU and its derivates (so you can say goodbye to "dietlibc" for example)
Good luck, and by all means, please keep me informed. I'll do my best to help.

Best,
Bogdan

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
Matt Wilbur Matt Wilbur
Reply | Threaded
Open this post in threaded view
|

Re: C library

I'm using eLua at work right now (I hope to be able to push my ppc work back out, but I'm still waiting for the green light), so I'm keenly aware of the licensing stuff. 

On Sun, Jun 19, 2011 at 10:53 AM, Bogdan Marinescu <[hidden email]> wrote:
On Sun, Jun 19, 2011 at 5:44 PM, Dean Hall <[hidden email]> wrote:
avr-libc provides vfscanf and friends, but not vsfscanf.  The license seems compatible.  Might be something to learn from or adapt.

http://svn.savannah.nongnu.org/viewvc/*checkout*/trunk/avr-libc/libc/stdio/vfscanf.c?root=avr-libc&revision=2191&content-type=text%2Fplain

Thanks! Until a few minutes ago I was sure that avr libc is GPL. With the new information I got from Dean I can also recommend avr libc as a good model.

http://pdclib.rootdirectory.de/ is the url I have for pdclib.  
 
Thanks for the tip on avrlibc.  I will give a look.  If work does get sticky, I will work on the libc stuff on my own time to ensure it can be part of the project as a whole.

I will probably start by writing unit tests on the existing code in order to ensure whatever C library is used, it won't break (e)lua.

Matt
 

Best,
Bogdan


On Jun 19, 2011, at 09:27 , Bogdan Marinescu wrote:

Hi,

On Sun, Jun 19, 2011 at 4:01 PM, Matt Wilbur <[hidden email]> wrote:
Hi All,

I was horrified this week to see that the newlib implementation of scanf ( well, the lower level grunt called by vsfscanf ) was adding 100k to my binary size.

I think I may experiment with another c library ( as Bogdan once suggested ) such as the Minix libc, or pdlibc.

Anyone else interested?  If yes, I'll post my progress to the list in order to elicit feedback and to ease integration into the trunk or a branch.  If no, I'll just work on it myself and merge the  eLua trunk into my own stuff.

Totally interested. This is something I've been meaning to do for quite a while now, but never got the time (and things don't look like they're about to improve soon).
I've heard about pdlibc, but right now (after a quick Google search, that is) it seems lost somewhere in the great bit bucket, can't find any relevant links. The Minix libc looked like a very good place to start when I checked it. Whatever you choose, please consider the licensing. If we are to include your work into the eLua tree we have to be sure that we use a libc with a compatible licensing. That includes MIT, BSD and (probably) Apache. It also completely excludes GNU and its derivates (so you can say goodbye to "dietlibc" for example)
Good luck, and by all means, please keep me informed. I'll do my best to help.

Best,
Bogdan

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev



_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
jbsnyder jbsnyder
Reply | Threaded
Open this post in threaded view
|

Re: C library

avr-libc is probably a good option to consider, especially if all the
functions are available in non-assembler forms.  I hadn't heard about
pdclib, that sounds interesting as well.  I'm willing to chip in on
helping on implementation for this since I've started digging into
what doing this would require a few times, but just haven't gotten
around to starting to replace newlib functionality with an alternate
library.

One thing of note however:  I don't think that any of the scanf
functions should be taking up 100 kB unless this is on a platform I'm
not familiar with.  If that's really the case for some situation, I
hope that version of scanf is getting a whole virtual machine +
compiler for that amount of space, since an integer-only version eLua
including c library can be scraped into 128 kB :-)  When I check a
build of eLua for lm3s6965, floating point these are the top offenders
(first column is decimal sizes):

$ arm-none-eabi-nm elua_lua_lm3s6965.elf --size-sort -r -t d
00004548 T _vfprintf_r
00004344 T _svfprintf_r
00003080 T _dtoa_r
00003040 T _strtod_r
00003032 T __svfscanf_r
00003032 T __ssvfscanf_r
00003024 T luaV_execute
00003020 T uip_process
00002676 T __ieee754_pow
00001740 T __kernel_rem_pio2
00001502 t chunk
00001392 T __ieee754_acos
00001300 t llex
00001180 T __ieee754_asin
00001118 T __gethex
00001064 T expm1
00001044 T __ieee754_log
00001032 T _malloc_r
00001032 D __malloc_av_
00001028 T __ieee754_rem_pio2
.... continues ....

Here's some other random bits of code I've stumbled across when
looking into this in the past:
Also, Fabrice Bellard included a really horribly, horribly stripped
down libc implementation in the linuxstart tarball for the JSLinux
emulator. The functions he did include are BSD licensed and do include
an implementation of vsnprintf (from BSD PPP):
http://bellard.org/jslinux/linuxstart-20110526.tar.gz
http://bellard.org/jslinux/
(which, by the way, is awesome.. and I think would make for a neat
demo or virtual platform for us :-) )

Here's another snprintf, looks somewhat large though:
http://www.jhweiss.de/software/snprintf.html

printf implementation comparison:
http://www.and.org/vstr/printf_comparison

Also there's the current BSD libc, though I'm guessing it'd be a bit
on the heavy side:
http://code.bsd64.org/cvsweb/freebsd/src/lib/libc/

And there's the Android Bionic libc which is a modified version of the
BSD libc with some modifications:
http://lxr.e2g.org/source/bionic/libc/
- no support for locales
- no support for wide chars (i.e. multi-byte characters)
- its own smallish implementation of pthreads based on Linux futexes
- support for x86, ARM and ARM thumb CPU instruction sets and kernel interfaces

-jsnyder


On Sun, Jun 19, 2011 at 12:03 PM, Matt Wilbur <[hidden email]> wrote:

> I'm using eLua at work right now (I hope to be able to push my ppc work back
> out, but I'm still waiting for the green light), so I'm keenly aware of the
> licensing stuff.
> On Sun, Jun 19, 2011 at 10:53 AM, Bogdan Marinescu
> <[hidden email]> wrote:
>>
>> On Sun, Jun 19, 2011 at 5:44 PM, Dean Hall <[hidden email]> wrote:
>>>
>>> avr-libc provides vfscanf and friends, but not vsfscanf.  The license
>>> seems compatible.  Might be something to learn from or adapt.
>>>
>>>
>>> http://svn.savannah.nongnu.org/viewvc/*checkout*/trunk/avr-libc/libc/stdio/vfscanf.c?root=avr-libc&revision=2191&content-type=text%2Fplain
>>
>> Thanks! Until a few minutes ago I was sure that avr libc is GPL. With the
>> new information I got from Dean I can also recommend avr libc as a good
>> model.
>
> http://pdclib.rootdirectory.de/ is the url I have for pdclib.
>
> Thanks for the tip on avrlibc.  I will give a look.  If work does get
> sticky, I will work on the libc stuff on my own time to ensure it can be
> part of the project as a whole.
> I will probably start by writing unit tests on the existing code in order to
> ensure whatever C library is used, it won't break (e)lua.
>
> Matt
>
>>
>> Best,
>> Bogdan
>>>
>>> On Jun 19, 2011, at 09:27 , Bogdan Marinescu wrote:
>>>
>>>> Hi,
>>>>
>>>> On Sun, Jun 19, 2011 at 4:01 PM, Matt Wilbur <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> I was horrified this week to see that the newlib implementation of scanf
>>>> ( well, the lower level grunt called by vsfscanf ) was adding 100k to my
>>>> binary size.
>>>>
>>>> I think I may experiment with another c library ( as Bogdan once
>>>> suggested ) such as the Minix libc, or pdlibc.
>>>>
>>>> Anyone else interested?  If yes, I'll post my progress to the list in
>>>> order to elicit feedback and to ease integration into the trunk or a branch.
>>>>  If no, I'll just work on it myself and merge the  eLua trunk into my own
>>>> stuff.
>>>>
>>>> Totally interested. This is something I've been meaning to do for quite
>>>> a while now, but never got the time (and things don't look like they're
>>>> about to improve soon).
>>>> I've heard about pdlibc, but right now (after a quick Google search,
>>>> that is) it seems lost somewhere in the great bit bucket, can't find any
>>>> relevant links. The Minix libc looked like a very good place to start when I
>>>> checked it. Whatever you choose, please consider the licensing. If we are to
>>>> include your work into the eLua tree we have to be sure that we use a libc
>>>> with a compatible licensing. That includes MIT, BSD and (probably) Apache.
>>>> It also completely excludes GNU and its derivates (so you can say goodbye to
>>>> "dietlibc" for example)
>>>> Good luck, and by all means, please keep me informed. I'll do my best to
>>>> help.
>>>>
>>>> Best,
>>>> Bogdan
>>>>
>>>> _______________________________________________
>>>> eLua-dev mailing list
>>>> [hidden email]
>>>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>>
>>> _______________________________________________
>>> eLua-dev mailing list
>>> [hidden email]
>>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>
>>
>> _______________________________________________
>> eLua-dev mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>
>
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
Robert G. Jakabosky Robert G. Jakabosky
Reply | Threaded
Open this post in threaded view
|

Re: C library


On Sunday 19, James Snyder wrote:
> $ arm-none-eabi-nm elua_lua_lm3s6965.elf --size-sort -r -t d
> 00004548 T _vfprintf_r
> 00004344 T _svfprintf_r


> 00003080 T _dtoa_r
> 00003040 T _strtod_r

I haven't tried it, but somewhere I heard about this dtoa() & strtod()
implementation (MIT licensed):
http://www.netlib.org/fp/dtoa.c

There is a lot more code on that site that might be useful:
http://www.netlib.org/liblist.html

--
Robert G. Jakabosky
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
Matt Wilbur Matt Wilbur
Reply | Threaded
Open this post in threaded view
|

Re: C library

In reply to this post by jbsnyder


On Sun, Jun 19, 2011 at 5:42 PM, James Snyder <[hidden email]> wrote:
avr-libc is probably a good option to consider, especially if all the
functions are available in non-assembler forms.  I hadn't heard about
pdclib, that sounds interesting as well.  I'm willing to chip in on
helping on implementation for this since I've started digging into
what doing this would require a few times, but just haven't gotten
around to starting to replace newlib functionality with an alternate
library.

One thing of note however:  I don't think that any of the scanf
functions should be taking up 100 kB unless this is on a platform I'm
not familiar with.  If that's really the case for some situation, I
hope that version of scanf is getting a whole virtual machine +
compiler for that amount of space, since an integer-only version eLua
including c library can be scraped into 128 kB :-)  When I check a
build of eLua for lm3s6965, floating point these are the top offenders
(first column is decimal sizes):

I'm using a PPC, which really stinks for code size.  I have been using CodeSourcery tools (powerpc-eabi lite).  I tried downloading the sources that they make available, to see if I could make things smaller.  What I have found has assumptions that I need to examine, but __svfscanf_r (the real worker behind scanf) seems to be really bloating things.

<snip>

Here's some other random bits of code I've stumbled across when
looking into this in the past:
Also, Fabrice Bellard included a really horribly, horribly stripped
down libc implementation in the linuxstart tarball for the JSLinux
emulator. The functions he did include are BSD licensed and do include
an implementation of vsnprintf (from BSD PPP):
http://bellard.org/jslinux/linuxstart-20110526.tar.gz
http://bellard.org/jslinux/

I'll take a look.  Perhaps this is like klibc for the linux kernel?  Which is then out due to GPL.  I've seen several "lightweight" printfs out there.  It's scanf that's tougher to find.


 

(which, by the way, is awesome.. and I think would make for a neat
demo or virtual platform for us :-) )

Here's another snprintf, looks somewhat large though:
http://www.jhweiss.de/software/snprintf.html

printf implementation comparison:
http://www.and.org/vstr/printf_comparison

Also there's the current BSD libc, though I'm guessing it'd be a bit
on the heavy side:
http://code.bsd64.org/cvsweb/freebsd/src/lib/libc/

And there's the Android Bionic libc which is a modified version of the
BSD libc with some modifications:
http://lxr.e2g.org/source/bionic/libc/
- no support for locales
- no support for wide chars (i.e. multi-byte characters)
- its own smallish implementation of pthreads based on Linux futexes
- support for x86, ARM and ARM thumb CPU instruction sets and kernel interfaces

-jsnyder


On Sun, Jun 19, 2011 at 12:03 PM, Matt Wilbur <[hidden email]> wrote:
> I'm using eLua at work right now (I hope to be able to push my ppc work back
> out, but I'm still waiting for the green light), so I'm keenly aware of the
> licensing stuff.
> On Sun, Jun 19, 2011 at 10:53 AM, Bogdan Marinescu
> <[hidden email]> wrote:
>>
>> On Sun, Jun 19, 2011 at 5:44 PM, Dean Hall <[hidden email]> wrote:
>>>
>>> avr-libc provides vfscanf and friends, but not vsfscanf.  The license
>>> seems compatible.  Might be something to learn from or adapt.
>>>
>>>
>>> http://svn.savannah.nongnu.org/viewvc/*checkout*/trunk/avr-libc/libc/stdio/vfscanf.c?root=avr-libc&revision=2191&content-type=text%2Fplain
>>
>> Thanks! Until a few minutes ago I was sure that avr libc is GPL. With the
>> new information I got from Dean I can also recommend avr libc as a good
>> model.
>
> http://pdclib.rootdirectory.de/ is the url I have for pdclib.
>
> Thanks for the tip on avrlibc.  I will give a look.  If work does get
> sticky, I will work on the libc stuff on my own time to ensure it can be
> part of the project as a whole.
> I will probably start by writing unit tests on the existing code in order to
> ensure whatever C library is used, it won't break (e)lua.
>
> Matt
>
>>
>> Best,
>> Bogdan
>>>
>>> On Jun 19, 2011, at 09:27 , Bogdan Marinescu wrote:
>>>
>>>> Hi,
>>>>
>>>> On Sun, Jun 19, 2011 at 4:01 PM, Matt Wilbur <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> I was horrified this week to see that the newlib implementation of scanf
>>>> ( well, the lower level grunt called by vsfscanf ) was adding 100k to my
>>>> binary size.
>>>>
>>>> I think I may experiment with another c library ( as Bogdan once
>>>> suggested ) such as the Minix libc, or pdlibc.
>>>>
>>>> Anyone else interested?  If yes, I'll post my progress to the list in
>>>> order to elicit feedback and to ease integration into the trunk or a branch.
>>>>  If no, I'll just work on it myself and merge the  eLua trunk into my own
>>>> stuff.
>>>>
>>>> Totally interested. This is something I've been meaning to do for quite
>>>> a while now, but never got the time (and things don't look like they're
>>>> about to improve soon).
>>>> I've heard about pdlibc, but right now (after a quick Google search,
>>>> that is) it seems lost somewhere in the great bit bucket, can't find any
>>>> relevant links. The Minix libc looked like a very good place to start when I
>>>> checked it. Whatever you choose, please consider the licensing. If we are to
>>>> include your work into the eLua tree we have to be sure that we use a libc
>>>> with a compatible licensing. That includes MIT, BSD and (probably) Apache.
>>>> It also completely excludes GNU and its derivates (so you can say goodbye to
>>>> "dietlibc" for example)
>>>> Good luck, and by all means, please keep me informed. I'll do my best to
>>>> help.
>>>>
>>>> Best,
>>>> Bogdan
>>>>
>>>> _______________________________________________
>>>> eLua-dev mailing list
>>>> [hidden email]
>>>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>>
>>> _______________________________________________
>>> eLua-dev mailing list
>>> [hidden email]
>>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>
>>
>> _______________________________________________
>> eLua-dev mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/elua-dev
>>
>
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
jbsnyder jbsnyder
Reply | Threaded
Open this post in threaded view
|

Re: C library

On Mon, Jun 20, 2011 at 8:58 AM, Matt Wilbur <[hidden email]> wrote:

>
>
> On Sun, Jun 19, 2011 at 5:42 PM, James Snyder <[hidden email]>
> wrote:
>>
>> avr-libc is probably a good option to consider, especially if all the
>> functions are available in non-assembler forms.  I hadn't heard about
>> pdclib, that sounds interesting as well.  I'm willing to chip in on
>> helping on implementation for this since I've started digging into
>> what doing this would require a few times, but just haven't gotten
>> around to starting to replace newlib functionality with an alternate
>> library.
>>
>> One thing of note however:  I don't think that any of the scanf
>> functions should be taking up 100 kB unless this is on a platform I'm
>> not familiar with.  If that's really the case for some situation, I
>> hope that version of scanf is getting a whole virtual machine +
>> compiler for that amount of space, since an integer-only version eLua
>> including c library can be scraped into 128 kB :-)  When I check a
>> build of eLua for lm3s6965, floating point these are the top offenders
>> (first column is decimal sizes):
>
> I'm using a PPC, which really stinks for code size.  I have been using
> CodeSourcery tools (powerpc-eabi lite).  I tried downloading the sources
> that they make available, to see if I could make things smaller.  What I
> have found has assumptions that I need to examine, but __svfscanf_r (the
> real worker behind scanf) seems to be really bloating things.
> <snip>

Wow, that is still pretty huge though..  If you can, I would recommend
trying building the codesourcery tools from scratch with newlib and
libraries optimized for size over speed.  This turns off hand unrolled
loops and platform specific code that while faster can be quite a bit
larger.  If you want a starting point, you could try the build guide
for arm on the elua site or the Makefile that I've put together for
building an ARM EABI toolchain:
https://github.com/jsnyder/arm-eabi-toolchain

There will be some changes you'll need to make to configure flags and
such aside from just the target, and you may want to take a look at
the .sh script that's bundled in the codesourcery tarball since that
provides the incantations they use for doing their builds, which can
be useful because at least on ARM they've done things like
add/customize flags for enabling multilib builds.

>>
>> Here's some other random bits of code I've stumbled across when
>> looking into this in the past:
>> Also, Fabrice Bellard included a really horribly, horribly stripped
>> down libc implementation in the linuxstart tarball for the JSLinux
>> emulator. The functions he did include are BSD licensed and do include
>> an implementation of vsnprintf (from BSD PPP):
>> http://bellard.org/jslinux/linuxstart-20110526.tar.gz
>> http://bellard.org/jslinux/
>
> I'll take a look.  Perhaps this is like klibc for the linux kernel?  Which
> is then out due to GPL.  I've seen several "lightweight" printfs out there.
>  It's scanf that's tougher to find.

I haven't looked at what's included in klibc, but this is just a
handful of functions. I'd have to examine it again, but I think he is
using it for the bootloader or something similar?  In any case, what's
in his "libc" is BSD licensed (probably because things like vsnprintf
come from BSD PPP) whereas most of what Bellard releases tends to be
GPL.

I've not compared scanf implementations for size. There certainly are
various implementations out there.  Google code search might be useful
though, if for nothing else to compare and see what's out there that
they've indexed:

http://www.google.com/codesearch#search/&q=lang:%5Ec$%20function:vfscanf%20license:(mit%7Cbsd)%20file:c$&type=cs

>
>
>>
>> (which, by the way, is awesome.. and I think would make for a neat
>> demo or virtual platform for us :-) )
>>
>> Here's another snprintf, looks somewhat large though:
>> http://www.jhweiss.de/software/snprintf.html
>>
>> printf implementation comparison:
>> http://www.and.org/vstr/printf_comparison
>>
>> Also there's the current BSD libc, though I'm guessing it'd be a bit
>> on the heavy side:
>> http://code.bsd64.org/cvsweb/freebsd/src/lib/libc/
>>
>> And there's the Android Bionic libc which is a modified version of the
>> BSD libc with some modifications:
>> http://lxr.e2g.org/source/bionic/libc/
>> - no support for locales
>> - no support for wide chars (i.e. multi-byte characters)
>> - its own smallish implementation of pthreads based on Linux futexes
>> - support for x86, ARM and ARM thumb CPU instruction sets and kernel
>> interfaces
>>
>> -jsnyder
>>
>>
>> On Sun, Jun 19, 2011 at 12:03 PM, Matt Wilbur <[hidden email]> wrote:
>> > I'm using eLua at work right now (I hope to be able to push my ppc work
>> > back
>> > out, but I'm still waiting for the green light), so I'm keenly aware of
>> > the
>> > licensing stuff.
>> > On Sun, Jun 19, 2011 at 10:53 AM, Bogdan Marinescu
>> > <[hidden email]> wrote:
>> >>
>> >> On Sun, Jun 19, 2011 at 5:44 PM, Dean Hall <[hidden email]> wrote:
>> >>>
>> >>> avr-libc provides vfscanf and friends, but not vsfscanf.  The license
>> >>> seems compatible.  Might be something to learn from or adapt.
>> >>>
>> >>>
>> >>>
>> >>> http://svn.savannah.nongnu.org/viewvc/*checkout*/trunk/avr-libc/libc/stdio/vfscanf.c?root=avr-libc&revision=2191&content-type=text%2Fplain
>> >>
>> >> Thanks! Until a few minutes ago I was sure that avr libc is GPL. With
>> >> the
>> >> new information I got from Dean I can also recommend avr libc as a good
>> >> model.
>> >
>> > http://pdclib.rootdirectory.de/ is the url I have for pdclib.
>> >
>> > Thanks for the tip on avrlibc.  I will give a look.  If work does get
>> > sticky, I will work on the libc stuff on my own time to ensure it can be
>> > part of the project as a whole.
>> > I will probably start by writing unit tests on the existing code in
>> > order to
>> > ensure whatever C library is used, it won't break (e)lua.
>> >
>> > Matt
>> >
>> >>
>> >> Best,
>> >> Bogdan
>> >>>
>> >>> On Jun 19, 2011, at 09:27 , Bogdan Marinescu wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> On Sun, Jun 19, 2011 at 4:01 PM, Matt Wilbur <[hidden email]>
>> >>>> wrote:
>> >>>> Hi All,
>> >>>>
>> >>>> I was horrified this week to see that the newlib implementation of
>> >>>> scanf
>> >>>> ( well, the lower level grunt called by vsfscanf ) was adding 100k to
>> >>>> my
>> >>>> binary size.
>> >>>>
>> >>>> I think I may experiment with another c library ( as Bogdan once
>> >>>> suggested ) such as the Minix libc, or pdlibc.
>> >>>>
>> >>>> Anyone else interested?  If yes, I'll post my progress to the list in
>> >>>> order to elicit feedback and to ease integration into the trunk or a
>> >>>> branch.
>> >>>>  If no, I'll just work on it myself and merge the  eLua trunk into my
>> >>>> own
>> >>>> stuff.
>> >>>>
>> >>>> Totally interested. This is something I've been meaning to do for
>> >>>> quite
>> >>>> a while now, but never got the time (and things don't look like
>> >>>> they're
>> >>>> about to improve soon).
>> >>>> I've heard about pdlibc, but right now (after a quick Google search,
>> >>>> that is) it seems lost somewhere in the great bit bucket, can't find
>> >>>> any
>> >>>> relevant links. The Minix libc looked like a very good place to start
>> >>>> when I
>> >>>> checked it. Whatever you choose, please consider the licensing. If we
>> >>>> are to
>> >>>> include your work into the eLua tree we have to be sure that we use a
>> >>>> libc
>> >>>> with a compatible licensing. That includes MIT, BSD and (probably)
>> >>>> Apache.
>> >>>> It also completely excludes GNU and its derivates (so you can say
>> >>>> goodbye to
>> >>>> "dietlibc" for example)
>> >>>> Good luck, and by all means, please keep me informed. I'll do my best
>> >>>> to
>> >>>> help.
>> >>>>
>> >>>> Best,
>> >>>> Bogdan
>> >>>>
>> >>>> _______________________________________________
>> >>>> eLua-dev mailing list
>> >>>> [hidden email]
>> >>>> https://lists.berlios.de/mailman/listinfo/elua-dev
>> >>>
>> >>> _______________________________________________
>> >>> eLua-dev mailing list
>> >>> [hidden email]
>> >>> https://lists.berlios.de/mailman/listinfo/elua-dev
>> >>
>> >>
>> >> _______________________________________________
>> >> eLua-dev mailing list
>> >> [hidden email]
>> >> https://lists.berlios.de/mailman/listinfo/elua-dev
>> >>
>> >
>> >
>> > _______________________________________________
>> > eLua-dev mailing list
>> > [hidden email]
>> > https://lists.berlios.de/mailman/listinfo/elua-dev
>> >
>> >
>> _______________________________________________
>> eLua-dev mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
jbsnyder jbsnyder
Reply | Threaded
Open this post in threaded view
|

Re: C library

In reply to this post by Robert G. Jakabosky
On Sun, Jun 19, 2011 at 6:25 PM, Robert G. Jakabosky
<[hidden email]> wrote:

>
> On Sunday 19, James Snyder wrote:
>> $ arm-none-eabi-nm elua_lua_lm3s6965.elf --size-sort -r -t d
>> 00004548 T _vfprintf_r
>> 00004344 T _svfprintf_r
>
>
>> 00003080 T _dtoa_r
>> 00003040 T _strtod_r
>
> I haven't tried it, but somewhere I heard about this dtoa() & strtod()
> implementation (MIT licensed):
> http://www.netlib.org/fp/dtoa.c

Interesting.  I hadn't looked into dtoa or strtod implementations
before.  I'd expect those to be generally large if they're done
carefully and without error.  There's some discussion on that here as
well as links to some experiments with quick and dirty floating point
conversion:
http://stackoverflow.com/questions/3173056/why-does-dtoa-c-contain-so-much-code/3173426#3173426

Discussions on cheaper/smaller approaches and under what conditions
they make errors:
http://www.exploringbinary.com/quick-and-dirty-floating-point-to-decimal-conversion/
http://www.exploringbinary.com/15-digit-quick-and-dirty-conversions-dont-round-trip/

Interesting that sqlite apparently limits conversions to 15 digits..
For eLua there are likely cases where it might be worth it to trade a
few kB for this type of limitation, but I don't really like the idea
of making it default especially since there are plenty of devices
where this space tradeoff wouldn't make it worth it.

Also, apparently newlib is using an older version of that same dtoa.c
written by David M. Gay:
http://sourceware.org/cgi-bin/cvsweb.cgi/~checkout~/src/newlib/libc/stdlib/dtoa.c?rev=1.5&content-type=text/plain&cvsroot=src

avr-libc does have a dtoa currently, but it's dependent on assembler
code.  Previously I think they just used what was provided by gcc.
I'm not sure how much is easily usable from there for other functions.
 There's definitely some avr-specific functionality even in ones that
are all C code.

>
> There is a lot more code on that site that might be useful:
> http://www.netlib.org/liblist.html

Thanks.  I think I've definitely borrowed

>
> --
> Robert G. Jakabosky
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
Matt Wilbur Matt Wilbur
Reply | Threaded
Open this post in threaded view
|

Re: C library

I had definitely screwed something up in my newlib re-compile to come up with such a huge number.  That being said, I still like the idea of seeing if there is a smaller solution than newlib.

I have modified the CodeSourcery script in order to do my work, but I will make something to follow James' Makefile example so anyone can do the same for a ppc toolchain.

On Mon, Jun 20, 2011 at 1:18 PM, James Snyder <[hidden email]> wrote:
On Sun, Jun 19, 2011 at 6:25 PM, Robert G. Jakabosky
<[hidden email]> wrote:
>
> On Sunday 19, James Snyder wrote:
>> $ arm-none-eabi-nm elua_lua_lm3s6965.elf --size-sort -r -t d
>> 00004548 T _vfprintf_r
>> 00004344 T _svfprintf_r
>
>
>> 00003080 T _dtoa_r
>> 00003040 T _strtod_r
>
> I haven't tried it, but somewhere I heard about this dtoa() & strtod()
> implementation (MIT licensed):
> http://www.netlib.org/fp/dtoa.c

Interesting.  I hadn't looked into dtoa or strtod implementations
before.  I'd expect those to be generally large if they're done
carefully and without error.  There's some discussion on that here as
well as links to some experiments with quick and dirty floating point
conversion:
http://stackoverflow.com/questions/3173056/why-does-dtoa-c-contain-so-much-code/3173426#3173426

Discussions on cheaper/smaller approaches and under what conditions
they make errors:
http://www.exploringbinary.com/quick-and-dirty-floating-point-to-decimal-conversion/
http://www.exploringbinary.com/15-digit-quick-and-dirty-conversions-dont-round-trip/

Interesting that sqlite apparently limits conversions to 15 digits..
For eLua there are likely cases where it might be worth it to trade a
few kB for this type of limitation, but I don't really like the idea
of making it default especially since there are plenty of devices
where this space tradeoff wouldn't make it worth it.

Also, apparently newlib is using an older version of that same dtoa.c
written by David M. Gay:
http://sourceware.org/cgi-bin/cvsweb.cgi/~checkout~/src/newlib/libc/stdlib/dtoa.c?rev=1.5&content-type=text/plain&cvsroot=src

avr-libc does have a dtoa currently, but it's dependent on assembler
code.  Previously I think they just used what was provided by gcc.
I'm not sure how much is easily usable from there for other functions.
 There's definitely some avr-specific functionality even in ones that
are all C code.

>
> There is a lot more code on that site that might be useful:
> http://www.netlib.org/liblist.html

Thanks.  I think I've definitely borrowed

>
> --
> Robert G. Jakabosky
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev


_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev
jbsnyder jbsnyder
Reply | Threaded
Open this post in threaded view
|

Re: C library

On Tue, Jun 21, 2011 at 11:28 AM, Matt Wilbur <[hidden email]> wrote:
> I had definitely screwed something up in my newlib re-compile to come up
> with such a huge number.  That being said, I still like the idea of seeing
> if there is a smaller solution than newlib.

No worries. We have talked about this a number of times in the past
and have wanted to pare down some of the functions that have been
expensive in terms of code size.  Additionally, going to our own C
library allows us to control compilation flags for all these library
functions at compile time, and also make decisions about tradeoffs
(especially where, for example, where the API that is provided to the
Lua user is more restricted than what the newlib function provides.)

> I have modified the CodeSourcery script in order to do my work, but I will
> make something to follow James' Makefile example so anyone can do the same
> for a ppc toolchain.

I hope that works well for you. I've also started experimenting with a
slightly more granular/staged approach with targets for more separate
stages (download, extract, prep, configure, build, install) with files
being used to mark when these stages have finished.  I haven't refined
it quite as much as I'd like, but I wanted something that could resume
building without having to start everything over from scratch each
time and I may migrate the arm makefile to this approach in the
future:
https://github.com/jsnyder/avr32-toolchain/blob/master/Makefile

Feel free to use whichever you like or another approach if that suits
you.  The arm one might be easiest to adapt right now since that's
also building from CodeSourcery sources.

>
> On Mon, Jun 20, 2011 at 1:18 PM, James Snyder <[hidden email]>
> wrote:
>>
>> On Sun, Jun 19, 2011 at 6:25 PM, Robert G. Jakabosky
>> <[hidden email]> wrote:
>> >
>> > On Sunday 19, James Snyder wrote:
>> >> $ arm-none-eabi-nm elua_lua_lm3s6965.elf --size-sort -r -t d
>> >> 00004548 T _vfprintf_r
>> >> 00004344 T _svfprintf_r
>> >
>> >
>> >> 00003080 T _dtoa_r
>> >> 00003040 T _strtod_r
>> >
>> > I haven't tried it, but somewhere I heard about this dtoa() & strtod()
>> > implementation (MIT licensed):
>> > http://www.netlib.org/fp/dtoa.c
>>
>> Interesting.  I hadn't looked into dtoa or strtod implementations
>> before.  I'd expect those to be generally large if they're done
>> carefully and without error.  There's some discussion on that here as
>> well as links to some experiments with quick and dirty floating point
>> conversion:
>>
>> http://stackoverflow.com/questions/3173056/why-does-dtoa-c-contain-so-much-code/3173426#3173426
>>
>> Discussions on cheaper/smaller approaches and under what conditions
>> they make errors:
>>
>> http://www.exploringbinary.com/quick-and-dirty-floating-point-to-decimal-conversion/
>>
>> http://www.exploringbinary.com/15-digit-quick-and-dirty-conversions-dont-round-trip/
>>
>> Interesting that sqlite apparently limits conversions to 15 digits..
>> For eLua there are likely cases where it might be worth it to trade a
>> few kB for this type of limitation, but I don't really like the idea
>> of making it default especially since there are plenty of devices
>> where this space tradeoff wouldn't make it worth it.
>>
>> Also, apparently newlib is using an older version of that same dtoa.c
>> written by David M. Gay:
>>
>> http://sourceware.org/cgi-bin/cvsweb.cgi/~checkout~/src/newlib/libc/stdlib/dtoa.c?rev=1.5&content-type=text/plain&cvsroot=src
>>
>> avr-libc does have a dtoa currently, but it's dependent on assembler
>> code.  Previously I think they just used what was provided by gcc.
>> I'm not sure how much is easily usable from there for other functions.
>>  There's definitely some avr-specific functionality even in ones that
>> are all C code.
>>
>> >
>> > There is a lot more code on that site that might be useful:
>> > http://www.netlib.org/liblist.html
>>
>> Thanks.  I think I've definitely borrowed
>>
>> >
>> > --
>> > Robert G. Jakabosky
>> > _______________________________________________
>> > eLua-dev mailing list
>> > [hidden email]
>> > https://lists.berlios.de/mailman/listinfo/elua-dev
>> >
>> _______________________________________________
>> eLua-dev mailing list
>> [hidden email]
>> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>
>
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev