IMPORTANT: new build system in master

[BogdanM]

Hello all,

The new build system and configurator are now in master. This means that using scons to build eLua is no longer supported, we switched to a Lua based build system (and a completely new way to configure an image, which replaces the old platform_conf.h mechanism). Check here for more details:

http://www.eluaproject.net/doc/master/en_building.html

This is a bit change and there are probably some bugs around, so please report any problems you'll find.

Best,
Bogdan

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Björn Kalkbrenner]

Hi,

great! Thank you very much! :)
I'll try it the next days on FreeBSD.

Bye
Björn


On 27.05.2013 10:56, Bogdan Marinescu wrote:

> Hello all,
>
> The new build system and configurator are now in master. This means that
> using scons to build eLua is no longer supported, we switched to a Lua
> based build system (and a completely new way to configure an image,
> which replaces the old platform_conf.h mechanism). Check here for more
> details:
>
> http://www.eluaproject.net/doc/master/en_building.html
>
> This is a bit change and there are probably some bugs around, so please
> report any problems you'll find.
>
> Best,
> Bogdan
>
>
>
> _______________________________________________
> eLua-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/elua-dev
>

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[raman]

Dear Bogdan,

> The new build system and configurator are now in master.

Wow! I was just looking into the code for this commit. I love the idea of
configuring the eLua image with Lua scripts. I'm yet to understand how
the header files are being generated. I can't wait to give this a try.

Thank you.

Best,
Raman

PS: I feel a bit sad about SConstruct - we won't have it any more :(

[Martin Guy]

> PS: I feel a bit sad about SConstruct - we won't have it any more :(

Bogdan, can you summarise what advantages the "new build system"
brings please, and how this affects developers.

Do we just call "build-elua.lua" with the same parameters as before?
Or are there user-visible changes too?

   M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Hello Martin and list,

On Fri, May 31, 2013 at 4:25 PM, Martin Guy <[hidden email]> wrote:

> PS: I feel a bit sad about SConstruct - we won't have it any more :(

Bogdan, can you summarise what advantages the "new build system"
brings please, and how this affects developers.

The new build system appeared mostly because I felt it integrates better with the rest of the system, especially the new configurator. As this whole system is written in Lua, integrated it with the Python-based scons felt quite a bit weird. What also felt weird is that we needed two separate interpreters (Pyton and Lua) only to compile Lua. All in all, things seemed to converge naturally towards a Lua-only build system. Plus, I was having trouble getting along with scons in some regards, which made the decision easier. The new build system is fully documented here:

http://www.eluaproject.net/doc/master/en_building.html

And the configurator too:

http://www.eluaproject.net/doc/master/en_configurator.html

The impact should be minimal to most users, which will now have to use "lua build_elua.lua" pretty much the same way they used "scons" in the past (and install a few things with luarocks, this is documented in the links above).

 

Do we just call "build-elua.lua" with the same parameters as before?
Or are there user-visible changes too?

They are all documented in  http://www.eluaproject.net/doc/master/en_building.html. Probably the most important part besides using build_elua.lua instead of scons is that now we are always bulding for a board. In other words, the old invocation 'scons cpu=<...>" doesn't work at all anymore. This makes sense, since eLua is generally built to run on a board, not a CPU. For the same name, the output is now named 'elua_lua_<boardname>.elf' instead of the previous 'elua_lua_<cpu>.elf'.

HTH,
Bogdan

 

   M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Martin Guy]

On 31 May 2013 16:27, Bogdan Marinescu <[hidden email]> wrote:
> On Fri, May 31, 2013 at 4:25 PM, Martin Guy <[hidden email]> wrote:
>> > PS: I feel a bit sad about SConstruct - we won't have it any more :(
>> Bogdan, can you summarise what advantages the "new build system"
>> brings please, and how this affects developers.

As you know, I'm no fan of changes to a working build system to a new
and different build system unless it is the only solution to overcome
specific technical obstacles.
  For me, as an eLua developer, the new build system meant having to
make the same changes twice in two files in two different languages,
Python and Lua, hoping they remained in sync and having to test every
change twice in every build configuration instead of once, doubling
the work.
  The new has no parallel build option, taking twice as long to
compile here during development, tripling it for testing.
  In these tests with two biuld systems, I found that it generates not
only different code, but larger code, using the same compiler with the
same options. For my very tight 120K target that makes the difference
between fitting in flash and overflowing it.
  Apart from the specific technical shortcomings outlined above, the
very fact of changing buld system has disadvantages to the eLua
program because it distracts developer time and attention away from
eLua itself, makes us maintain and debug two pieces of software
instead of one (since bugs in the build system are as much fun as bugs
in the compiler), and it leaves incompatibility with the other 40
branches and 50 forks of eLua, which would have to (but will not)
rewrite all their changes to the build system.

Half of builder.simplemachines.it stopped working when SConstruct
disappeared from the master branch, the half that compiles the master
branch instead of 0.8, which is the only 0.9 option it has at the
moment, for users to have system timer, mizar32.rtc and all the rest
of the good new things in elua0.9.
If SConstruct still works, is it too much to ask to put it back at
least until I get round to fixing the SM web builder, which I didn't
write and don't know how to fix yet?

    M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Fri, May 31, 2013 at 6:41 PM, Martin Guy <[hidden email]> wrote:

On 31 May 2013 16:27, Bogdan Marinescu <[hidden email]> wrote:
> On Fri, May 31, 2013 at 4:25 PM, Martin Guy <[hidden email]> wrote:
>> > PS: I feel a bit sad about SConstruct - we won't have it any more :(
>> Bogdan, can you summarise what advantages the "new build system"
>> brings please, and how this affects developers.

As you know, I'm no fan of changes to a working build system to a new
and different build system unless it is the only solution to overcome
specific technical obstacles.
  For me, as an eLua developer, the new build system meant having to
make the same changes twice in two files in two different languages,
Python and Lua, hoping they remained in sync and having to test every
change twice in every build configuration instead of once, doubling
the work.

This is exactly why I decided to give up scons entirely. Having two build systems in parallel was a pain.

 

  The new has no parallel build option, taking twice as long to
compile here during development, tripling it for testing.

I know, but this will be fixed soon (hopefully).

 

  In these tests with two biuld systems, I found that it generates not
only different code, but larger code, using the same compiler with the
same options. For my very tight 120K target that makes the difference
between fitting in flash and overflowing it.

We talked about that and I suggested that you check for different compiler flags and different link order. If it's a compiler flags issue, that's an easy fix. If it's just a question of the build order, I guess we could change that too, but this feels like pushing it a bit too far. I understand that you're working on a very space-constrained target, but depending on a specific link order to make your image fit doesn't feel quite right.

 

  Apart from the specific technical shortcomings outlined above, the
very fact of changing buld system has disadvantages to the eLua
program because it distracts developer time and attention away from
eLua itself, makes us maintain and debug two pieces of software
instead of one (since bugs in the build system are as much fun as bugs
in the compiler),

This will go away soon enough, once we switch to lake (because having an eLua specific build system doesn't make much sense indeed).

 

and it leaves incompatibility with the other 40
branches and 50 forks of eLua, which would have to (but will not)
rewrite all their changes to the build system.

I might be wrong about this, but I expect that most of the branches/forks that you're talking about didn't modify the build system too much; even if they did, there's a high chance that only a bunch of macros/flags were changed. If the change was more profound, I can help with the transition.

 

Half of builder.simplemachines.it stopped working when SConstruct
disappeared from the master branch, the half that compiles the master
branch instead of 0.8, which is the only 0.9 option it has at the
moment, for users to have system timer, mizar32.rtc and all the rest
of the good new things in elua0.9.

The builder is indeed an unfortunate collateral victim and I'm sorry this caused you so much trouble (our own builder suffered because of this too). I'll do what I can to help you transition to the new build system. Our builder was prepared to handle this change, but I simply didn't realise that you have a builder of your own (I never used it myself); if I did, you would've been notified about this change quite a while ago.

 

If SConstruct still works, is it too much to ask to put it back at
least until I get round to fixing the SM web builder, which I didn't
write and don't know how to fix yet?

I will try to put the old build system back on a branch (for AVR32 only). I'll get back to you as soon I'm done.

Best,

Bogdan

 

    M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

Hello again,

I've pushed a branch called 'scons_avr32' to github. It can be used to build _only_ AVR32 targets using the old build system. It uses the old platform_conf.h mechanism, not the configurator. It's the best I can do without significant effort, so I hope that's useful to you.

Best,
Bogdan

On Fri, May 31, 2013 at 7:41 PM, Bogdan Marinescu <[hidden email]> wrote:

On Fri, May 31, 2013 at 6:41 PM, Martin Guy <[hidden email]> wrote:

On 31 May 2013 16:27, Bogdan Marinescu <[hidden email]> wrote:
> On Fri, May 31, 2013 at 4:25 PM, Martin Guy <[hidden email]> wrote:
>> > PS: I feel a bit sad about SConstruct - we won't have it any more :(
>> Bogdan, can you summarise what advantages the "new build system"
>> brings please, and how this affects developers.

As you know, I'm no fan of changes to a working build system to a new
and different build system unless it is the only solution to overcome
specific technical obstacles.
  For me, as an eLua developer, the new build system meant having to
make the same changes twice in two files in two different languages,
Python and Lua, hoping they remained in sync and having to test every
change twice in every build configuration instead of once, doubling
the work.

This is exactly why I decided to give up scons entirely. Having two build systems in parallel was a pain.

 

  The new has no parallel build option, taking twice as long to
compile here during development, tripling it for testing.

I know, but this will be fixed soon (hopefully).

 

  In these tests with two biuld systems, I found that it generates not
only different code, but larger code, using the same compiler with the
same options. For my very tight 120K target that makes the difference
between fitting in flash and overflowing it.

We talked about that and I suggested that you check for different compiler flags and different link order. If it's a compiler flags issue, that's an easy fix. If it's just a question of the build order, I guess we could change that too, but this feels like pushing it a bit too far. I understand that you're working on a very space-constrained target, but depending on a specific link order to make your image fit doesn't feel quite right.

 

  Apart from the specific technical shortcomings outlined above, the
very fact of changing buld system has disadvantages to the eLua
program because it distracts developer time and attention away from
eLua itself, makes us maintain and debug two pieces of software
instead of one (since bugs in the build system are as much fun as bugs
in the compiler),

This will go away soon enough, once we switch to lake (because having an eLua specific build system doesn't make much sense indeed).

 

and it leaves incompatibility with the other 40
branches and 50 forks of eLua, which would have to (but will not)
rewrite all their changes to the build system.

I might be wrong about this, but I expect that most of the branches/forks that you're talking about didn't modify the build system too much; even if they did, there's a high chance that only a bunch of macros/flags were changed. If the change was more profound, I can help with the transition.

 

Half of builder.simplemachines.it stopped working when SConstruct
disappeared from the master branch, the half that compiles the master
branch instead of 0.8, which is the only 0.9 option it has at the
moment, for users to have system timer, mizar32.rtc and all the rest
of the good new things in elua0.9.

The builder is indeed an unfortunate collateral victim and I'm sorry this caused you so much trouble (our own builder suffered because of this too). I'll do what I can to help you transition to the new build system. Our builder was prepared to handle this change, but I simply didn't realise that you have a builder of your own (I never used it myself); if I did, you would've been notified about this change quite a while ago.

 

If SConstruct still works, is it too much to ask to put it back at
least until I get round to fixing the SM web builder, which I didn't
write and don't know how to fix yet?

I will try to put the old build system back on a branch (for AVR32 only). I'll get back to you as soon I'm done.

Best,

Bogdan

 

    M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Martin Guy]

> It uses the old
> platform_conf.h mechanism, not the configurator.

Does that mean that platform_conf's BUILD_* options or other are now
performed by something in the build system?
That would be an improvement!

Lake? Do you mean that, now that all the SConstruct-dependent work has
been destroyed, and we all have to put work into build-elua.lua,  you
intend to implement a third build system and, presumably, then delete
build-elua.lua?

No wonder all serious developers run away from eLua screaming!

    M

   M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Martin Guy]

On 31 May 2013 22:39, Bogdan Marinescu <[hidden email]> wrote:
> I've pushed a branch called 'scons_avr32' to github.

Thanks, but that still means modifying the builder, which is something
I'm still mustering the courage to do.
When/if I do, I'll make it build from mizar32_v0.9 branch, which is
stock elua0.9 with just my code size hacks.

Cheers

   m
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[BogdanM]

On Sat, Jun 1, 2013 at 9:55 AM, Martin Guy <[hidden email]> wrote:

> It uses the old
> platform_conf.h mechanism, not the configurator.

Does that mean that platform_conf's BUILD_* options or other are now
performed by something in the build system?
That would be an improvement!

Really? You didn't even read the configurator docs before deciding to trash the new build system? In any case, this is what the configurator does, yes (http://www.eluaproject.net/doc/master/en_configurator.html). However, this only works with the new, Lua based build system, because it is written in Lua and interfaces with the builder's Lua code. The branch that I've created for you uses the old scons-based build system and thus you still have to edit your platform_conf.h manually.

 

Lake? Do you mean that, now that all the SConstruct-dependent work has
been destroyed, and we all have to put work into build-elua.lua,  you
intend to implement a third build system and, presumably, then delete
build-elua.lua?

No wonder all serious developers run away from eLua screaming!

One of the things that I've learnt from life is that it is ocasionally helpful to get informed properly on a topic before getting all outraged about said topic. I recommend this to you, it's a good exercise. For example, should you have done that in this particular situation (especially since I've already told you that there is now a functional "uselake" branch on github that everybody can look at), you would've found that the differences to build_elua.lua are actually quite small. In fact, after getting myself accustomed with lake, it took me little over an hour to have the first eLua target bulding with lake. There are two main reasons for this:

- builld_elua.lua is mostly about executing the configurator, reading the backend specific configuration files, setting the compiler/linker/assembler flags, getting the list of files and other such tasks. The part that actually invokes the build process is in the last section of the file; when that section is reached, the things that are build-system independent (file list/compilation flags) are already set.

- lake and my actual builder are not that different conceptually, thus switching to lake was not that hard. When I say "my builder", I reffer to utils/build.lua, the build system that I wrote specifically for eLua. But this is only the part that does the actual build (invoking the compiler/linker/assembler, taking care of dependency analysis and so on); the rest of the system (configurator and the main part of build_elua.lua) are completely independent of the builder. As I already said in a number of ocasions, writing my own build system for eLua (and again, by this I'm reffering only to utils/build.lua) was a mistake; we simply didn't need another build system, it was something that started as an exercise and went too far eventually. This is why I am removing this part and ONLY this part and replacing it with lake, which is a dedicated build tool that seems to fit eLua perfectly.

To summarize, after lake will be in place, the only thing that MIGHT change is that instead of:

# lua build_elua.lua <options> ...

you'll have to do:

# lake <options> ...

And not even that will change if Steve (lake's author) agrees to package lake as a module that I'll be able to use from build_elua.lua, in which case the build system invocation won't change at all. On the backend configuration side, I expect very little things to change, so you won't have anything to maintain there. Although there is another branch that I'm working on to refactor the various toolchain options in a single place, which does change the build system a bit more. The idea here is that eLua is a dynamic project, it changes and it will keep on changing. I understand your resistance to change, but this is not going to stop, so we have to find a common ground somewhere.

Best,
Bogdan

    M

   M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev

[Martin Guy]

Thanks for the explanations.

I've replaced "master" with "avr32_scons" as you suggested and that
seems to have made it keep working

Thanks again

   M
_______________________________________________
eLua-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/elua-dev