On Mon, 2017-04-10 at 08:39 +0000, Atila Neves via Digitalmars-d- announce wrote:
> […]
> 
> As far as I know the only build system that does this by default for D is reggae.

I will be adding a new builder to the SCons D tools to do whole source and per package compiling – to add to the module at a time compiling. It would be good to add this for CMake-D and the D support in Meson.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Monday, 10 April 2017 at 12:10:41 UTC, Russel Winder wrote:
> On Mon, 2017-04-10 at 08:39 +0000, Atila Neves via Digitalmars-d- announce wrote:
>> […]
>> 
>> As far as I know the only build system that does this by default for D is reggae.
>
> I will be adding a new builder to the SCons D tools to do whole source and per package compiling – to add to the module at a time compiling. It would be good to add this for CMake-D and the D support in Meson.

I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff).

Having some level of dub integration is Meson would be neat indeed - maybe one could make a small helper binary Meson can call to fetch things from the dub registry.
I wonder though how that would jive with Meson's own subprojects/wrap system. Probably worth investigating.

On Mon, 2017-04-10 at 12:41 +0000, Matthias Klumpp via Digitalmars-d- announce wrote:
> 
[…].
> 
> I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff).

I am investigating this architecture because Chapel code cannot really
be compiled separately as far as I can see. cf. http://chapel.cray.com/
 Chapel is a PGAS language (like X10) for use with Big Computation™ on
serious computers and also any computer. I'm interested as a way of
connecting Python visualisation to computation that NumPy can't really
handle.

It seems sensible therefore to offer this way of working for D since whether there is actually any optimisation benefit or not some people think there is and use it as a stick to beat you with if it isn't there.

> Having some level of dub integration is Meson would be neat
> indeed - maybe one could make a small helper binary Meson can
> call to fetch things from the dub registry.
> I wonder though how that would jive with Meson's own
> subprojects/wrap system. Probably worth investigating.

My thought for SCons was to delegate the package fetching to Dub as a
subprocess or write some Python to use the Dub API. I'm not clear on
that as yet, the issue is whether the Dub local source repo is the
right way forward – using Dub for preparing the compiled artefact is
likely not the right way forward for SCons. This would then make it
easy to do something for Rust/Cargo – except that SCons doesn't really
support Rust yet, and with Cargo are there any Rust users not using
Cargo.

Having said all this SCons stuff, if there was Meson support for this
"get the source from the Dub repository, compile it and make it a
dependency" I'd likely stay with Meson for my codes.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Monday, 10 April 2017 at 15:27:25 UTC, Russel Winder wrote:
> On Mon, 2017-04-10 at 12:41 +0000, Matthias Klumpp via Digitalmars-d- announce wrote:
>> 
> […].
>> 
>> I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff).
>
> I am investigating this architecture because Chapel code cannot really
> be compiled separately as far as I can see. cf. http://chapel.cray.com/
>  Chapel is a PGAS language (like X10) for use with Big Computation™ on
> serious computers and also any computer. I'm interested as a way of
> connecting Python visualisation to computation that NumPy can't really
> handle.

That's pretty cool! One way to do this with Meson is to spawn a shell script as custom target, but that obviously sucks. It might be worth reporting this as issue upstream, with a concrete usecase like this, the Meson maintainers will highly likely add support for it.
One could also always write a plugin as a last resort.

> It seems sensible therefore to offer this way of working for D since whether there is actually any optimisation benefit or not some people think there is and use it as a stick to beat you with if it isn't there.
>
>> Having some level of dub integration is Meson would be neat
>> indeed - maybe one could make a small helper binary Meson can
>> call to fetch things from the dub registry.
>> I wonder though how that would jive with Meson's own
>> subprojects/wrap system. Probably worth investigating.
>
> My thought for SCons was to delegate the package fetching to Dub as a
> subprocess or write some Python to use the Dub API. I'm not clear on
> that as yet, the issue is whether the Dub local source repo is the
> right way forward – using Dub for preparing the compiled artefact is
> likely not the right way forward for SCons. This would then make it
> easy to do something for Rust/Cargo – except that SCons doesn't really
> support Rust yet, and with Cargo are there any Rust users not using
> Cargo.
>
> Having said all this SCons stuff, if there was Meson support for this
> "get the source from the Dub repository, compile it and make it a
> dependency" I'd likely stay with Meson for my codes.

SCons is considered evil, last time I checked ^^ => https://wiki.debian.org/UpstreamGuide#line867
(unless it's used right, which seems to be hard) - I have no idea though on whether the issues with it were fixed, the entry on SCons hasn't been updated in a while.

On Mon, 2017-04-10 at 17:56 +0000, Matthias Klumpp via Digitalmars-d- announce wrote:
> […]
> SCons is considered evil, last time I checked ^^ =>
> https://wiki.debian.org/UpstreamGuide#line867
> (unless it's used right, which seems to be hard) - I have no idea
> though on whether the issues with it were fixed, the entry on
> SCons hasn't been updated in a while.

Given that some of the "facts" there are actually wrong, I think we can infer Debian people assume Autotools is all that is needed and are uninterested in new things. SCons has supported sonames for a while, for example.

Prejudice in these things, as in all things, tends to get reified and become unalterable fact. This no longer worries me.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Mon, 2017-04-10 at 17:56 +0000, Matthias Klumpp via Digitalmars-d- announce wrote:
> […]
> 
> That's pretty cool! One way to do this with Meson is to spawn a
> shell script as custom target, but that obviously sucks. It might
> be worth reporting this as issue upstream, with a concrete
> usecase like this, the Meson maintainers will highly likely add
> support for it.
> One could also always write a plugin as a last resort.
> 
> > […]

I'll look to ensuring my facts are correct, and then find out where to put an issue about this – I am assuming a GitHub repository with issues .

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Monday, 10 April 2017 at 18:11:44 UTC, Russel Winder wrote:
> [...]
> I'll look to ensuring my facts are correct, and then find out where to put an issue about this – I am assuming a GitHub repository with issues .

Just file one at https://github.com/mesonbuild/meson/issues - it might even be that Meson supports this already, the project is moving so fast it's really hard to keep up with all the changes (Mesa and X investigating using it boosted it's development quite a bit, and it was really fast even before).

On Mon, Apr 10, 2017 at 05:56:38PM +0000, Matthias Klumpp via Digitalmars-d-announce wrote:
> On Monday, 10 April 2017 at 15:27:25 UTC, Russel Winder wrote:
[...]
> > My thought for SCons was to delegate the package fetching to Dub as
> > a subprocess or write some Python to use the Dub API. I'm not clear
> > on that as yet, the issue is whether the Dub local source repo is
> > the right way forward – using Dub for preparing the compiled
> > artefact is likely not the right way forward for SCons. This would
> > then make it easy to do something for Rust/Cargo – except that SCons
> > doesn't really support Rust yet, and with Cargo are there any Rust
> > users not using Cargo.
> > 
> > Having said all this SCons stuff, if there was Meson support for this "get the source from the Dub repository, compile it and make it a dependency" I'd likely stay with Meson for my codes.
> 
> SCons is considered evil, last time I checked ^^ =>
> https://wiki.debian.org/UpstreamGuide#line867
> (unless it's used right, which seems to be hard) - I have no idea
> though on whether the issues with it were fixed, the entry on SCons
> hasn't been updated in a while.

I use SCons exclusively for my D projects, and haven't faced any major problems.

I understand, however, that from a distro's POV, it can be annoying to work with if you're not familiar with how to patch SConstruct scripts to do what is needed.

It's true that unless upstream explicitly supports installation targets, SCons won't do it for you. I do actually maintain a Debian package that uses SCons (well, written by yours truly, so perhaps that's cheating), and I did find that I have to write rules explicitly for supporting installation targets just so it will work properly with Debian's packaging scripts.

As for SONAME support (mentioned by the wiki), I'm not sure what the big deal is... isn't it just a matter of passing the right linker flags to the compile command? So either adding something to CFLAGS or LDFLAGS in the construction environment should do the trick.

SCons refusing to pick up settings from the user's environment can be annoying to distros, but in return for that, it gives you 100% reproducible (incremental!) builds by default, whereas most Makefiles require you to `make clean` just to be sure, to the point that it has become accepted practice to always "build from clean" because otherwise you just never know. Even Debian's packaging scripts have a `clean` target because of this. But in an SCons-based package, the `clean` target is a one-liner `scons -c`, since SCons knows what targets it has produced and practically guarantees you're back to a clean slate.  Even though that's actually unnecessary to produce a good build! (Ironically, leaving the `clean` rule blank causes the packaging scripts to complain because they notice stray files lying around, obviously a feature written with a Makefile mindset because stray files are problematic in Makefile-based projects, but hardly worth attention in an SCons-based project!)


T

-- 
Curiosity kills the cat. Moral: don't be the cat.

On 04/10/2017 02:41 PM, Matthias Klumpp wrote:
> On Monday, 10 April 2017 at 12:10:41 UTC, Russel Winder wrote:
>> On Mon, 2017-04-10 at 08:39 +0000, Atila Neves via Digitalmars-d- announce wrote:
>>> […]
>>>
>>> As far as I know the only build system that does this by default for D is reggae.
>>
>> I will be adding a new builder to the SCons D tools to do whole source and per package compiling – to add to the module at a time compiling. It would be good to add this for CMake-D and the D support in Meson.
> 
> I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff).

The argument is not so much about optimization (which can be obtained via LTO as well), but primarily about build speed.

Turns out that recompiling 10 modules at once (a whole package) is often faster than doing 3 single-module compilations when each of the 3 modules imports the same 4 modules.

  10 < 3 + 3 * 4

Sure the latter is parallelizable, but a lot of computation is wasted repeating the same work.

dmd has another build mode `-c` with multiple source files producing multiple object files, but that mode is fundamentally broken, b/c template instances are only emitted to one of the object files, leading to funny linker errors when rebuilding only some modules.

We kinda know how to fix this (without resorting to `-allinst`), but it's a lot of work, and splitting into subpackages simply works, is fast, and can be parallelized as well.

-Martin

On Mon, 2017-04-10 at 12:41 +0000, Matthias Klumpp via Digitalmars-d- announce wrote:
> 
[…]
> I am not buying the necessity of not-splitbuilding for optimizations yet. If that would be the case, how do optimizations work with projects using GCC/Clang where splitbuilding is the default and often only option (like Mesa, Linux, lots of scientific stuff).
[…]

It seems that unit-threaded cannot be built module at a time due to various reflection bits, the only solution is all source at once.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder