On 8/11/2013 3:18 PM, Jonathan M Davis wrote:
> On Sunday, August 11, 2013 14:43:13 Walter Bright wrote:
>> That said, as soon as the D *package* starts to depend on
>> non-default-installed libraries, trouble happens. With libcurl, the only
>> solution so far seems to be to BUILD OUR OWN LIBCURL binary!
>
> At this point, I'm inclined to think that while it's great for us to have
> bindings to C libraries and to have user-friendly, D wrappers around them,
> it's better that they don't end up in Phobos.

My sentiments exactly.

On Sun, Aug 11, 2013 at 12:11:14AM -0700, Jonathan M Davis wrote:
> On Saturday, August 10, 2013 22:48:14 Walter Bright wrote:
> > On 8/10/2013 4:21 PM, Jonathan M Davis wrote:
> > > Another suggestion that I kind of liked was to just build them all with a single script written in D and ditch make entirely, which would seriously reduce the amount of duplication across platforms. But that's obviously a much bigger change and would likely be much more controversial than simply using a more standard make.
> > 
> > I don't see much point in that. The dmd build is straightforward, and I see no particular gain from reinventing that wheel.
> 
> Well, make is horrible, and while posix.mak is way better than win32.mak or win64.mak, it's still pretty bad. Personally, I would never use make without something like cmake in front of it. If we were to write up something in D, it could be properly cross-platform (so only one script instead of 3+), and I fully expect that it could be far, far cleaner than what we're forced to do in make.
[...]

Maybe my previous post didn't get the idea across clearly, so let me try again. My underlying thrust was: instead of maintaining 3 different makefiles (or more) by hand, have a single source for all of them, and write a small D program to generate posix.mak, win32.mak, win64.mak, whatever, from that source.

That way, adding/removing files from the build, etc., involves only editing a single file, and regenerating the makefiles/whatever we use. If there's a problem with a platform-specific makefile, then it's just a matter of fixing the platform-specific output handler in the D program.

The way we're currently doing it essentially amounts to the same thing as copy-n-pasting the same piece of code 3 times and trying to maintain all 3 copies separately, instead of writing a template that can be specialized 3 times, thus avoiding boilerplate and maintenance headaches.


T

-- 
For every argument for something, there is always an equal and opposite argument against it. Debates don't give answers, only wounded or inflated egos.

Walter Bright:

> as soon as the D *package* starts to depend on non-default-installed libraries, trouble happens. With libcurl, the only solution so far seems to be to BUILD OUR OWN LIBCURL binary!
>
> http://d.puremagic.com/issues/show_bug.cgi?id=10710
>
> This is a terrible situation.

For Haskell they release two different kinds of compilers+libraries: one is just a core distribution with the compiler with the standard Haskell modules (including the GMP compiled binaries), and the other contains the compiler with its standard library, plus modules+binaries for the most common libraries.

Python on Windows uses a similar strategy.


> Consider things like the trig functions. D started out by forwarding to the C versions. Unfortunately, the C versions are of spotty, unreliable quality (even today!). Because of that, we've been switching to our own implementations.
>
> And, consider that using GMP means CTFE would not be supported.

At the moment BigInt doesn't run at compile-time.

You could wrap an external fast multi-precision library in Phobos D code that uses __ctfw to switch to a simpler pure D implementation at compile-time.

Is it useful to use BigInts at compile-time? If the answer is very positive then perhaps the D interpreter could be modified to allow calling external numerical libraries even at compile-time.


> Note that D was developed with existing backends and linkers.

But isn't optlink being rewritten in C? Perhaps I am just confused, sorry.

Bye,
bearophile

On Sun, Aug 11, 2013 at 09:26:11AM +0100, Russel Winder wrote:
> On Sat, 2013-08-10 at 14:27 -0400, Nick Sabalausky wrote: […]
> > is discovering and dealing with all the fun little differences between the posix and win32 makefiles (and now we have some win64 makefiles as well).
> […]
> 
> Isn't this sort of problem solved by using SCons, Waf or (if you
> really have to) CMake?
[...]

+1. But people around here seems to have a beef against anything that isn't make. *shrug*


T

-- 
If you think you are too small to make a difference, try sleeping in a closed room with a mosquito. -- Jan van Steenbergen

On 8/11/2013 3:40 PM, bearophile wrote:
> For Haskell they release two different kinds of compilers+libraries: one is just
> a core distribution with the compiler with the standard Haskell modules
> (including the GMP compiled binaries), and the other contains the compiler with
> its standard library, plus modules+binaries for the most common libraries.
>
> Python on Windows uses a similar strategy.

This is not really a strategy, it addresses none of the issues I raised.


> Is it useful to use BigInts at compile-time? If the answer is very positive then
> perhaps the D interpreter could be modified to allow calling external numerical
> libraries even at compile-time.

Don keeps extending CTFE to make it work with more stuff, as people find it more and more useful to do things at compile time. I see no reason BigInt should be excluded from that.


>> Note that D was developed with existing backends and linkers.
>
> But isn't optlink being rewritten in C? Perhaps I am just confused, sorry.

Optlink was used for D because it was existing, free, and it worked. You seemed to have the idea that optlink was developed to use with D. Optlink predated D by 12-15 years.

On Sunday, August 11, 2013 15:38:09 H. S. Teoh wrote:
> Maybe my previous post didn't get the idea across clearly, so let me try again. My underlying thrust was: instead of maintaining 3 different makefiles (or more) by hand, have a single source for all of them, and write a small D program to generate posix.mak, win32.mak, win64.mak, whatever, from that source.
> 
> That way, adding/removing files from the build, etc., involves only editing a single file, and regenerating the makefiles/whatever we use. If there's a problem with a platform-specific makefile, then it's just a matter of fixing the platform-specific output handler in the D program.
> 
> The way we're currently doing it essentially amounts to the same thing as copy-n-pasting the same piece of code 3 times and trying to maintain all 3 copies separately, instead of writing a template that can be specialized 3 times, thus avoiding boilerplate and maintenance headaches.

But if you're going that far, why not just do the whole thing with D and ditch make entirely? If it's to avoid bootstrapping issues, we're going to have those anyway once we move the compiler to D (which is well on is well underway), so that really isn't going to matter.

- Jonathan M Davis

On Sun, Aug 11, 2013 at 06:14:18PM -0700, Jonathan M Davis wrote:
> On Sunday, August 11, 2013 15:38:09 H. S. Teoh wrote:
> > Maybe my previous post didn't get the idea across clearly, so let me try again. My underlying thrust was: instead of maintaining 3 different makefiles (or more) by hand, have a single source for all of them, and write a small D program to generate posix.mak, win32.mak, win64.mak, whatever, from that source.
> > 
> > That way, adding/removing files from the build, etc., involves only editing a single file, and regenerating the makefiles/whatever we use.  If there's a problem with a platform-specific makefile, then it's just a matter of fixing the platform-specific output handler in the D program.
> > 
> > The way we're currently doing it essentially amounts to the same thing as copy-n-pasting the same piece of code 3 times and trying to maintain all 3 copies separately, instead of writing a template that can be specialized 3 times, thus avoiding boilerplate and maintenance headaches.
> 
> But if you're going that far, why not just do the whole thing with D and ditch make entirely? If it's to avoid bootstrapping issues, we're going to have those anyway once we move the compiler to D (which is well on is well underway), so that really isn't going to matter.
[...]

If you like, think of it this way: the build tool will be written in D, with the option of generating scripts in legacy formats like makefiles or shell scripts so that it can be bootstrapped by whoever needs it to.

We pay zero cost for this because the source document is the input format for the D tool, and the D tool takes care of producing the right sequence of commands. There is only one place to update when new files need to be added or old files removed -- or, if we integrate it with rdmd fully, even this may not be necessary. When somebody asks for a makefile, we just run the program with --generate=makefile. When somebody asks for a shell script, we just run it with --generate=shellscript. The generated makefiles/shell scripts are guaranteed to be consistent with the current state of the code, which is the whole point behind this exercise.


T

-- 
Designer clothes: how to cover less by paying more.

On Sunday, August 11, 2013 20:07:17 H. S. Teoh wrote:
> On Sun, Aug 11, 2013 at 06:14:18PM -0700, Jonathan M Davis wrote:
> > On Sunday, August 11, 2013 15:38:09 H. S. Teoh wrote:
> > > Maybe my previous post didn't get the idea across clearly, so let me try again. My underlying thrust was: instead of maintaining 3 different makefiles (or more) by hand, have a single source for all of them, and write a small D program to generate posix.mak, win32.mak, win64.mak, whatever, from that source.
> > > 
> > > That way, adding/removing files from the build, etc., involves only editing a single file, and regenerating the makefiles/whatever we use.  If there's a problem with a platform-specific makefile, then it's just a matter of fixing the platform-specific output handler in the D program.
> > > 
> > > The way we're currently doing it essentially amounts to the same thing as copy-n-pasting the same piece of code 3 times and trying to maintain all 3 copies separately, instead of writing a template that can be specialized 3 times, thus avoiding boilerplate and maintenance headaches.
> > 
> > But if you're going that far, why not just do the whole thing with D and ditch make entirely? If it's to avoid bootstrapping issues, we're going to have those anyway once we move the compiler to D (which is well on is well underway), so that really isn't going to matter.
> 
> [...]
> 
> If you like, think of it this way: the build tool will be written in D, with the option of generating scripts in legacy formats like makefiles or shell scripts so that it can be bootstrapped by whoever needs it to.
> 
> We pay zero cost for this because the source document is the input format for the D tool, and the D tool takes care of producing the right sequence of commands. There is only one place to update when new files need to be added or old files removed -- or, if we integrate it with rdmd fully, even this may not be necessary. When somebody asks for a makefile, we just run the program with --generate=makefile. When somebody asks for a shell script, we just run it with --generate=shellscript. The generated makefiles/shell scripts are guaranteed to be consistent with the current state of the code, which is the whole point behind this exercise.

But what is the point of the makefile? As far as I can see, it gains you nothing. It doesn't help bootstrapping at all, because dmd itself will soon be written in D and therefore require that a D compiler already be installed. And the cost definitely isn't zero, because it requires extra code to be able to generate a makefile on top of doing the build purely with the D script (and I fully expect that donig the build with the D script will be simpler than generating the makefile would be). I see no benefit whatsoever in generating makefiles from a D script over simply doing the whole build with the D script. There would be an argument for it if dmd itself were going to stay in C++, because then you could avoid a circular dependency, but dmd is being converted to D, and so we're going to have that circular dependency anyway, negating that argument.

- Jonathan M Davis

On Sun, 2013-08-11 at 15:41 -0700, H. S. Teoh wrote:
> On Sun, Aug 11, 2013 at 09:26:11AM +0100, Russel Winder wrote:
> > On Sat, 2013-08-10 at 14:27 -0400, Nick Sabalausky wrote: […]
> > > is discovering and dealing with all the fun little differences between the posix and win32 makefiles (and now we have some win64 makefiles as well).
> > […]
> > 
> > Isn't this sort of problem solved by using SCons, Waf or (if you
> > really have to) CMake?
> [...]
> 
> +1. But people around here seems to have a beef against anything that isn't make. *shrug*

Make was a revolution and a revelation in 1977, it changed my life. However, it is sad to see projects such as Rust, Julia and D clinging to a 35 year old build concept when it has been proved time and time again that external DSL frameworks for build do not work for cross-platform working. Only internal DSL build frameworks have succeeded in that arena, cf. Gradle, SBT, SCons, Waf,…

The only part of this thread that has any up side at all is to ditch all build frameworks and write the build in D over the bootstrap D that will be essential for the D build since D is written in D. It's a pity Rust hasn't twigged to this.

I note that the Go tooling is written is C and Go, they ditched make when they realized their vision for packaging – which works very well indeed, particularly pulling in source packages from GitHub, BitBucket and Launchpad, compiling and installing the compiled package into the appropriate place for use.

On the other hand, I bet a cross-platform SCons build of D could be in place and production within days as opposed to the <substitute-your-favourite-long-time> that a D rewrite in D will take. It doesn't matter than the SCons build may be thrown away down the line, it solves a problem now for not that much effort.

Still if the core D community are clinging to "build == make", then they will have to suffer the irritant of having to have a separate build system for each and every platform. That's they way Make is.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Mon, Aug 12, 2013 at 11:16:19AM +0100, Russel Winder wrote:
> On Sun, 2013-08-11 at 15:41 -0700, H. S. Teoh wrote:
> > On Sun, Aug 11, 2013 at 09:26:11AM +0100, Russel Winder wrote:
> > > On Sat, 2013-08-10 at 14:27 -0400, Nick Sabalausky wrote: […]
> > > > is discovering and dealing with all the fun little differences between the posix and win32 makefiles (and now we have some win64 makefiles as well).
> > > […]
> > > 
> > > Isn't this sort of problem solved by using SCons, Waf or (if you
> > > really have to) CMake?
> > [...]
> > 
> > +1. But people around here seems to have a beef against anything that isn't make. *shrug*
> 
> Make was a revolution and a revelation in 1977, it changed my life. However, it is sad to see projects such as Rust, Julia and D clinging to a 35 year old build concept when it has been proved time and time again that external DSL frameworks for build do not work for cross-platform working. Only internal DSL build frameworks have succeeded in that arena, cf. Gradle, SBT, SCons, Waf,…

+1. If I were the one making the decisions, I'd go for SCons. Or tup (http://gittup.org/tup/), but tup seems to be currently posix-specific, so SCons still wins if you want cross-platform building.


> The only part of this thread that has any up side at all is to ditch all build frameworks and write the build in D over the bootstrap D that will be essential for the D build since D is written in D. It's a pity Rust hasn't twigged to this.

I think the D build tool should extend / be built on top of rdmd to be able to handle non-D sources. Once we have that, we basically already have a working build system.


> I note that the Go tooling is written is C and Go, they ditched make when they realized their vision for packaging – which works very well indeed, particularly pulling in source packages from GitHub, BitBucket and Launchpad, compiling and installing the compiled package into the appropriate place for use.

I ditched make about a decade ago, and I would never go back if I had the choice. Sadly, most of the rest of the world still seems stuck in that quagmirem, unable to move on.


> On the other hand, I bet a cross-platform SCons build of D could be in place and production within days as opposed to the <substitute-your-favourite-long-time> that a D rewrite in D will take. It doesn't matter than the SCons build may be thrown away down the line, it solves a problem now for not that much effort.

What do you say? Let's throw together an SConstruct for DMD, druntime, and phobos, and submit a pull for it?

The only downside is that I can predict people will start complaining about the Python dependency. (Which is why I proposed writing a build system in D -- it will be superior to make (anything would be!), and people will have no excuse about what language it's written in.)


> Still if the core D community are clinging to "build == make", then they will have to suffer the irritant of having to have a separate build system for each and every platform. That's they way Make is.
[...]

I used to evangelize SCons to everybody I meet... but after people adamantly refused to abandon their precious outdated crappy makefiles, I gave up. If they wish to continue suffering, it's not really my business to stop them.


T

-- 
WINDOWS = Will Install Needless Data On Whole System -- CompuMan