On Wednesday, 24 September 2014 at 19:26:46 UTC, Jacob Carlborg
wrote:
> On 2014-09-24 12:16, Walter Bright wrote:
>
>> I've never heard of a non-trivial project that didn't have constant
>> breakage of its build system. All kinds of reasons - add a file, forget
>> to add it to the manifest. Change the file contents, neglect to update
>> dependencies. Add new dependencies on some script, script fails to run
>> on one configuration. And on and on.
>
> Again, if changing the file contents breaks the build system you're doing it very, very wrong.

People do it very, very wrong all the time - that's the problem
:)  Build systems are felt by most developers to be a tax they
have to pay to do what they want to do, which is write code and
solve non-build-related problems.  Unfortunately, build
engineering is effectively a specialty of its own when you step
outside the most trivial of systems.  It's really no surprise how
few people can get it right - most people can't even agree on
what a build system is supposed to do...

On Wed, Sep 24, 2014 at 12:30:23PM -0700, Andrei Alexandrescu via Digitalmars-d wrote:
> On 9/24/14, 12:20 PM, H. S. Teoh via Digitalmars-d wrote:
> >On Wed, Sep 24, 2014 at 03:16:58AM -0700, Walter Bright via Digitalmars-d wrote:
[...]
> >>Building of the compiler/library itself is stable because the autotester won't pass it if they won't build. That isn't the problem - the problem is the package scripts fail. (This is why I want the package building to be part of the autotester.)
> >
> >That's a good idea. Packaging the compiler toolchain should be automated so that we don't have a packaging crisis every other release when inevitably some script fails to do what we thought it would, or git got itself into one of those wonderful obscure strange states that only an expert can untangle.
> 
> We of course agree on all these good things but it's all vacuous unless somebody champions it.
[...]

Wasn't Nick Sabalausky working on an automated (or automatable) packaging script some time ago? Whatever happened with that?


T

-- 
"Real programmers can write assembly code in any language. :-)" -- Larry Wall

On Wednesday, 24 September 2014 at 18:46:29 UTC, H. S. Teoh via Digitalmars-d wrote:
> On Tue, Sep 23, 2014 at 10:37:59PM -0700, Walter Bright via Digitalmars-d wrote:
>> On 9/23/2014 10:10 PM, H. S. Teoh via Digitalmars-d wrote:
>> >Yeah, I wish that at least *some* attention would be paid to refining
>> >existing features so that problematic corner cases could be ironed
>> >out.
>> 
>> It's kinda maddening to hear statements like that. Just in 2.066:
>> 
>> 103 compiler regressions fixed
>> 235 compiler bugs fixed
>> 39 language enhancements
>> 12 phobos regressions fixed
>> 110 phobos bugs fixed
>> 41 phobos enhancements
>> 9 druntime regressions fixed
>> 17 druntime bugs fixed
>> 9 druntime enhancements
>> 
>> https://dlang.org/changelog.html#list2066
>> 
>> 
>> >Like identifier lookup rules for local imports.
>> 
>> Suddenly this issue goes to a mountain overnight. Is it really the
>> most critical, important problem, overshadowing everything else?
>
> No, I just named it as a representative case of many such wrinkles
> within existing language features. The fact of the matter is, wherever
> you turn, there's always something else that hasn't been fully ironed
> out yet. Features that interact with each other in unexpected ways.
> Corner cases that weren't considered / are hard to fix due to the nature
> of the features involved. Fixes that require a decision -- which are
> often neglected because there are too many other things being worked on.
>
> Sometimes I wish there were less features in D, but far more refined.
> I'd rather go without the myriad of awesome features in D if I can only
> have a small set of features that have been fully worked out such that
> there are no nasty corner cases, deep-seated compiler bugs, or
> surprising gotchas that lurk around the corner as soon as you start
> writing non-trivial code.
>
>
>> >And what to do about dtors. And so many little niggling details that
>> >seem minor, but added together, can form a pretty big mountain of
>> >frustration sometimes.
>> 
>> So help out!
>
> I am, as you yourself point out later. But it's frustrating when pull
> requests sit in the queue for weeks (sometimes months, or, in the case
> of dmd pulls, *years*) without any indication of whether it's on the
> right track, and dismaying when your PR is just one of, oh, 100+ others
> that also all need attention, many of which are just languishing there
> for lack of attention even though there is nothing obviously blocking
> them, except perhaps the reviewers' / committers' time / interest.
>
> The situation with Phobos has improved dramatically, thanks to a
> well-timed rant some months ago, which triggered a coordinated effort of
> aggressive merging, pinging, reviewing, etc. -- we've managed to cut the
> Phobos PR queue from around 90+ to around 29 as of last night (from 4+
> pages on github to only barely 2 pages). For that, I applaud my fellow
> Phobos reviewers, and I hope the trend will continue until the Phobos PR
> queue is firmly back at 1 page (<=25 open PRs).
>
> Unfortunately, the problem persists in druntime, dlang.org, and dmd. It
> feels like there's a forest fire raging and only a handful of
> firefighters, and now we want to add more fires (new development
> directions) without adding more people. What about reviewing and merging
> / rejecting the 100+ PRs in the dmd queue, most of which contain fixes
> and language improvements that people have been waiting for, for a long
> time, before we think about new directions? Some PRs appear to fix bugs
> opened *years* ago, and yet nothing is done about them. Some PRs are at
> an impasse due to decisions that need to be made, yet nobody is making
> said decisions or even discussing them, and the PRs just continue to rot
> there. There are already a *ton* of new features / fixes / language
> improvements that are waiting to be decided upon (and, given the size of
> the dmd PR queue, I submit this is no exaggeration), and yet we are
> indifferent and instead look away to new directions.
>
> Many of us are indeed ready to help, but it would *really* be nice if
> our help is received more readily, or, at the very least, *some*
> indication is shown that more effort will be put into reviewing said
> help before more energy is expended in new directions. The dramatic
> shortening of the Phobos PR queue over the last 2-3 months proves that
> this is not only possible, but also beneficial (more fixes are making it
> into Phobos than ever before) and improves morale (people are more
> likely to contribute if they don't have to wait 3 weeks before getting
> any feedback for their contribution) -- if we would only put our efforts
> into it. So here's to hoping that the other PR queues will shorten
> quickly in the near future. ;-)
>
> And mind you, the lengths of the PR queues are only the tip of the
> iceberg of stuff we ought to finish doing before embarking on the next
> new direction. There are a ton of old bugs that need attention, and a
> ton of language features that need improvement / refinement. We could at
> least pay those *some* heed even if we absolutely have to start
> something new right now at this moment. It would be a great tragedy if D
> goes down in history as the project that had so many great ideas, none
> of them carried out to completion.
>
>
> T

Being an outsider, I haven't wanted to jump into this discussion, but can't hold back after reading your post. The things you're talking about here are what was going through my mind a few days ago when Andrei said the only thing that has value is C++ compatibility. That doesn't make sense to me when you've just released a compiler with a bunch of regressions and there are all these other issues.

I'm happy that someone is working on valuable new long-term projects, but I don't see how anyone could describe the things you've listed as being unworthy of attention.

On Wed, Sep 24, 2014 at 07:36:05PM +0000, Cliff via Digitalmars-d wrote:
> On Wednesday, 24 September 2014 at 19:26:46 UTC, Jacob Carlborg wrote:
> >On 2014-09-24 12:16, Walter Bright wrote:
> >
> >>I've never heard of a non-trivial project that didn't have constant breakage of its build system. All kinds of reasons - add a file, forget to add it to the manifest. Change the file contents, neglect to update dependencies. Add new dependencies on some script, script fails to run on one configuration. And on and on.
> >
> >Again, if changing the file contents breaks the build system you're doing it very, very wrong.
> 
> People do it very, very wrong all the time - that's the problem :) Build systems are felt by most developers to be a tax they have to pay to do what they want to do, which is write code and solve non-build-related problems.

That's unfortunate indeed. I wish I could inspire them as to how cool a properly-done build system can be. Automatic parallel building, for example. Fully-reproducible, incremental builds (never ever do `make clean` again). Automatic build + packaging in a single command. Incrementally *updating* packaging in a single command. Automatic dependency discovery. And lots more. A lot of this technology actually already exists. The problem is that still too many people think "make" whenever they hear "build system".  Make is but a poor, antiquated caricature of what modern build systems can do. Worse is that most people are resistant to replacing make because of inertia. (Not realizing that by not throwing out make, they're subjecting themselves to a lifetime of unending, unnecessary suffering.)


> Unfortunately, build engineering is effectively a specialty of its own when you step outside the most trivial of systems.  It's really no surprise how few people can get it right - most people can't even agree on what a build system is supposed to do...

It's that bad, huh?

At its most fundamental level, a build system is really nothing but a dependency management system. You have a directed, acyclic graph of objects that are built from other objects, and a command which takes said other objects as input, and produces the target object(s) as output. The build system takes as input this dependency graph, and runs the associated commands in topological order to produce the product(s). A modern build system can parallelize independent steps automatically. None of this is specific to compiling programs, in fact, it works for any process that takes a set of inputs and incrementally derives intermediate products until the final set of products are produced.

Although the input is the (entire) dependency graph, it's not desirable to specify this graph explicitly (it's far too big in non-trivial projects); so most build systems offer ways of automatically deducing dependencies. Usually this is done by scanning the inputs, and modern build systems would offer ways for the user to define new scanning methods for new input types.  One particularly clever system, Tup (http://gittup.org/tup/), uses OS call proxying to discover the *exact* set of inputs and outputs for a given command, including hidden dependencies (like reading a compiler configuration file that may change compiler behaviour) that most people don't even know about.

It's also not desirable to have to derive all products from its original inputs all the time; what hasn't changed shouldn't need to be re-processed (we want incremental builds).  So modern build systems implement some way of detecting when a node in the dependency graph has changed, thereby requiring all derived products downstream to be rebuilt. The most unreliable method is to scan for file change timestamps (make). A reliable (but slow) method is to compare file hash checksums.  Tup uses OS filesystem change notifications to detect changes, thereby cutting out the scanning overhead, which can be quite large in complex projects (but it may be unreliable if the monitoring daemon isn't running / after rebooting).

These are all just icing on the cake; the fundamental core of a build system is basically dependency graph management.


T

-- 
Political correctness: socially-sanctioned hypocrisy.

On Wednesday, 24 September 2014 at 20:12:40 UTC, H. S. Teoh via
Digitalmars-d wrote:
> On Wed, Sep 24, 2014 at 07:36:05PM +0000, Cliff via Digitalmars-d wrote:
>> On Wednesday, 24 September 2014 at 19:26:46 UTC, Jacob Carlborg
>> wrote:
>> >On 2014-09-24 12:16, Walter Bright wrote:
>> >
>> >>I've never heard of a non-trivial project that didn't have constant
>> >>breakage of its build system. All kinds of reasons - add a file,
>> >>forget to add it to the manifest. Change the file contents, neglect
>> >>to update dependencies. Add new dependencies on some script, script
>> >>fails to run on one configuration. And on and on.
>> >
>> >Again, if changing the file contents breaks the build system you're
>> >doing it very, very wrong.
>> 
>> People do it very, very wrong all the time - that's the problem :)
>> Build systems are felt by most developers to be a tax they have to pay
>> to do what they want to do, which is write code and solve
>> non-build-related problems.
>
> That's unfortunate indeed. I wish I could inspire them as to how cool a
> properly-done build system can be. Automatic parallel building, for
> example. Fully-reproducible, incremental builds (never ever do `make
> clean` again). Automatic build + packaging in a single command.
> Incrementally *updating* packaging in a single command. Automatic
> dependency discovery. And lots more. A lot of this technology actually
> already exists. The problem is that still too many people think "make"
> whenever they hear "build system".  Make is but a poor, antiquated
> caricature of what modern build systems can do. Worse is that most
> people are resistant to replacing make because of inertia. (Not
> realizing that by not throwing out make, they're subjecting themselves
> to a lifetime of unending, unnecessary suffering.)
>
>
>> Unfortunately, build engineering is effectively a specialty of its own
>> when you step outside the most trivial of systems.  It's really no
>> surprise how few people can get it right - most people can't even
>> agree on what a build system is supposed to do...
>
> It's that bad, huh?
>
> At its most fundamental level, a build system is really nothing but a
> dependency management system. You have a directed, acyclic graph of
> objects that are built from other objects, and a command which takes
> said other objects as input, and produces the target object(s) as
> output. The build system takes as input this dependency graph, and runs
> the associated commands in topological order to produce the product(s).
> A modern build system can parallelize independent steps automatically.
> None of this is specific to compiling programs, in fact, it works for
> any process that takes a set of inputs and incrementally derives
> intermediate products until the final set of products are produced.
>
> Although the input is the (entire) dependency graph, it's not desirable
> to specify this graph explicitly (it's far too big in non-trivial
> projects); so most build systems offer ways of automatically deducing
> dependencies. Usually this is done by scanning the inputs, and modern
> build systems would offer ways for the user to define new scanning
> methods for new input types.  One particularly clever system, Tup
> (http://gittup.org/tup/), uses OS call proxying to discover the *exact*
> set of inputs and outputs for a given command, including hidden
> dependencies (like reading a compiler configuration file that may change
> compiler behaviour) that most people don't even know about.
>
> It's also not desirable to have to derive all products from its original
> inputs all the time; what hasn't changed shouldn't need to be
> re-processed (we want incremental builds).  So modern build systems
> implement some way of detecting when a node in the dependency graph has
> changed, thereby requiring all derived products downstream to be
> rebuilt. The most unreliable method is to scan for file change
> timestamps (make). A reliable (but slow) method is to compare file hash
> checksums.  Tup uses OS filesystem change notifications to detect
> changes, thereby cutting out the scanning overhead, which can be quite
> large in complex projects (but it may be unreliable if the monitoring
> daemon isn't running / after rebooting).
>
> These are all just icing on the cake; the fundamental core of a build
> system is basically dependency graph management.
>
>
> T

Yes, Google in fact implemented must of this for their internal
build systems, I am led to believe.  I have myself written such a
system before.  In fact, the first project I have been working on
in D is exactly this, using OS call interception for
validating/discovering dependencies, building execution graphs,
etc.

I haven't seen TUP before, thanks for pointing it out.

On 9/24/14, 1:10 PM, H. S. Teoh via Digitalmars-d wrote:
> That's unfortunate indeed. I wish I could inspire them as to how cool a
> properly-done build system can be.
[snip]

That's all nice. However: (1) the truth is there's no clear modern build tool that has "won" over make; oh there's plenty of them, but each has its own quirks that makes it tenuous to use; (2) any build system for a project of nontrivial size needs a person/team minding it - never saw such a thing as it's just all automated and it works; (3) especially if the build system is not that familiar, the role of the build czar is all the more important.

So the reality is quite a bit more complicated than the shiny city on a hill you describe.


Andrei

On Wed, 24 Sep 2014 13:16:23 +0000
Paolo Invernizzi via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> This is starting to be a little offensive...
sorry, i don't meant to. excuse me if i'm getting rude and iffensive.

On 9/24/14, 2:14 PM, ketmar via Digitalmars-d wrote:
> On Wed, 24 Sep 2014 13:16:23 +0000
> Paolo Invernizzi via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
>> This is starting to be a little offensive...
> sorry, i don't meant to. excuse me if i'm getting rude and iffensive.

Much appreciated. -- Andrei

On Wed, 24 Sep 2014 07:44:38 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com>
wrote:

> Your guidance of my career is uncalled for.
excuse me, i'm not trying to tell you what to do. neither i was trying to say that you are forced to work on the features you don't want. same for Walter. it was a bad example and i missed the thing i wanted to highlight.

> This is Walter's and my vision. He is working on C++ interop, and I am working on C++ interop. To the extent possible we hope the larger community will share this vision and help us to implement it.
from my side i'm trying to at least not hijacking technical threads (sorry if i sent some of my rants to some of them, it's by accident). this is the maximum of my possible help in this area.

On Wed, 24 Sep 2014 13:10:46 -0700
"H. S. Teoh via Digitalmars-d" <digitalmars-d@puremagic.com> wrote:

> The problem is that still too many people think "make"
> whenever they hear "build system".  Make is but a poor, antiquated
> caricature of what modern build systems can do. Worse is that most
> people are resistant to replacing make because of inertia. (Not
> realizing that by not throwing out make, they're subjecting themselves
> to a lifetime of unending, unnecessary suffering.)
+many. 'make' is a PITA even for small personal projects. i myself using heavily modified jam as build system. it's not shiny good too, but it is at least usable without much suffering and small enough to be managed by one man.