On Tuesday, 13 January 2015 at 23:20:13 UTC, Andrei Alexandrescu wrote:
> Hey folks,
>
>
> Over the time a number of stuff has become quite duplicated across our makefiles for dmd, druntime, and phobos.
>
> These include fetching OS and model but (newer) general-purpose macros for e.g. rebasing repos, see https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
>
> I think it's time to reduce clutter and duplication by migrating such common stuff into one common.mak file. Question is, where should that file sit? One candidate is tools/ but it's not really a tool. Another candidate is dmd/src/ because that would be the first thing anyone depends on.
>
> Ideas?
>
>
> Andrei

IMO helpers (such as git helpers) are not part of the build process, so they should not be part of the build files. Few people care about make rebase. I'm not saying we should delete them (they obviously are useful to you), but moving them to tools might be a good start.

For druntime / phobos build... That's really a shame we don't have a standard D tool to build D code, cross platform.
If we had, not only would we promote an even saner image of our tooling ('hey look, dogfooding'), we won't have to solve the same problems over and over again, so both druntime / phobos and the tool would benefit from this, but ofc it will require some initial work.
If someone wants to get this project started, I have a suggestion for the name: dub

Okay, sarcasms appart, would that be a possibility for us to switch to dub in the long run ? I'm not sure dub is ready for that ATM, but I'm confident it could be made ready without much work.

On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d wrote:
> On 2015-01-14 00:20, Andrei Alexandrescu wrote:
[...]
> >I think it's time to reduce clutter and duplication by migrating such common stuff into one common.mak file. Question is, where should that file sit? One candidate is tools/ but it's not really a tool. Another candidate is dmd/src/ because that would be the first thing anyone depends on.
> 
> Oh God, please no. I really, really hate that the makefiles are are looking in folders outside of the repositories.

+1. Makefiles that reference paths outside the repo are really really evil. They only ever work for the specific setup the author has, and breaks for everybody else.


> >Ideas?
> 
> If you really want to do something like this I would recommend creating a new repository, acting like a super repository, including the other repositories as submodules. Then you can create a new makefile that builds everything. Although, I still don't want the makefiles in the existing repositories be dependent on the new repository.
[...]

+1, this is a much better approach than introducing yet more extra-repo hard-coded paths into the makefiles. There are too many of those already.


T

-- 
Truth, Sir, is a cow which will give [skeptics] no more milk, and so they are gone to milk the bull. -- Sam. Johnson

On Wed, Jan 14, 2015 at 02:58:02AM -0800, Andrei Alexandrescu via Digitalmars-d wrote:
> On 1/14/15 2:09 AM, Daniel Murphy wrote:
[...]
> >Do enough people need to that it warrants being in the makefile instead of a local shell script?  Does everyone do it the same way?
> 
> Yes.
> 
> Currently I'm the site build master by default. In the near future there'll be several. I'd rather have anyone run a simple command than telling them to write their scripts.
> 
> So if next month I transfer this responsibility to Martin, I can tell: "to update dlang.org, run 'make rebase -j && make rsync -j'" as opposed to "yeah, I have some scripts somewhere, do your own or let me email them to you".
> 
> I don't really understand your position here. It's like you reject robust automation over tribal knowledge and folklore.

What stops us from adding said scripts to the dlang.org repo? Or tools, for that matter, which is probably more appropriate.

Not everything needs to be shoehorned into the makefiles.


> >>Overall I have difficulty understanding counterarguments.
> >>
> >>Challenge: "We should eliminate some unpleasant duplication."
> >>
> >>Response: "Make sucks. Make includes are unreliable for mythical unexplained reasons. You don't know git. You should do scripts instead."
> >>
> >>The response just doesn't follow. Care to clarify what you think we should be doing here?
> >
> >Apologies for not being clear.
> >
> >Challenge: "We should eliminate some unpleasant duplication."
> >
> >Response:  "We should avoid adding more dependencies between projects at all costs."
> 
> Why? As an aside, "at all costs" is suspicious in many contexts.

Because it imposes a specific external directory structure which the casual user has no idea about. This stuff is documented nowhere currently -- I had to read the makefiles to figure it out for myself; not many people have that much patience.  It makes the whole thing more fragile because as soon as one or more repos go out-of-sync (not on github, but in the user's local clones, which can happen very easily) things can break horribly without any warning.

I'm astonished this has to be spelled out in, of all places, a D forum. Surely, of all people, we D people must be all too familiar with exactly why things like #include <../../../mylib/mymod/myheader.h> are evil? I mean, why else would we have D modules rather than continue this glorious C heritage? Yet here we're proposing to add the equivalent of such out-of-tree #include's to our makefiles *and* we're straining to find reasons why we shouldn't do it? I'm at a loss of words to describe the cognitive dissonance here.


T

-- 
Creativity is not an excuse for sloppiness.

On 1/14/15 3:29 AM, Daniel Murphy wrote:
> "Andrei Alexandrescu"  wrote in message
> news:m95i3q$1t6$1@digitalmars.com...
>
>>
>> > I don't mind using include files, I mind adding more dependencies
>> > between projects.
>>
>> Why? Druntime depends on dmd. Phobos depends on dmd and druntime.
>> Dlang.org and tools depend on dmd, druntime, and phobos. It's a fact
>> of life.
>
> Druntime, phobos and dlang.org don't (currently) depend on dmd's source.

I see your point, though I'll note that DMD everywhere defaults to ../dmd/src/dmd (and that's a huge improvement from the time is was defined as just dmd). But granted setting it in the command line would make those independent of the dmd location.

So the question is - is there a place that's common to all?


Andrei

On 1/14/15 9:01 AM, H. S. Teoh via Digitalmars-d wrote:
> On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d wrote:
>> On 2015-01-14 00:20, Andrei Alexandrescu wrote:
> [...]
>>> I think it's time to reduce clutter and duplication by migrating such
>>> common stuff into one common.mak file. Question is, where should that
>>> file sit? One candidate is tools/ but it's not really a tool. Another
>>> candidate is dmd/src/ because that would be the first thing anyone
>>> depends on.
>>
>> Oh God, please no. I really, really hate that the makefiles are are
>> looking in folders outside of the repositories.
>
> +1. Makefiles that reference paths outside the repo are really really
> evil. They only ever work for the specific setup the author has, and
> breaks for everybody else.

Currently there must be a place where druntime finds dmd for building purposes. Also phobos needs to find druntime. Also dlang.ord needs to find the source of both phobos and druntime.

There's no specific setup of one person as much as a boring:

/some/dir/
  dmd
  druntime
  phobos
  dlang.org
  tools

I have a hard time making it any simpler.

>>> Ideas?
>>
>> If you really want to do something like this I would recommend
>> creating a new repository, acting like a super repository, including
>> the other repositories as submodules. Then you can create a new
>> makefile that builds everything. Although, I still don't want the
>> makefiles in the existing repositories be dependent on the new
>> repository.
> [...]
>
> +1, this is a much better approach than introducing yet more extra-repo
> hard-coded paths into the makefiles. There are too many of those
> already.

Where would that super-repo sit?


Andrei

On 1/14/15 8:50 AM, Mathias LANG wrote:
> IMO helpers (such as git helpers) are not part of the build process, so
> they should not be part of the build files. Few people care about make
> rebase. I'm not saying we should delete them (they obviously are useful
> to you), but moving them to tools might be a good start.

This is about scaling up and passing responsibility around. I could write a script and then tell whoever maintains dlang.org "first run that script then run make" etc.

With make it turns out it's dead simple to get parallel rebase going:

REBASE = git pull upstream master --rebase

rebase: rebase-dlang rebase-dmd rebase-druntime rebase-phobos
rebase-dlang: ; $(REBASE)
rebase-dmd: ; cd $(DMD_DIR) && $(REBASE)
rebase-druntime: ; cd $(DRUNTIME_DIR) && $(REBASE)
rebase-phobos: ; cd $(PHOBOS_DIR) && $(REBASE)

So I'm done in 6 lines and it's parallel automatically, no code needed. With zsh I'd probably go (untested)...

===========================
#/bin/zsh
set -e

function rebase() {
  ( cd $1 && git pull upstream master --rebase )
}

rebase . &&
rebase ../dmd &&
rebase ../druntime &&
rebase ../phobos &&

wait
===========================

But wait, there's less. The script:

* does not return nonzero if either subtask fails - and doing so needs e.g. temporary files and takes extra code. So I need to carefully look at the textual garbled output to make sure it all works.

* does not accept setting variables DMD_DIR etc. It can be made to, but that's yet more code.

* cannot run serially if I want to debug something, only parallel. It can be made do, but that's again more code!

So we're looking at an ideological response "don't use make, make sucks" to a pragmatic matter - I want to get work done, and we shouldn't use inferior/more intensive technical solutions on an ideological basis alone. I agree make sucks! But that doesn't suddenly eliminate my common sense.

> For druntime / phobos build... That's really a shame we don't have a
> standard D tool to build D code, cross platform.
> If we had, not only would we promote an even saner image of our tooling
> ('hey look, dogfooding'), we won't have to solve the same problems over
> and over again, so both druntime / phobos and the tool would benefit
> from this, but ofc it will require some initial work.
> If someone wants to get this project started, I have a suggestion for
> the name: dub
>
> Okay, sarcasms appart, would that be a possibility for us to switch to
> dub in the long run ? I'm not sure dub is ready for that ATM, but I'm
> confident it could be made ready without much work.

I'd be glad to switch to dub but I confess at this time I can't invest the time. I'm already spread too thin.


Andrei

On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
> What stops us from adding said scripts to the dlang.org repo? Or tools,
> for that matter, which is probably more appropriate.
>
> Not everything needs to be shoehorned into the makefiles.

As I explained in my previous message it's a sensible solution. -- Andrei

On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
> Because it imposes a specific external directory structure which the
> casual user has no idea about.

I think that's a given already and has been for a long time. -- Andrei

On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
> I'm astonished this has to be spelled out in, of all places, a D forum.
> Surely, of all people, we D people must be all too familiar with exactly
> why things like #include <../../../mylib/mymod/myheader.h> are evil?

I think that would be applying one canned lesson to a situation where it doesn't quite apply. -- Andrei

On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
> What stops us from adding said scripts to the dlang.org repo? Or tools,
> for that matter, which is probably more appropriate.

Oh, I just figured that would make everything dependent on dlang.org. That would be a new dependency :o). -- Andrei