Hey folks,


Over the time a number of stuff has become quite duplicated across our makefiles for dmd, druntime, and phobos.

These include fetching OS and model but (newer) general-purpose macros for e.g. rebasing repos, see https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

I think it's time to reduce clutter and duplication by migrating such common stuff into one common.mak file. Question is, where should that file sit? One candidate is tools/ but it's not really a tool. Another candidate is dmd/src/ because that would be the first thing anyone depends on.

Ideas?


Andrei

On Tue, 13 Jan 2015 15:20:12 -0800
Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com>
wrote:

> Hey folks,
> 
> 
> Over the time a number of stuff has become quite duplicated across our makefiles for dmd, druntime, and phobos.
> 
> These include fetching OS and model but (newer) general-purpose macros for e.g. rebasing repos, see https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
> 
> I think it's time to reduce clutter and duplication by migrating such common stuff into one common.mak file. Question is, where should that file sit? One candidate is tools/ but it's not really a tool. Another candidate is dmd/src/ because that would be the first thing anyone depends on.
> 
> Ideas?
i thing that dmd is the right place. there is little sense in building brand new phobos without brand new dmd with all fixes and enhancements anyway, so i believe that most people who wants to rebuild phobos and druntime are already has dmd repo cloned.

besides, this is a basic thing, which is of almost no use for ldc and gdc anyway.

i vote for dmd.

On Tue, Jan 13, 2015 at 03:20:12PM -0800, Andrei Alexandrescu via Digitalmars-d wrote:
> Hey folks,
> 
> 
> Over the time a number of stuff has become quite duplicated across our makefiles for dmd, druntime, and phobos.
> 
> These include fetching OS and model but (newer) general-purpose macros for
> e.g. rebasing repos, see https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
> 
> I think it's time to reduce clutter and duplication by migrating such common stuff into one common.mak file. Question is, where should that file sit? One candidate is tools/ but it's not really a tool. Another candidate is dmd/src/ because that would be the first thing anyone depends on.
> 
> Ideas?
[...]

Actually, my first reaction is to go the opposite direction. Currently, we have way too many complicated interdependencies between different repos. For example, the 'html' target of phobos/posix.mak depends on files in the dlang.org repos, and the website build scripts of dlang.org in turn depends on files generated by the Phobos ddoc targets. This may be fine for the core devs and hard-core bleeding-edge people like myself who don't mind running off git HEAD and dealing with breakages, but it's bad for anybody else:

1) These interdependencies are currently expressed by hard-coded filesystem paths, which presumes a specific directory layout for checking out dmd, dlang.org, phobos, druntime. This means the build will break if somebody doesn't know the correct directory structure to use when checking out these repos.

2) This makes the build fragile, because any changes to makefiles that might affect these dependencies have to be merged in sync between multiple repos, e.g., a PR that changes the layout of dlang.org needs to have a corresponding PR that updates the Phobos makefiles as well. This is kinda what we already have to do with compiler-dependent changes to druntime/phobos anyway, but in the case of documentation, any oversight is not caught by the autotester and may wind up on the public-facing dlang.org website before the problem is noticed.

3) Due to these interdependencies, you always have to checkout all 4 repos (in a specific directory layout), otherwise the build will fail one way or another. Again, not a problem for core devs, but a gratuitous obstacle for the casual user who wants to try things out. (Actually, to this day I'm almost certain I *still* don't have the exact layout assumed by the build scripts, because dlang.org appears to deposit generated html files in dlang.org/web, whereas phobos/posix.mak's 'html' target generates them in ../web/, which in my setup is one directory level off. I hacked around the problem with a symlink from web to dlang.org/web, but I'm pretty sure many users who wanted to experiment with Phobos docs gave up because they didn't have the patience to figure out how to get the thing to work properly.)

Because of this, my inclination right now is to *reduce* interdependencies between these repos, rather than introduce yet more.

Either that, or we should just forget about trying to manually keep 3-4 disparate repos in sync, and just use a single repo for dmd, druntime, phobos, dlang.org, etc.. Except that some people will then complain that they just want the bare compiler and nothing else, so why should they have to clone excess baggage like druntime/phobos. *shrug* Some days you just can't win.

(Of course, the *ideal* solution is that the makefiles will tell you if something is missing and where to get it and how to set it up correctly. But I'm skeptical whether this will ever materialize.  I'm not holding my breath.)


T

-- 
Why waste time reinventing the wheel, when you could be reinventing the engine? -- Damian Conway

On 1/13/15 3:55 PM, H. S. Teoh via Digitalmars-d wrote:
> On Tue, Jan 13, 2015 at 03:20:12PM -0800, Andrei Alexandrescu via Digitalmars-d wrote:
>> Hey folks,
>>
>>
>> Over the time a number of stuff has become quite duplicated across our
>> makefiles for dmd, druntime, and phobos.
>>
>> These include fetching OS and model but (newer) general-purpose macros for
>> e.g. rebasing repos, see https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
>>
>> I think it's time to reduce clutter and duplication by migrating such
>> common stuff into one common.mak file. Question is, where should that
>> file sit? One candidate is tools/ but it's not really a tool. Another
>> candidate is dmd/src/ because that would be the first thing anyone
>> depends on.
>>
>> Ideas?
> [...]
>
> Actually, my first reaction is to go the opposite direction. Currently,
> we have way too many complicated interdependencies between different
> repos. For example, the 'html' target of phobos/posix.mak depends on
> files in the dlang.org repos, and the website build scripts of dlang.org
> in turn depends on files generated by the Phobos ddoc targets.

Yah but when you look closer it's not that bad; just an attribution tradeoff. Building the phobos docs belongs with the phobos makefile because that's where the module informations resides; however, part of the build requires macro definitions that are at best consolidated in the website directory.

Building phobos's docs without dlang.org around is not sensible, just the same as building dlang.org without phobos. So I'm not too worried about that.

If someone builds phobos without caring about the docs, that's fine as long as they don't try to build html.

One simplification would be to not set STDDOC in https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L95, and refuse to build html if STDDOC is not set from the outside. But again I see this as just minor progress.

> This may
> be fine for the core devs and hard-core bleeding-edge people like myself
> who don't mind running off git HEAD and dealing with breakages, but it's
> bad for anybody else:
>
> 1) These interdependencies are currently expressed by hard-coded
> filesystem paths, which presumes a specific directory layout for
> checking out dmd, dlang.org, phobos, druntime. This means the build will
> break if somebody doesn't know the correct directory structure to use
> when checking out these repos.

Yah, that should be in a wiki. I keep on thinking about producing a 10-minutes video "Your First Contribution to D" in which I explain step by step how to get a pull request in starting from a scratch.

We also have tools/update.sh which should probably be more prominently advertised.

But in the grand scheme of things I don't see a foolproof solution to that - you need dmd for building druntime, druntime for building phobos, etc. If you got dmd, druntime, and phobos in the wrong places then You're Gonna Have A Bad Time.

> 2) This makes the build fragile, because any changes to makefiles that
> might affect these dependencies have to be merged in sync between
> multiple repos, e.g., a PR that changes the layout of dlang.org needs to
> have a corresponding PR that updates the Phobos makefiles as well. This
> is kinda what we already have to do with compiler-dependent changes to
> druntime/phobos anyway, but in the case of documentation, any oversight
> is not caught by the autotester and may wind up on the public-facing
> dlang.org website before the problem is noticed.

I don't understand how this is a problem, how my suggestion makes it any worse, what a solution to it all would be.

> 3) Due to these interdependencies, you always have to checkout all 4
> repos (in a specific directory layout), otherwise the build will fail
> one way or another.

This is expected, no? What is an alternative? How would one e.g. build druntime without dmd?

> Again, not a problem for core devs, but a gratuitous
> obstacle for the casual user who wants to try things out. (Actually, to
> this day I'm almost certain I *still* don't have the exact layout
> assumed by the build scripts, because dlang.org appears to deposit
> generated html files in dlang.org/web, whereas phobos/posix.mak's 'html'
> target generates them in ../web/, which in my setup is one directory
> level off. I hacked around the problem with a symlink from web to
> dlang.org/web, but I'm pretty sure many users who wanted to experiment
> with Phobos docs gave up because they didn't have the patience to figure
> out how to get the thing to work properly.)

That's an inconsistency we should fix. Right now phobos is best build starting from the dlang.org build. We should probably disable phobos' own docs build.

> Because of this, my inclination right now is to *reduce*
> interdependencies between these repos, rather than introduce yet more.
>
> Either that, or we should just forget about trying to manually keep 3-4
> disparate repos in sync, and just use a single repo for dmd, druntime,
> phobos, dlang.org, etc.. Except that some people will then complain that
> they just want the bare compiler and nothing else, so why should they
> have to clone excess baggage like druntime/phobos. *shrug* Some days you
> just can't win.

Yah, that's why I think a set of simple tactical tools a la tools/update.sh are the proportional response here.

> (Of course, the *ideal* solution is that the makefiles will tell you if
> something is missing and where to get it and how to set it up correctly.
> But I'm skeptical whether this will ever materialize.  I'm not holding
> my breath.)

dlang.org does something of that kind already - "make rebase".


Andrei

"Andrei Alexandrescu"  wrote in message news:m9497d$1pv2$1@digitalmars.com...

> Over the time a number of stuff has become quite duplicated across our makefiles for dmd, druntime, and phobos.
>
> These include fetching OS and model but (newer) general-purpose macros for e.g. rebasing repos, see https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
>
> I think it's time to reduce clutter and duplication by migrating such common stuff into one common.mak file. Question is, where should that file sit? One candidate is tools/ but it's not really a tool. Another candidate is dmd/src/ because that would be the first thing anyone depends on.
>
> Ideas?

If it's optional extra stuff like git shortcuts, then removing it from the makefiles and putting it in tools makes sense.  If it's mandatory stuff like OS detection, then it needs to stay where it is.

What's the actual problem you're trying to solve with this?  IMO duplication is less of a problem than overcomplicating the makefiles. 

On 01/13/2015 09:35 PM, Andrei Alexandrescu wrote:
<snip>
>> 1) These interdependencies are currently expressed by hard-coded filesystem paths, which presumes a specific directory layout for checking out dmd, dlang.org, phobos, druntime. This means the build will break if somebody doesn't know the correct directory structure to use when checking out these repos.
> 
> Yah, that should be in a wiki. I keep on thinking about producing a 10-minutes video "Your First Contribution to D" in which I explain step by step how to get a pull request in starting from a scratch.
> 
> We also have tools/update.sh which should probably be more prominently advertised.
> 
<snip>

This shouldn't go (only) in the wiki.  It should be in the README of
every repository.  And if not in the README, then the CONTRIBUTING file.
 A minimal list of "clone these things, run make in these directories"
is fine here with a reference to the wiki for more details.

I put off some documentation updates a few months ago because it wasn't worth the investment yet to figure out how to build the website.  The wiki pages appeared to be about building the compiler, which I didn't realize I should be interested in.  This barrier is too high.

-- 
Paul O'Neil
Github / IRC: todayman

On 1/13/2015 3:20 PM, Andrei Alexandrescu wrote:
> Over the time a number of stuff has become quite duplicated across our makefiles
> for dmd, druntime, and phobos.
>
> These include fetching OS and model but (newer) general-purpose macros for e.g.
> rebasing repos, see
> https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
>
> I think it's time to reduce clutter and duplication by migrating such common
> stuff into one common.mak file. Question is, where should that file sit? One
> candidate is tools/ but it's not really a tool. Another candidate is dmd/src/
> because that would be the first thing anyone depends on.
>
> Ideas?


It's a few lines of duplication, as opposed to adding another point of failure to the build process, i.e. common.mak isn't found or the wrong one is found, etc.

I've always tended to avoid using make macro include's for this reason.

On 1/13/15 7:44 PM, Daniel Murphy wrote:
> "Andrei Alexandrescu"  wrote in message
> news:m9497d$1pv2$1@digitalmars.com...
>
>> Over the time a number of stuff has become quite duplicated across our
>> makefiles for dmd, druntime, and phobos.
>>
>> These include fetching OS and model but (newer) general-purpose macros
>> for e.g. rebasing repos, see
>> https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
>>
>>
>> I think it's time to reduce clutter and duplication by migrating such
>> common stuff into one common.mak file. Question is, where should that
>> file sit? One candidate is tools/ but it's not really a tool. Another
>> candidate is dmd/src/ because that would be the first thing anyone
>> depends on.
>>
>> Ideas?
>
> If it's optional extra stuff like git shortcuts, then removing it from
> the makefiles and putting it in tools makes sense.  If it's mandatory
> stuff like OS detection, then it needs to stay where it is.

Hmmm... I wonder why the distinction.

> What's the actual problem you're trying to solve with this?  IMO
> duplication is less of a problem than overcomplicating the makefiles.

The problem is basic code duplication with its known liabilities. I'm looking at stuff like this:

https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L58

https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L31

https://github.com/D-Programming-Language/druntime/blob/master/posix.mak#L8

Three's a charm. That code is relatively verbose (required newlines). I trust it doesn't need updates often, but it occupies real estate in a readily accessible position (beginning of file).

The real issue here is that acquiescing to such duplication discourages attempts at uniformization before they occur. I'm thinking of simpler, clearer naming conventions and uniform ways of doing the same things (such as unittesting or documentation).

Getting back to REBASE (https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37), it greatly simplifies my building the entire website in a way I trust is nice:

  make rebase -j && make clean && make rsync -j

(Sadly make clean is necessary because of dub, which makes the process take a lot longer than it should; with luck, somebody will improve on that soon.)

Now "make rebase" is really useful for druntime and phobos as well. But between duplicating more code or just doing it by hand, one may as well give up and choose the latter.

I'd also like to define "make rebase-dirty" based on this:

REBASE_DIRTY = MYBRANCH=`git rev-parse --abbrev-ref HEAD` &&\
 git stash &&\
 git co master &&\
 git pull --ff-only upstream master &&\
 git co $$MYBRANCH &&\
 git rebase master &&\
 git stash pop

which is useful because it lets me rebase on top of work in progress. Again the prospect of duplicating this other macro across 3-4 projects is fairly unattractive.



Andrei

On 1/13/15 8:29 PM, Paul O'Neil wrote:
> On 01/13/2015 09:35 PM, Andrei Alexandrescu wrote:
> <snip>
>>> 1) These interdependencies are currently expressed by hard-coded
>>> filesystem paths, which presumes a specific directory layout for
>>> checking out dmd, dlang.org, phobos, druntime. This means the build will
>>> break if somebody doesn't know the correct directory structure to use
>>> when checking out these repos.
>>
>> Yah, that should be in a wiki. I keep on thinking about producing a
>> 10-minutes video "Your First Contribution to D" in which I explain step
>> by step how to get a pull request in starting from a scratch.
>>
>> We also have tools/update.sh which should probably be more prominently
>> advertised.
>>
> <snip>
>
> This shouldn't go (only) in the wiki.  It should be in the README of
> every repository.  And if not in the README, then the CONTRIBUTING file.
>   A minimal list of "clone these things, run make in these directories"
> is fine here with a reference to the wiki for more details.
>
> I put off some documentation updates a few months ago because it wasn't
> worth the investment yet to figure out how to build the website.  The
> wiki pages appeared to be about building the compiler, which I didn't
> realize I should be interested in.  This barrier is too high.

Great ideas! I created https://issues.dlang.org/show_bug.cgi?id=13980 to track them. Thanks, Andrei

On 1/13/15 8:49 PM, Walter Bright wrote:
> It's a few lines of duplication, as opposed to adding another point of
> failure to the build process, i.e. common.mak isn't found or the wrong
> one is found, etc.

I'd have difficulty agreeing with this. If dmd is found then whatever is there is found. It's a deterministic process, not an arbitrary path lookup dependent on system vagaries.

Plus... duplication only grows.

> I've always tended to avoid using make macro include's for this reason.

I think I failed to grasp the reason.


Andrei