https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #9 from Andrei Alexandrescu <andrei@erdani.com> ---
(In reply to Ketmar Dark from comment #7)
> (In reply to Andrei Alexandrescu from comment #5)
> > Again: hoisting this into the compiler lowers the barrier of entry and simplifies the life of anyone building D code with .di files.
> again: no, it's not.

Just stating it won't make it true. I'd appreciate if you stopped these content-less comments, they are disruptive and paralyze any progress in the matter. Feel free to substantiate the points you're making. Thanks.

--

https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #10 from Ketmar Dark <ketmar@ketmar.no-ip.org> ---
(In reply to Andrei Alexandrescu from comment #9)
> Just stating it won't make it true.
same for you. all this conversation boils down to "i believe that this will help." "and i don't." "and i believe." "and i don't."

what build tools do you mean? is there any sane build tool that can't cope with (re)generating ".di" from ".d"? why compiler should do the work for which build tools were built?

it's compiler work to (re)generate everything we asked it to. and it's build tool's work to minimize compiler invocations by doing various checks. good build tool is perfectly able to notice that we need both new ".di" and new ".o" files and do it with one exec().

also imagine that i'm doing "force rebuilding". we'll need new command line flag that which tells dmd to regenerate .di file even if it's source .d file was not changed. sure we can workaround that by teaching build tool to call 'touch' or something, but for what reason?

either we have full-blown build tool in the compiler itself or we don't try to teach compiler some tricks which build tool should do and then start fighting with compiler to stop it tricking.

i'm on for the concept of "cached ASTs" (something like delphi's .dcu files) and teaching dmd to rebuild all changed modules and updating cached ASTs. this will greatly imporoves compilation speed (no need to do lexing, parsing and semantic analyzis again and again) and will become a nice and simple build system. but this is a very complex change and it's completely independent of ones you proposing.

--

https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #11 from Andrei Alexandrescu <andrei@erdani.com> ---
(In reply to Ketmar Dark from comment #10)
> (In reply to Andrei Alexandrescu from comment #9)
> > Just stating it won't make it true.
> same for you. all this conversation boils down to "i believe that this will help." "and i don't." "and i believe." "and i don't."

I've explained my reasons: it's the right place to put this kind of logic because it simplifies the use of .di files for everyone.

> what build tools do you mean? is there any sane build tool that can't cope with (re)generating ".di" from ".d"? why compiler should do the work for which build tools were built?

There is indeed some complication. make must be invoked two times in order to first generate and then use the dependencies. This is because make has separate stages for reading dependencies and building stuff; at work we used to drive two calls to make, but more recently we have a build system (actually two) that know how to maintain dependencies dynamically.

It is customary for today's advanced languages to provide dependency information for build tools, and D is no exception. Maintaining dependencies manually by tools that are poorly suited to grok language semantics does not scale well.

> it's compiler work to (re)generate everything we asked it to. and it's build tool's work to minimize compiler invocations by doing various checks. good build tool is perfectly able to notice that we need both new ".di" and new ".o" files and do it with one exec().

This view is rather simplistic. The "various checks" depend on how the language manages dependencies and need to be helped by language-specific tools, of which the most obvious is the compiler itself.

> also imagine that i'm doing "force rebuilding". we'll need new command line flag that which tells dmd to regenerate .di file even if it's source .d file was not changed. sure we can workaround that by teaching build tool to call 'touch' or something, but for what reason?

A force rebuild would start by deleting generated .di files.

> either we have full-blown build tool in the compiler itself or we don't try to teach compiler some tricks which build tool should do and then start fighting with compiler to stop it tricking.

This is simple correct generated file management: what doesn't change doesn't get touched. I don't understand the agitation around this.

> i'm on for the concept of "cached ASTs" (something like delphi's .dcu files) and teaching dmd to rebuild all changed modules and updating cached ASTs. this will greatly imporoves compilation speed (no need to do lexing, parsing and semantic analyzis again and again) and will become a nice and simple build system. but this is a very complex change and it's completely independent of ones you proposing.

I encourage you to submit that as a separate enhancement request for that. But please stop bullying this into the ground. I don't have the time to defend it and I have no interest in "winning" the argument consider you've won; I do have skin in the game and need to see this through. On the other hand it's relatively easy for you to naysay this by brute force. Clearly we could talk at great lengths about the relative responsibilities of compilers and external build tools, seeing as it's a matter in which reasonable people do disagree. Also scripting around the language is a definite possibility. But a stalemate here leads to the net result that nothing gets done and there's no progress. So please don't make this all into a net negative. Thanks.

--

https://issues.dlang.org/show_bug.cgi?id=13517

Ketmar Dark <ketmar@ketmar.no-ip.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|ketmar@ketmar.no-ip.org     |

--- Comment #12 from Ketmar Dark <ketmar@ketmar.no-ip.org> ---
(In reply to Andrei Alexandrescu from comment #11)
> I've explained my reasons
no, you don't. you just keep saying "it will simplify everyone's life". and when i'm saying that it will not (using your own style of argumentation, actually), you accused me of bullying.

i don't want to talk on this issue anymore, it's meaningless. i don't see any way of making discussion productive using such "no-op arguments".

--

https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #13 from Andrei Alexandrescu <andrei@erdani.com> ---
(In reply to Ketmar Dark from comment #12)
> (In reply to Andrei Alexandrescu from comment #11)
> > I've explained my reasons
> no, you don't. you just keep saying "it will simplify everyone's life".

My original argument: "Running dmd -H always generates a fresh .di file, even if an existing .di file exists and has identical content. This confuses build tools - such as make and derivatives - that rely on file timestamp to assess what needs to be rebuilt." Your conter-argument has been that that's the responsibility of the build tool. That's true, but it makes creating build tools for D more elaborate and more complicated than it should. My take is D should offer a simpler and better interface to classic build tools that consult timestamps and rely on the compiler to provide dependency information. I'll elaborate below.

> and
> when i'm saying that it will not (using your own style of argumentation,
> actually), you accused me of bullying.

With respect, that's exactly what happens. This is disproportionate response to a simple matter.

> i don't want to talk on this issue anymore, it's meaningless. i don't see any way of making discussion productive using such "no-op arguments".

Let me elaborate how I see this would help building D projects.

Currently (incremental builds, simple dependency management, no .di files), following the touch of xyz.d:

1. The deliverable depends on xyz.o and xyz.o depends on xyz.d. Therefore xyz.o is rebuilt.

2. The dependencies (previously saved with dmd -deps) list xyz.d as a dependence for all other modules that transitively import it. The timestamp of xyz.d has changed. Therefore the transitive closure of modules importing xyz must be rebuilt.

3. Finally deliverable(s) are linked etc.

With .di files and without any special steps, the above doesn't change /because/ the .di files will be touched during the building of xyz.o. (There is a change in the sense that now modules depend on .di files, not .d files.)

Now consider there's scripting around the build system that only touches xyz.di if it has new content. In that case, only a subset of changes to xyz.d will be reflected in xyz.di. So the process would go:

1. The deliverable depends on xyz.o and xyz.o depends on xyz.d. Therefore xyz.o is rebuilt.

2. The dependencies (previously saved with dmd -deps) list xyz.di as a dependence for all other modules that transitively import it. The timestamp of xyz.di may or may not have changed. In particular, changes to implementation of non-generic functions don't affect the interface. In that case there's no ripples.

3. Finally deliverable(s) are linked etc.

This is going to go the same for all timestamp-based build systems so it's somewhat generic. The bit about touching the .di files conditionally is more D-specific. Now the question is where to place that code. Clearly the build tool could muster it. In my opinion it's easier to interface with generic build tools if the timestamps just work, as opposed to describing specifics on how these could be made to work.

I'll also note that the story with .o files is quite different but may be worth looking at in the context of D. There are just few changes that don't affect .o file generation. The two that I can think of are comments and bodies of templates that aren't used within the same file. If we conclude these are somewhat frequent, conditional touching of the .o file may be interesting to do.

--

https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #14 from Ketmar Dark <ketmar@ketmar.no-ip.org> ---
(In reply to Andrei Alexandrescu from comment #13)
> With respect, that's exactly what happens. This is disproportionate response to a simple matter.
this proposal leads to increasing complexity of compiler code. and it's for nothing, actually, 'cause this work can be easily done with external script or with build tool itself if it's not 'make'.

i'm just strogly against:
1. adding unnecessary code to compiler.
2. turning dmd to half-baked build tool.

> Now consider there's scripting around the build system that only touches xyz.di if it has new content.
then we have strange hidden dependency: .di file *seems* to be dependend of .d file, yet it is clearly older than .d file and somehow it's not regenerated (why?). so the build tool which relies on timestamps is either confused or the user need to remember the dependency but don't inform build tool about it.

i see only added complexity here. such complexity is unnecessary for small projects with small build times, and it can easily turn to complete disaster with big complex projects (been there, seen that). complex projects *will* fight with this feature if their build tools are timestamp-based (or build process will *inevitably* go out of control). and it's really easy to do this with simple comparison script (ah, just do 'sort' and then calculate md5, for example).

that's why i'm saying that overal win is close to nothing, but compiler implementation gains more code.

all in all you aren't convinced me. but your position is clear to me now, so we can stop arguing and wait for PR. and then let Walter to decide.

sorry for being rude, it was luck of clear understanding from my side.

--

https://issues.dlang.org/show_bug.cgi?id=13517

Steven Schveighoffer <schveiguy@yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |schveiguy@yahoo.com

--- Comment #15 from Steven Schveighoffer <schveiguy@yahoo.com> ---
(In reply to Vladimir Panteleev from comment #3)
> (In reply to Ketmar Dark from comment #1)
> > but why compiler should do that? it's a build tool task. the build tool should know that .di file is generated from .d and invoke generator only if .di file is missing or older than .d.
> 
> That idea only applies if you want to do .di generation as a separate compiler invocation from compilation.

I was about to argue against Andrei's position, but this makes sense -- the .di has no dependencies in the build tool, nor does it know how to build the .di file. It's just a side effect of building the .o

The only caveat is that you can't have the same project that generates the .di files (as a side effect) depend on the .di files in any way. Given the purpose of .di files, that seems reasonable.

--

https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #16 from Ketmar Dark <ketmar@ketmar.no-ip.org> ---
(In reply to Steven Schveighoffer from comment #15)
> .di has no dependencies in the build tool, nor does it know how to build the .di file. It's just a side effect of building the .o
ahem. just tried `dmd -c z00.d`. no .di file was built. what kind of "side effect" are you talking here? ah, and another try: `dmd -c -o- -H z00.d`. this time i got .di file, but no .o file. i'm confused.

and why build tool doesn't know about '.di'? any sane build tool allows adding dependencies. and no build tools except DUB and my own knows about D out-of-the-box anyway.

--

https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #17 from Steven Schveighoffer <schveiguy@yahoo.com> ---
(In reply to Ketmar Dark from comment #16)
> ahem. just tried `dmd -c z00.d`. no .di file was built. what kind of "side effect" are you talking here? ah, and another try: `dmd -c -o- -H z00.d`. this time i got .di file, but no .o file. i'm confused.

-c builds .o
-o- suppresses .o file.
-H builds.di file.

So:

Stevens-MacBook-Pro-2:testd steves$ ls testdi.*
testdi.d
Stevens-MacBook-Pro-2:testd steves$ cat testdi.d
void foo() {}
Stevens-MacBook-Pro-2:testd steves$ dmd -c -H testdi.d
Stevens-MacBook-Pro-2:testd steves$ ls testdi.*
testdi.d    testdi.di    testdi.o

> 
> and why build tool doesn't know about '.di'? any sane build tool allows adding dependencies. and no build tools except DUB and my own knows about D out-of-the-box anyway.

I mean, the build tool knows it only as a dependency, not as a target. It has no idea how to build the .di file. The .di file is built as a side effect of building the .o file.

But as I said, you would need to ensure the project that builds the .di file cannot depend on the .di file, because it doesn't know how to build them, it just does so as a side effect.

--

https://issues.dlang.org/show_bug.cgi?id=13517

--- Comment #18 from Steven Schveighoffer <schveiguy@yahoo.com> ---
I would add a stipulation that if the purpose of the compiler invocation is ONLY to build the .di file, then the compiler should rewrite it. So dmd -c -o- -H should still touch the .di file even if it's identical to the existing one.

Otherwise, this disfavors a build system where the .di is built as a separate step (and causes a continual "rebuild" of a .di file every time the build is called).

--