Wanted to get peoples thoughts on this.  The idea is to have a way to tell the compiler (probably with a command line option) that you'd like to "compile imported modules".  Say you have a program "prog" that depends on modules "foo" and "bar".

    import foo;
    import bar;

Compilation could look like:

    dmd prog.d foo.d bar.d

Or it could look like

    dmd -c foo.d
    dmd -c bar.d
    dmd prog.d foo.obj bar.obj

With this command line option, let's call it "-compile-imports" for now, you could do something like:

    dmd -compile-imports prog.d

This tells the compiler that after it has processed all the input files (source code/object files/library files), if it is missing modules, it should go back and look for those modules in it's list of imported modules, then compile them from there.  It's important that it only checks this after processing all the input files so that precompiled modules take precedence.  So you could still do something like this:

    dmd -c foo.d
    dmd -compile-imports prog.d foo.obj

In this example we use the precompiled foo module and then the compiler notices that the bar module is missing.  So it looks for the source in it's list of imports, then includes that in it's list of files to compile essentialy behaving as if that file was passed on the command line.

This is a simple example with only 2 modules, but for projects that use alot of libraries it could turn into something like this:

    dmd prog.d -Isomelib somelib\foo\module1.d somelib\foo\module2.d somelib\foo\module3.d somelib\foo\module4.d somelib\foo\module5.d somelib\foo\module6.d -Ianotherlib anotherlib\bar\module1.d anotherlib\bar\module2.d anotherlib\bar\module3.d anotherlib\bar\module4.d anotherlib\bar\module5.d

into this:

    dmd -compile-imports prog.d -Isomelib -Ianotherlib

This would also simplify rdmd and make it "less brittle" because it will not need to duplicate the logic inside the compiler that locates and selects which module files to compile.  Instead, it can simply use the -compile-imports switch leave that logic completely in the compiler.

On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via Digitalmars-d wrote:
> Wanted to get peoples thoughts on this.  The idea is to have a way to tell the compiler (probably with a command line option) that you'd like to "compile imported modules".
[...]

Isn't this what rdmd already does?


T

-- 
Do not reason with the unreasonable; you lose by definition.

On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:
> On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via Digitalmars-d wrote:
>> Wanted to get peoples thoughts on this.  The idea is to have a way to tell the compiler (probably with a command line option) that you'd like to "compile imported modules".
> [...]
>
> Isn't this what rdmd already does?
>
>
> T

That is one thing that rdmd does (as I mentioned in the original post).

I just looked through the rdmd code (https://github.com/dlang/tools/blob/master/rdmd.d) and it looks like it invokes the compiler using "dmd -v" to get the list of modules and then invokes the compiler again with the modules it found to perform the full compile.  So my original thought that the logic to find modules is duplicated was incorrect.  Instead we just pay a performance hit to get the correct list of imports since running "dmd -v" seems to take almost as long as the actual compile itself.  So this method comes close to doubling the time it takes to compile than if the feature was implemented in the compiler itself.

In any case, the idea is to allow the compiler to resolve this on it's own without help from rdmd.  This would remove the need to invoke the compiler twice, once to find the imports and once to compile.  It would also allow some projects/applications that don't use rdmd to take advantage of this feature, this may or may not include dub (not sure on that one).

On Thursday, 24 August 2017 at 16:32:32 UTC, Jonathan Marler wrote:
> On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:
>> On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via Digitalmars-d wrote:
>>> Wanted to get peoples thoughts on this.  The idea is to have a way to tell the compiler (probably with a command line option) that you'd like to "compile imported modules".
>> [...]
>>
>> Isn't this what rdmd already does?
>>
>>
>> T
>
> That is one thing that rdmd does (as I mentioned in the original post).
>
> I just looked through the rdmd code (https://github.com/dlang/tools/blob/master/rdmd.d) and it looks like it invokes the compiler using "dmd -v" to get the list of modules and then invokes the compiler again with the modules it found to perform the full compile.  So my original thought that the logic to find modules is duplicated was incorrect.  Instead we just pay a performance hit to get the correct list of imports since running "dmd -v" seems to take almost as long as the actual compile itself.  So this method comes close to doubling the time it takes to compile than if the feature was implemented in the compiler itself.
>
> In any case, the idea is to allow the compiler to resolve this on it's own without help from rdmd.  This would remove the need to invoke the compiler twice, once to find the imports and once to compile.  It would also allow some projects/applications that don't use rdmd to take advantage of this feature, this may or may not include dub (not sure on that one).

rdmd is really bad in terms of performance. If you call a single D file with rdmd, it will always compile it twice. There was an attempt to fix this (https://github.com/dlang/tools/pull/194), but this has been reverted as it introduced a regression and no one had time to look at the regression.
Moving rdmd into DMD has been on the TODO list for quite a while and there is a consensus that the performance overhead if rdmd isn't nice. However, IIRC there was no clear consensus on how the integration should happen. I recall that the plan was to do try this with "dmd as a library", but I'm not sure whether that's really feasible ATM.

On Thu, Aug 24, 2017 at 04:49:08PM +0000, Seb via Digitalmars-d wrote: [...]
> rdmd is really bad in terms of performance. If you call a single D file with rdmd, it will always compile it twice. There was an attempt to fix this (https://github.com/dlang/tools/pull/194), but this has been reverted as it introduced a regression and no one had time to look at the regression.  Moving rdmd into DMD has been on the TODO list for quite a while and there is a consensus that the performance overhead if rdmd isn't nice. However, IIRC there was no clear consensus on how the integration should happen. I recall that the plan was to do try this with "dmd as a library", but I'm not sure whether that's really feasible ATM.

Hmm. An interesting thought occurred to me: dmd already has a -run option, so perhaps it wouldn't be too hard to add an auto-import option like Jonathan proposes, then dmd would essentially have the functionality of rdmd?  Well, other than caching the executable, that is.  But once auto-import is in, redundant compilation will become a thing of the past, as rdmd could just invoke dmd, and the only thing extra it would do is the executable caching.


T

-- 
What doesn't kill me makes me stranger.

On Thursday, 24 August 2017 at 16:49:08 UTC, Seb wrote:
> On Thursday, 24 August 2017 at 16:32:32 UTC, Jonathan Marler wrote:
>> On Thursday, 24 August 2017 at 15:56:32 UTC, H. S. Teoh wrote:
>>> On Thu, Aug 24, 2017 at 03:53:05PM +0000, Jonathan Marler via Digitalmars-d wrote:
>>>> Wanted to get peoples thoughts on this.  The idea is to have a way to tell the compiler (probably with a command line option) that you'd like to "compile imported modules".
>>> [...]
>>>
>>> Isn't this what rdmd already does?
>>>
>>>
>>> T
>>
>> That is one thing that rdmd does (as I mentioned in the original post).
>>
>> I just looked through the rdmd code (https://github.com/dlang/tools/blob/master/rdmd.d) and it looks like it invokes the compiler using "dmd -v" to get the list of modules and then invokes the compiler again with the modules it found to perform the full compile.  So my original thought that the logic to find modules is duplicated was incorrect.  Instead we just pay a performance hit to get the correct list of imports since running "dmd -v" seems to take almost as long as the actual compile itself.  So this method comes close to doubling the time it takes to compile than if the feature was implemented in the compiler itself.
>>
>> In any case, the idea is to allow the compiler to resolve this on it's own without help from rdmd.  This would remove the need to invoke the compiler twice, once to find the imports and once to compile.  It would also allow some projects/applications that don't use rdmd to take advantage of this feature, this may or may not include dub (not sure on that one).
>
> rdmd is really bad in terms of performance. If you call a single D file with rdmd, it will always compile it twice. There was an attempt to fix this (https://github.com/dlang/tools/pull/194), but this has been reverted as it introduced a regression and no one had time to look at the regression.
> Moving rdmd into DMD has been on the TODO list for quite a while and there is a consensus that the performance overhead if rdmd isn't nice. However, IIRC there was no clear consensus on how the integration should happen. I recall that the plan was to do try this with "dmd as a library", but I'm not sure whether that's really feasible ATM.

Well this should solve the rdmd performance problem as well as make other user cases easier that don't necessarilly use rdmd.

I had another thought that instead of making this an "opt-in" feature, it would probably make more sense to be an "opt-out" feature.  So by default the compiler would compile missing imported modules unless you indicate otherwise, maybe a command line switch like "-dont-compile-imports".  And I don't see how this would break anything.  Everything should work the same as it did before, it's just now you can omit imported module files from the command line and it should just work.

On Thu, Aug 24, 2017 at 05:37:12PM +0000, Jonathan Marler via Digitalmars-d wrote: [...]
> I had another thought that instead of making this an "opt-in" feature, it would probably make more sense to be an "opt-out" feature.  So by default the compiler would compile missing imported modules unless you indicate otherwise, maybe a command line switch like "-dont-compile-imports".  And I don't see how this would break anything.  Everything should work the same as it did before, it's just now you can omit imported module files from the command line and it should just work.

Uh, no.  This will definitely break separate compilation, and some people will be very unhappy about that.  I think it's good enough to leave it as an opt-in feature.


T

-- 
Today's society is one of specialization: as you grow, you learn more and more about less and less. Eventually, you know everything about nothing.

On Thursday, 24 August 2017 at 17:49:27 UTC, H. S. Teoh wrote:
> Uh, no.  This will definitely break separate compilation, and some people will be very unhappy about that.

I couldn't think of a case that it would break.  Can you share the cases you thought of?

On Thu, Aug 24, 2017 at 06:00:15PM +0000, Jonathan Marler via Digitalmars-d wrote:
> On Thursday, 24 August 2017 at 17:49:27 UTC, H. S. Teoh wrote:
> > Uh, no.  This will definitely break separate compilation, and some people will be very unhappy about that.
> 
> I couldn't think of a case that it would break.  Can you share the cases you thought of?

Suppose you have main.d and module.d, and you want to compile them separately:

	dmd -c main.d
	dmd -c module.d
	dmd -ofmyprogram main.o module.o

If dmd defaulted to auto-importing, then `dmd -c main.d` would also compile module.d (assuming main.d imports `module`), contrary to what was intended in a separate compilation scenario, and the last command will produce a linker error from duplicated symbols.

This is just a simple case, of course. But in general, changing the meaning of `dmd -c source.d` will break existing build scripts.  Sure, you could ask people to update their build scripts to include `-no-auto-imports`, but that requires effort from users, who will be unhappy that upgrading dmd broke their build scripts.  For large projects, such a change may not be trivial as in the above example.


T

-- 
Life is too short to run proprietary software. -- Bdale Garbee

On Thursday, 24 August 2017 at 18:12:03 UTC, H. S. Teoh wrote:
> On Thu, Aug 24, 2017 at 06:00:15PM +0000, Jonathan Marler via Digitalmars-d wrote:
>> On Thursday, 24 August 2017 at 17:49:27 UTC, H. S. Teoh wrote:
>> > Uh, no.  This will definitely break separate compilation, and some people will be very unhappy about that.
>> 
>> I couldn't think of a case that it would break.  Can you share the cases you thought of?
>
> Suppose you have main.d and module.d, and you want to compile them separately:
>
> 	dmd -c main.d
> 	dmd -c module.d
> 	dmd -ofmyprogram main.o module.o
>
> If dmd defaulted to auto-importing, then `dmd -c main.d` would also compile module.d (assuming main.d imports `module`), contrary to what was intended in a separate compilation scenario, and the last command will produce a linker error from duplicated symbols.
>
> This is just a simple case, of course. But in general, changing the meaning of `dmd -c source.d` will break existing build scripts.  Sure, you could ask people to update their build scripts to include `-no-auto-imports`, but that requires effort from users, who will be unhappy that upgrading dmd broke their build scripts.  For large projects, such a change may not be trivial as in the above example.
>
>
> T

Actually this feature is mutually exclusive with the "-c" case.  It doesn't make sense to compile imported modules unless you are also linking an executable.  So your example would work as expected.

Do you have any other cases you thought of that would not work? Like I said I couldn't think of any.  I'm not saying that that's enough reason to make it an "opt-out" feature, it's just something to think about.  The feature could also be an "opt-in" feature at first and eventually made "opt-out" if it makes sense.  But I'd still like to know people's thoughts/concerns either way.