On Sat, 24 Dec 2016 09:20:19 +0000, John Colvin wrote:
> There are a lot of templates in Phobos that never use the template keyword. The proposal doesn't only apply to constraints, it applies to the whole declaration.

If you have a better way of estimating the impact, I'd love to see it.

On Saturday, 24 December 2016 at 17:52:04 UTC, Chris Wright wrote:
>
> The minimum isn't terribly useful because it gets to the point of testing the process scheduler and IO more than the compiler. If we want numbers that we can trust on the low end, we'll need to put timing information into the compiler, maybe control for IO by using a ramfs, that sort of thing.

As a good approximation you can use a profile build of dmd.
It will give you a breakdown of how long the individual functions took.
Due to the way the profiling works, it does homogenize the values a bit.
Files are not streamed into dmd, it reads them as one Block.
Therefore there is no need for a ramfs.

On 12/24/2016 12:52 PM, Chris Wright wrote:
> On Sat, 24 Dec 2016 04:34:03 -0500, Andrei Alexandrescu wrote:
>> Upon more investigation, I see a large discrepancy between the findings
>> of DIP1005 and yours.
>
> There's no discrepancy.
>
> In part, you are misinterpreting most of what I said.

Yah, I suspected so.

* Your post provided no description of the methodology used beyond "relatively simple regex" to come up with the magic number 26, so I tried to use interpretation.

* There is no answer to "where is cmp on that list for example?" to shed light on what 26 is about.

* I have no access to your method of measurement, the scripts you used, or really what the claims are.

So I really don't have a good understanding of what hypothesis you have and how you arrived to it. DIP1005 carefully describes methodology and publishes measurements and results (which I'd be in your debt if you used - not those "published elsewhere"). Then it interprets the results.

> In part, you are assuming that imports on non-template declarations will
> be handled lazily, even though that is not part of this DIP, even though
> that is likewise possible with static and selective imports.

Imports in non-template declarations ARE handled lazily when those non-templates are imported. Consider:

module test;
void fun()
{
    import std.stdio;
    writeln("hello");
}

module test2;
import test;
void main() {} // let's not call fun

dmd -v test2.d

====
binary    dmd
version   v2.073.0-devel-bd0dec2
config    /home/andrei/bin/dmd.conf
parse     test2
importall test2
import    object	(/home/andrei/d/druntime/import/object.d)
import    test	(test.d)
semantic  test2
entry     main      	test2.d
semantic2 test2
semantic3 test2
code      test2
function  D main
cc test2.o -o test2 -m64 -L/home/andrei/d/phobos/generated/linux/release/64 -Xlinker -Bstatic -lphobos2 -Xlinker -Bdynamic -lpthread -lm -lrt -ldl
====

Destroyed? This might be another misinterpretation of what you said though.

> In part, you are using lines of code as a proxy for compile time.

What do you suggest to use?

> In part, you dispute that this only affects template constraints, but:
> * An import used only in the body of a template can be made a local
> import today.

Correct. To the best of my knowledge, that fruit has been picked in Phobos.

> * An import used in the declaration of a templated type or function can
> be addressed by using explicit template syntax, offering a place to
> insert your imports.

Good thought. Would be a bit overkill though.

> * An import used anywhere else must still be processed, even assuming
> this DIP is implemented.

How do you mean that? On the face of it the sentence is true. Are you saying that you need to remove top-level imports to benefit from the DIP? That would indeed be the case.

> * If, in a future DIP, we make it so that `with(import)` is handled
> lazily, we can also make it so that static and selective imports are
> handled lazily.

That would be nice, but DIP1005 provides advantages beyond latency.

BTW it would be great if the discussion focused on (a) things that are true but the DIP doesn't mention (or misrepresents), or (b) things that are false that the DIP claims. I clearly concede that the DIP cannot convince someone with enough cognitive bias without an implementation and a few years of experience (and even then - aren't there folks out there who believe things like constraints, static if, or ranges are failures?).

>> The findings of DIP1005 are the following:
>>
>> * Importing a single std module also imports on average 10.5 other
>> modules.
>
> Seems reasonable. Between 2 and 3.5 direct dependencies, by my count, and
> you're counting transitive dependencies.

Shouldn't I? Shouldn't you?

> We're concerned with the effects of DIP1005, though, which only affects
> template constraints.
>
>> * Importing a single std module costs on average 64.6 ms.
>
> 55-ish for your hardware, you reported elsewhere. 47-ish for mine.

Let's stick with the numbers published in the DIP.

>> * (Not stated in the DIP) A majority of std templates would acquire
>> inline imports.
>
> Again, that wouldn't impact compile times because these aren't template
> constraints.

I don't understand this. YES it would impact compile times!

> You can make a separate DIP to make imports lazy. That can impact static,
> selective, and `with` imports equally well. But it's not part of what
> we're discussing today.
>
>> According to the DIP, one may estimate that the proposed feature would
>> reduce additional imports to 0 and the average time to import a single
>> module by a factor of 10 to under 10 ms.
>
> "The proposed feature" must be lazy semantic analysis, especially of
> imports. That isn't part of DIP1005.

The proposed feature is DIP1005. I really don't understand how your discourse goes.

> You won't get to zero additional imports. You might get to zero
> *extraneous* imports -- that is, only the set of imports required to
> create a custom *.di file containing only the parts of the module that
> your application uses.

You are wrong. With no top-level imports and all imports hoisted into inline imports, the fixed cost of importing one module will be that module alone. To focus the discussion, could we please focus on this one point?


Andrei

On 12/24/2016 12:45 PM, Stefan Koch wrote:
> I just read over the dip, and it is a giant wall of text.
> I cannot really make heads or tails of it.

When it was shorter, people complained it had too little information.

If you're using the text form, the nicey formatted version may be helpful: https://github.com/dlang/DIPs/blob/0e15d550d6774dfd40ec78dc201dfad33f47a740/DIPs/DIP1005.md

> Maybe you could write the advantages you hit at in short bullet-point
> form ?

There is. Search for "The analysis above reveals that Dependency-Carrying Declarations have multiple benefits:".


Thanks,

Andrei

On Saturday, 24 December 2016 at 17:53:04 UTC, Chris Wright wrote:
> On Sat, 24 Dec 2016 09:20:19 +0000, John Colvin wrote:
>> There are a lot of templates in Phobos that never use the template keyword. The proposal doesn't only apply to constraints, it applies to the whole declaration.
>
> If you have a better way of estimating the impact, I'd love to see it.

Can we hold off on the performance discussion?  Walter says this DIP isn't hard to implement (https://github.com/dlang/DIPs/pull/51#issuecomment-269077790), so we will run some numbers and see what we get.  As you know, I too am skeptical of the benefit but Andrei has shown that import fanout has a non-negligible cost with his latest benchmark, and actual measurement will be the best way to decide.

On 12/24/2016 02:03 PM, Andrei Alexandrescu wrote:
> With no top-level imports and all imports hoisted into inline imports,
> the fixed cost of importing one module will be that module alone.

I'd really like to clarify this because that's where scalability comes from. The moment you need to open an unbounded amount of files for one import, it's gone. If I missed something and more than one module needs to be looked at I need to rethink the whole DIP. -- Andrei

On 12/24/2016 02:16 PM, Joakim wrote:
> On Saturday, 24 December 2016 at 17:53:04 UTC, Chris Wright wrote:
>> On Sat, 24 Dec 2016 09:20:19 +0000, John Colvin wrote:
>>> There are a lot of templates in Phobos that never use the template
>>> keyword. The proposal doesn't only apply to constraints, it applies
>>> to the whole declaration.
>>
>> If you have a better way of estimating the impact, I'd love to see it.
>
> Can we hold off on the performance discussion?  Walter says this DIP
> isn't hard to implement
> (https://github.com/dlang/DIPs/pull/51#issuecomment-269077790), so we
> will run some numbers and see what we get.  As you know, I too am
> skeptical of the benefit but Andrei has shown that import fanout has a
> non-negligible cost with his latest benchmark, and actual measurement
> will be the best way to decide.

Also (from the cycle "sounding like a broken record") the impact is in other dimension than import speed. From DIP1005:

==========
[...] Dependency-Carrying Declarations have multiple benefits:

* Specifies dependencies at declaration level, not at module level. This allows reasoning about the dependency cost of declarations in separation instead of aggregated at module level.

* If all declarations use Dependency-Carrying style and there is no top-level import, human reviewers and maintainers can immediately tell where each symbol in a given declaration comes from. This is a highly nontrivial exercise without specialized editor support in projects that pull several other modules and packages wholesale. Even a project newcomer could gather an understanding of a declaration without needing to absorb an arbitrary amount of implied context from the declaration at the top of the module.

* Dependency-Carrying Declarations are easier to move around, making for simpler and faster refactorings.

* Dependency-Carrying Declarations allow scalable template libraries. [...]

Dependency-Carrying Declarations also have drawbacks:

* If most declarations in a module need the same imports, then factoring them outside the declarations at top level is simpler and better than repeating them.

* Related, renaming one module is likely to require more edits in a Dependency-Carrying Declarations setup.

* Traditional dependency-tracking tools such as make and other build systems assume file-level dependencies and need special tooling (such as rdmd) in order to work efficiently.

* Dependencies at the top of a module are easier to inspect quickly than dependencies spread through the module.
==========


Andrei

On Fri, 23 Dec 2016 23:41:46 -0500, Andrei Alexandrescu wrote:

> On 12/23/16 9:23 PM, Chris Wright wrote:
>> The comparison to mach.d is a strawman.
> 
> The mach.d is given as an example of the approach of breaking code into fine-grained module. No comparison is made or implied.

"Assuming module size is a project invariant, the number of files scales roughly with project size. This means mach.d would need 2000 files to scale up to the same size as the D standard library"

That's a direct comparison that you made. You might want to remove that from the proposal if you don't want to compare them.

> Could you please give more detail about the method you used?

An insufficient one.

(I would think this sort of analysis would generally be the burden of the proposer, generally.)

https://gist.github.com/dhasenan/681b5178672556aa0e5ec8fb4c9eae7e uses dmd's json output instead of regular expressions and should be rather better.

This shows a total of 35 templates used in template constraints, 124 usages in total.

The quick usage breakdown:

 1 std.experimental.ndslice.internal
 1 std.experimental.ndslice.slice
24 std.meta
31 std.range.primitives
66 std.traits
 1 std.typecons

Notable mentions: the ndslice templates are used only in other ndslice modules, and the std.typecons reference is from std.experimental.typecons. So it's still ~10ms of overhead, by both our measurements.

> And again the
> feature's benefits go well beyond making general projects faster to
> build.

You may as well remove the parts about compilation efficiency from the proposal entirely, then.

The DIP's points here aren't invalid. However, static and selective imports work nearly as well. A doc generator that turns referenced identifiers into links (like dpldocs.info) works nearly as well for reading. For moving files between modules, that's relatively rare, and fixing imports is a small part of it. (In the worst case, I can copy all imports over and use dustmite to reduce them. Or just leave them.)

It provides small occasional benefits for a nontrivial maintenance cost. At least it doesn't mandate that I use it.

> A simple way to look at things is - local imports are The Right Thing, for many reasons beyond compilation speed. There needs to be a way to do the right thing for the declaration part as well.

Function-scoped imports are a good way to hide implementation details from people who don't need them.

On Sat, 24 Dec 2016 14:03:26 -0500, Andrei Alexandrescu wrote:
> * Your post provided no description of the methodology used beyond "relatively simple regex" to come up with the magic number 26, so I tried to use interpretation.
> 
> * There is no answer to "where is cmp on that list for example?" to shed light on what 26 is about.

You mentioned that in another branch of the conversation. It took some time for me to get to it. I'm a human.

>> In part, you are assuming that imports on non-template declarations will be handled lazily, even though that is not part of this DIP, even though that is likewise possible with static and selective imports.
> 
> Imports in non-template declarations ARE handled lazily when those non-templates are imported. Consider:

...

Y'know, I make assumptions, but I tend to test them. But that kind of depends on me writing valid tests, and in this case, I didn't.

My face is egged, and I concede.

Thank you for providing example code. That cleared everything up very fast.

With this new understanding, most modules become nearly free to import under your proposal. It does require a lot of work to get there, but on the plus side, it should be possible to automate that with a tool. I'd be willing to add that to my build process for anything I publish if it'll help others.

>> In part, you are using lines of code as a proxy for compile time.
> 
> What do you suggest to use?

You did write a script to check compile times for importing individual phobos modules.

>>> The findings of DIP1005 are the following:
>>>
>>> * Importing a single std module also imports on average 10.5 other modules.
>>
>> Seems reasonable. Between 2 and 3.5 direct dependencies, by my count, and you're counting transitive dependencies.
> 
> Shouldn't I? Shouldn't you?

I was agreeing with you. At the time, I didn't have a convenient way to check the number of transitive dependencies of a module.

>> We're concerned with the effects of DIP1005, though, which only affects template constraints.
>>
>>> * Importing a single std module costs on average 64.6 ms.
>>
>> 55-ish for your hardware, you reported elsewhere. 47-ish for mine.
> 
> Let's stick with the numbers published in the DIP.

Sure. You insinuated that I should run your script to verify your numbers, so I did.

On Sat, 24 Dec 2016 14:10:30 -0500, Andrei Alexandrescu wrote:

> On 12/24/2016 12:45 PM, Stefan Koch wrote:
>> I just read over the dip, and it is a giant wall of text.
>> I cannot really make heads or tails of it.
> 
> When it was shorter, people complained it had too little information.

It's not the length that's giving trouble; it's the organization.