On Friday, 16 December 2016 at 17:19:56 UTC, Andrei Alexandrescu wrote:
> Walter and I wanted to have a positive example of a good DIP that is relatively simple, is noncontroversial, and marks a definite move forward.
>
> We believe the feature's benefits are significant, obvious, and immediate. With this feature, any large project would be able to radically clarify and improve its dependency structure with only little refactoring work. I interpret the negative feedback as failings of the DIP to explain and argue matters appropriately, so I'm working hard on improving it. Please keep feedback coming.

A suggestion: Now that the proposal details are largely fleshed out, take a file from phobos and rewrite it in the new style. Should give insight into aesthetics and identify remaining issues to resolve.

--Jon

On 12/16/2016 01:24 PM, Jon Degenhardt wrote:
> A suggestion: Now that the proposal details are largely fleshed out,
> take a file from phobos and rewrite it in the new style. Should give
> insight into aesthetics and identify remaining issues to resolve.

What a great idea. For starters, I attempted the simple exercise of converting all module-level imports of std.array into static imports. I stopped here:

https://github.com/dlang/phobos/pull/4962

Soon the naive approach "let's replace this import with a static import and rebuild" ran into the midst of a comedy of errors. Code in various other modules fails to compile. Sometimes (happy case) the error indicates the symbol that is not seen, which is easy to fix. Other times (as you may see if you try to touch the PR above) there is zero indication on where the problem originates.

I invite anyone who believes DIP1005 is solving just a minor problem and/or that we can rein in by using lazy imports and no improvement to the language, to try their hand at continuing the PR above or starting a similar effort. Currently it is exhaustingly difficult to figure where names are looked up in a large D project.


Andrei

On 12/16/2016 01:24 PM, Jon Degenhardt wrote:
> On Friday, 16 December 2016 at 17:19:56 UTC, Andrei Alexandrescu wrote:
>> Walter and I wanted to have a positive example of a good DIP that is
>> relatively simple, is noncontroversial, and marks a definite move
>> forward.
>>
>> We believe the feature's benefits are significant, obvious, and
>> immediate. With this feature, any large project would be able to
>> radically clarify and improve its dependency structure with only
>> little refactoring work. I interpret the negative feedback as failings
>> of the DIP to explain and argue matters appropriately, so I'm working
>> hard on improving it. Please keep feedback coming.
>
> A suggestion: Now that the proposal details are largely fleshed out,
> take a file from phobos and rewrite it in the new style. Should give
> insight into aesthetics and identify remaining issues to resolve.

I also submit this as evidence:

https://github.com/dlang/phobos/pull/4963

This was a successful attempt to convert all imports to selective imports in std.array. It was a very difficult process that took a long time, as shown in the comment.


Andrei

On 12/16/2016 11:59 AM, Andrei Alexandrescu wrote:
> Soon the naive approach "let's replace this import with a static import and
> rebuild" ran into the midst of a comedy of errors. Code in various other modules
> fails to compile. Sometimes (happy case) the error indicates the symbol that is
> not seen, which is easy to fix. Other times (as you may see if you try to touch
> the PR above) there is zero indication on where the problem originates.

The switch:

    -verrors=spec

is very useful here. It prints the errors otherwise gagged when inside __traits(compiles,...).

On 2016-12-13 23:33, Andrei Alexandrescu wrote:
> Destroy.
>
> https://github.com/dlang/DIPs/pull/51/files

A couple of questions.

1. The text says:

"void process(File input) import (std.stdio);

With this syntax, the import is executed only if the declared name is actually looked up."

Is "declared name" referring to the function "process"? To me it sounds like it could be misinterpreted to refer to "File", which would indicate the import is lazily executed.

2. The text says:

"Currently inline imports do not apply to alias declarations, enumerated types declarations, or variable declarations"

What's the reasoning behind this?

3. Is it required to place the import after the declaration or can it be placed in front of it?

4. Related to the third question. Would it be better (if not already) to implement the import as an attribute, which would allow for a more flexible syntax:

void process(File input) import (std.stdio);
import (std.stdio) void process(File input);

import (std.stdio)
{
    void process(File input);
    void process2(File input);
}

import (std.stdio):

void process(File input);
void process2(File input);

5. About the original syntax that was used, i.e. "import.std.range.isInputRange!Range". That syntax could require to always use a colon to separate the module name from the imported symbol. It's the same syntax as a selective import, which I think fits since with this syntax it will always be selective imports, as far as I understand. Example:

void process(import.std.stdio:File input);

This syntax could also be extended to all symbols:

void process(import.std.stdio:File input)
{
    import.std.stdio:File a = input;
}

6. Not sure if it's a requirement for a DIP but I don't see any formal description of the necessary grammar changes.

-- 
/Jacob Carlborg

Just looking at this again:

> The obvious workaround to the problem that dependencies must be module- level is to simply define many small modules---in the extreme, one per declaration.

Andrei works in phobos a lot. Phobos has a lot of large modules. For instance, std.datetime is 35,000 lines. It's not unusual for a phobos module to have over 6,000 lines (std.math, std.typecons, std.traits, std.format, std.conv).

I'd normally recommend breaking up modules at one fifth that size. The standard library benefits from low granularity modules. It needs to implement a variety of related tools for working with particular things.

For the hunting-for-definitions case, you also need:

* a module with more than a few imports, from different libraries or
packages
* ambiguous names, or functions that are widely used
* the user can't use an IDE / ctags / dcd
* the user can't use ddox / dpldocs.info, which turns type references
into links; or the user is using that and needs to find the definition of
a template constraint
* the maintainer cannot use selective imports
* the maintainer cannot break the module up to reduce the number of
dependencies
* the maintainer is willing to spend the effort to convert top-level
imports into tightly scoped imports

For the compilation-speed case, you need:

* large dependencies that this allows you to skip (the module combines
several types of functionality with different dependencies)
* the imported module must be in another compilation unit (incremental
compilation or a separate library)
* the dependencies can't be used by any other module in the compilation
unit
* no selective imports
* the module being compiled depends on something in the same scope

That's a pretty marginal use case.

On 12/17/2016 02:34 PM, Chris Wright wrote:
> Just looking at this again:
>
>> The obvious workaround to the problem that dependencies must be module-
>> level is to simply define many small modules---in the extreme, one per
>> declaration.
>
> Andrei works in phobos a lot. Phobos has a lot of large modules. For
> instance, std.datetime is 35,000 lines. It's not unusual for a phobos
> module to have over 6,000 lines (std.math, std.typecons, std.traits,
> std.format, std.conv).

Let's take a look at that hypothesis. The example I chose randomly (and which turned to be a rat's nest of fuzzy dependencies) was std/array.d, clocking at 3585 lines. Then looking at the entire project:

wc -l std/*.d std/{algorithm,container,digest,experimental,internal,net,range,regex}/**/*.d | sort --key=1 -n | cat -n

This outputs the modules in the standard library (excluding those that are simple header translations), sorted by LoC, numbered. See result in http://paste.ofcode.org/Lc5xfcs8GqpT2cabApSSgk. That shows 137 modules, median length 903, average length 2055 --- including full documentation, unittests, and examples. These numbers seem quite reasonable and if anything compare favorably against other projects I've been on.

> I'd normally recommend breaking up modules at one fifth that size.

Yeah, std/datetime.d is a monster, from what I can tell owing to a rote and redundant way of handling unittesting. I didn't look at its dependencies, but I doubt they are special. I was quite vocal about breaking it up, but I got mellower with time since (a) someone measured its size without unittests and it was something like one order of magnitude smaller, and (b) there was really no more trouble using or maintaining it than with anything else in Phobos.

I should also add that each large project has a couple of outliers like that. I even recall a switch of a couple thousand lines once :o).

> The
> standard library benefits from low granularity modules. It needs to
> implement a variety of related tools for working with particular things.
>
> For the hunting-for-definitions case, you also need:
>
> * a module with more than a few imports, from different libraries or
> packages
> * ambiguous names, or functions that are widely used
> * the user can't use an IDE / ctags / dcd
> * the user can't use ddox / dpldocs.info, which turns type references
> into links; or the user is using that and needs to find the definition of
> a template constraint
> * the maintainer cannot use selective imports
> * the maintainer cannot break the module up to reduce the number of
> dependencies
> * the maintainer is willing to spend the effort to convert top-level
> imports into tightly scoped imports
>
> For the compilation-speed case, you need:
>
> * large dependencies that this allows you to skip (the module combines
> several types of functionality with different dependencies)
> * the imported module must be in another compilation unit (incremental
> compilation or a separate library)
> * the dependencies can't be used by any other module in the compilation
> unit
> * no selective imports
> * the module being compiled depends on something in the same scope
>
> That's a pretty marginal use case.

Most of these have been the case with all C++ and D projects I've been involved with at Facebook.

Please let me know what of this information I should include in the DIP to make it better. Thanks.


Andrei

On Sat, 17 Dec 2016 19:34:12 -0500, Andrei Alexandrescu wrote:
> Most of these have been the case with all C++ and D projects I've been involved with at Facebook.

I've had similar frustrations when using C# without an IDE (which is a problem because it encourages larger namespaces), and somewhat with Python (though much worse because of the lack of static typing).

D doesn't have either of those pitfalls, so I haven't seen it cause problems. I'm also a bit skeptical that this will see much use outside phobos.

This isn't really an argument against it. I just don't see any argument for it, not that's supported by my own experience.

> Please let me know what of this information I should include in the DIP to make it better. Thanks.

You don't mention improved documentation as an option for readability.

As a general comment, the Rationale section takes a while to get to the benefits (and with it the associated problems that the DIP is trying to address).

My advisor in college exhorted us to be *prefix-competitive* in our papers and talks: your audience is present at the start but will get bored and wander off at some random point, so make sure that you maximize the impact of the part of the message they hear. Every prefix of your talk should include as much value as you can pack into that many words.

On Sunday, 18 December 2016 at 02:40:59 UTC, Chris Wright wrote:
> D doesn't have either of those pitfalls, so I haven't seen it cause problems. I'm also a bit skeptical that this will see much use outside phobos.
>
> This isn't really an argument against it. I just don't see any argument for it, not that's supported by my own experience.

I would like to echo this sentiment.

I am developing a general-use library for D that is currently resting at around 50,000 lines. I have never felt a need for a feature like this, and I can't imagine a reason to begin using it. Dependency management has just never presented an issue. Very nearly all modules in the library are fewer than 1,000 lines long and very nearly all symbols are selectively imported, and the approach has proven to be completely manageable.

If it can be added without interfering with the existing patterns, I don't really have an argument against this feature. But I do think that what this DIP is meant to address is not really a problem experienced by all or even most who are working with D. It's a problem being experienced with Phobos, but there are very valid solutions to that problem that don't involve an addition to the language - only some refactoring. I think that makes the argument in favor somewhat weak.

On 2016-12-18 01:34, Andrei Alexandrescu wrote:

> Yeah, std/datetime.d is a monster, from what I can tell owing to a rote
> and redundant way of handling unittesting. I didn't look at its
> dependencies, but I doubt they are special. I was quite vocal about
> breaking it up, but I got mellower with time since (a) someone measured
> its size without unittests and it was something like one order of
> magnitude smaller, and (b) there was really no more trouble using or
> maintaining it than with anything else in Phobos.

Most other languages don't have inline unit tests, which saves a lot of lines of code.

Not sure if this is the case. But if we have unit tests that are more on the functional/integration side perhaps those should be moved to a separate file structure.

> I should also add that each large project has a couple of outliers like
> that. I even recall a switch of a couple thousand lines once :o).

Just because another project is worse doesn't mean we're in a good position.

Rubocop, the major linter in the Ruby world, will complain if a class is more than 100 lines of code. I think that is on the extreme side but I think any module with more than 2000 lines of code is too big.

-- 
/Jacob Carlborg