On 12/18/2016 05:18 PM, pineapple wrote:
> On Sunday, 18 December 2016 at 21:09:46 UTC, Andrei Alexandrescu wrote:
>> On 12/18/2016 10:01 AM, pineapple wrote:
>>> On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote:
>>>> Is the source code publicly available?
>>>
>>> https://github.com/pineapplemachine/mach.d
>>
>> The code looks clean, congrats. With your permission I'd like to give
>> this library as an example in the "Workaround: Increasing Granularity
>> of Modules" section. Please advise, thanks. -- Andrei
>
> No problem, feel free. Thank you for asking.

Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top.

https://github.com/dlang/DIPs/pull/51

https://github.com/dlang/DIPs/blob/dd46252e820dce66df746540d7ab94e0b00a6505/DIPs/DIP1005.md


Andrei

On Sunday, 18 December 2016 at 22:31:34 UTC, Andrei Alexandrescu wrote:
> Is there a simple command to e.g. unittest everything in the project? Also, is there a build process or it's all templated? -- Andrei

There's no build process. To run tests, I compile the root `package.d` file with rdmd, including the -debug, -unittest, and --main flags, and -I"path/to/repo/mach.d". To compile the mach.sdl package, which is not currently imported by the root `package.d`, you would have to include a directory containing the requisite Derelict dependencies for the -I option.

On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote:
> Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top.
>
> https://github.com/dlang/DIPs/pull/51
>
> https://github.com/dlang/DIPs/blob/dd46252e820dce66df746540d7ab94e0b00a6505/DIPs/DIP1005.md
>
>
> Andrei

It's reasonably accurate. The only thing I would point out is that few modules define only one public symbol; the median probably lies around 4 or 5. A few modules, most or all being in the traits package, define considerably more. The mach.traits.primitives module defines 28 public symbols, by my count. (Though that's still small stuff relative to Phobos.)

I think it would be most accurate to word the sentence as:

> It is organized as such that each module contains a small number of related declarations (such as canAdjoin, Adjoin, and AdjoinFlat in module adjoin.d) along with documentation and unit tests.

On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote:
> Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top.

I would also like to register that while I respect your argument regarding scalability, I have personally found that a greater number of smaller files is easier to manage than a smaller number of larger files. Including for the 580,000+ line project I work on for a living.

On Monday, 19 December 2016 at 00:44:14 UTC, pineapple wrote:
> On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote:
>> Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top.
>
> I would also like to register that while I respect your argument regarding scalability, I have personally found that a greater number of smaller files is easier to manage than a smaller number of larger files. Including for the 580,000+ line project I work on for a living.

(Granted, that old ugly codebase has a lot of problems of its own, and smaller files does not always mean small on the same scale that mach.d's files are small, but I can tell you from experience that once a single ~4,000 line module was broken down into many ~200 line modules, that code became a great deal easier to understand and to reason about and to maintain.)

On 12/18/16 7:44 PM, pineapple wrote:
> On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote:
>> Great, thanks. Please take a look at the accuracy of the discussion. I
>> expanded the "Workaround" section and moved it near the top.
>
> I would also like to register that while I respect your argument
> regarding scalability, I have personally found that a greater number of
> smaller files is easier to manage than a smaller number of larger files.
> Including for the 580,000+ line project I work on for a living.

Is it publicly available? -- Andrei

On Monday, 19 December 2016 at 00:54:13 UTC, Andrei Alexandrescu wrote:
> On 12/18/16 7:44 PM, pineapple wrote:
>> On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote:
>>> Great, thanks. Please take a look at the accuracy of the discussion. I
>>> expanded the "Workaround" section and moved it near the top.
>>
>> I would also like to register that while I respect your argument
>> regarding scalability, I have personally found that a greater number of
>> smaller files is easier to manage than a smaller number of larger files.
>> Including for the 580,000+ line project I work on for a living.
>
> Is it publicly available? -- Andrei

Sadly not.

On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote:
> Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top.
>
> https://github.com/dlang/DIPs/pull/51
>
> https://github.com/dlang/DIPs/blob/dd46252e820dce66df746540d7ab94e0b00a6505/DIPs/DIP1005.md
>
>
> Andrei

What's wrong with the parentheseless version ? The DIP says it looks "out of place" but that doesn't strike me as a very good argument. IMO the version without ";" is the way to go, as it doesn't require to add a new syntax for imports.

Identical things looking identical is valuable.

On Sunday, 18 December 2016 at 18:42:36 UTC, Andrei Alexandrescu wrote:
> On 12/18/16 1:03 PM, Joakim wrote:
>> I largely agree with Dmitry. Ilya refactored several Phobos modules to
>> use scoped, selective imports much more, and I pitched in for some
>> remaining imports in the largest modules, so that only these
>> module-level imports remain, ie those necessary for symbols imported in
>> template constraints:
>>
>> std.datetime - https://github.com/dlang/phobos/pull/4373/files
>> std.uni - https://github.com/dlang/phobos/pull/4365/files
>> std.string and std.traits - https://github.com/dlang/phobos/pull/4370/files
>
> Yah, there's been a lot of good work (Jack Stouffer did a lot as well IIRC) on pushing imports inside. The following searches should be relevant:
>
> git grep '^\(private \)\?import' | wc -l
>
> yields about 426 top-level import declarations. The number of indented imports is 4605:
>
> git grep '     *import\W' | wc -l
>
> So we're looking at 10% of imports being problematic. Sadly, they turn out to make things quite difficult for Phobos. Last time I looked at the module dependency graph it wasn't a lot better than it used to before scoped imports. (I don't have it handy... could anyone please produce it?)

Why do you care _so_ much about the module dependency graph?  To make this question concrete, let's look at an example, the std.array module you keep mentioning.  This is what it looked like before Ilya scoped as many imports as he could:

https://github.com/dlang/phobos/commit/3fcf723aa498b96de165361b5abb9d3450fdc069#diff-54cf8402b22024ae667d4048a5126f0e

That was a mess, similar to opaque C/C++ code, 13 modules imported at module-scope were whittled down to 4.  You just made those more specific in a commit related to this DIP, by listing the actual symbols selectively imported from those four modules:

https://github.com/dlang/phobos/commit/e064d5664f92c4b2f0866c08f6d0290ba66825ed#diff-54cf8402b22024ae667d4048a5126f0e

If I'm looking at the template constraints for any particular function and see a couple symbols I don't recognize, I don't think it's a big deal to find the symbols in that list at the top.

In other words, D already allows you to scope most imports.  I don't consider the dozen or two remaining symbols from templaint constraints and function arguments to provide much overhead.  Rather, I consider the weight of this additional syntax, ie the cognitive overhead from having to remember and parse more syntax in my head, to be worse than the remaining dependency reasoning problem you're trying to solve: the cost outweights the benefit.  Perhaps that's subjective and others may disagree.

Now, there's also the question of purely technical benefits, like compilation speed or executable bloat.  I looked at the latter a little last summer, after Ilya had cleaned up a lot of the standard library:

http://forum.dlang.org/thread/gmjqfjoemwtvgqrtdsdr@forum.dlang.org

I found that commenting out a single scoped, selective import of "std.string: format" in std.utf led to a 5% decrease in executable size for "hello world."  This is a problem with how dmd compiles or appends these module dependencies and would presumably still be there after this DIP, as you would not remove the dependency.

I think scoped, selective imports have been great at hacking away at the module dependency graph, as you lay out.  It is not clear what technical costs you see from the remaining few dependencies and if this DIP is the best way to remove them.  I think you should explain why you want to untangle the remaining dependency graph, and consider if this DIP is really doing that much.

>> When I first saw this DIP, like Dmitry I was happy that we could get rid
>> of those too, but the more I see these horrible syntax suggestions for
>> what is really a minor convenience, I changed my mind.  std.datetime,
>> the 35k line (17 kloc according to Dscanner) beast of phobos, only needs
>> 20 or so symbols at module-scope. std.uni- 10k lines, 4.2 kloc- only
>> needs 17 symbols, all from the three modules Dmitry mentioned.
>>  I don't
>> think his workaround of splitting up modules is even needed for such a
>> low amount of module-level imports.
>
> This paragraph is a good example of a couple of counterarguments that I think point directly to flaws in the DIP:
>
> (1) The DIP uses Phobos as an example, so it is applicable mostly to Phobos. In fact, Phobos is among the systems that would benefit least from the DIP: it has only druntime as dependency, and is distributed in its entirety. Many projects out there list multiple dependencies and may have various building and distribution policies.

It is not clear how those alternate dependency, building and distribution policies change the picture.  Perhaps you should cite one of those as an example.

> The converse is to believe that working around a problem in Phobos would render the DIP less useful in general.

The argument is not that Phobos has "worked around" the problem, but that once you scope everything you can other than template constraints (which you have said should be standard operating procedure in the DIP), the problem is minimal.

> (2) "I don't like the syntax, hence I don't like the feature." I see this as a good opportunity for tasteful design, not a downside of the feature.

The syntax is bad, but even if were beautiful, I don't consider the addition itself to be worth it.  Again, could be subjective.

>> Maybe there are other issues having to do with symbol resolution and
>> dependency encapsulation that are addressed by this DIP, ie the
>> technical performance of the compiler rather than refactoring or code
>> clarity, that I don't fully grasp, but from the first two points of the
>> claimed benefits of DCDs, ie ease of reasoning about dependencies and
>> refactoring code, I don't think this feature will come anywhere close to
>> carrying its own weight.
>
> Does the refactoring in https://github.com/dlang/phobos/pull/4962 make dependencies clearer?

I'm fine with your selective import refactoring, which I linked above:

https://github.com/dlang/phobos/pull/4963

> Are you e.g. clear on what you need in order to use Appender?

It is easy to see that it requires isDynamicArray and that is found in std.traits.  It may internally rely on other symbols that are currently imported from module scope, ie symbols from template constraints inside the struct Appender that happen to also be used outside that struct and so are imported at the top of the module.  If you really wanted to make that clear, you could scope those imports inside the struct too.

> Would you want to take this (or another) experiment to another module and see how it improves its dependency structure?

Ilya and I have already done this, in the PRs I linked above.  I was mildly disappointed back then that I could not get rid of the remaining module-level imports, because of template constraints.  I was predisposed to favor doing something about it, but I don't think this DIP is worth it.

>> As for the third benefit having to do with scalable template libraries,
>> I'm not sure I completely understand all the details there, but I wonder
>> if those problems aren't an artifact of the way dmd works now rather
>> than something that can't be fixed without this DIP.
>
> The DIP now dedicates an entire section to the pluses and minuses of lazy imports, and concludes that lazy imports would address scalability if carefully used.

Yes, there might be other ways to lower the technical costs of the dependency graph too, such as the size issue I mentioned above.

On 2016-12-18 14:38, Andrei Alexandrescu wrote:

> How does the lack of the feature save lines? Doesn't it just move them
> elsewhere?

Yes, I was referring to the file containing the implementation.

> Does that include full documentation and unittests?

It does not count the documentation in those lines. It would count the unit tests because there's no special syntax for that, just regular Ruby code. But one would most likely not have the implementation and the unit tests in the same file.

> Phobos' average file size 2055 is lines and median file size is 903
> lines. Do you find these appropriate for your preferences, considering
> this size includes documentation and unittests?

I would prefer smaller, but I would also prefer to have the unit tests separately.

> Would you find it helpful to use a tool that collapses these?

It's more helpful, yes. I do have that but since signatures in D can be arbitrary complex I'm doubtful that there's a tool/editor can fold all signatures that are valid in D. I know that TextMate that I use cannot. Take this for example:

typeof({
    class Foo
    {
        void bar()
        {
        }
    }
}) foo ();

I know that that's highly unlikely to happen in the real world but it shows how problematic it is to fold D code.

-- 
/Jacob Carlborg