On Sunday, 18 December 2016 at 11:47:06 UTC, Jacob Carlborg wrote:
> I think any module with more than 2000 lines of code is too big.

newCTFE has more than 2000 lines of code and is still growing,
There is no way it could be split up.

On 12/17/16 10:21 PM, pineapple wrote:
> On Sunday, 18 December 2016 at 02:40:59 UTC, Chris Wright wrote:
>> D doesn't have either of those pitfalls, so I haven't seen it cause
>> problems. I'm also a bit skeptical that this will see much use outside
>> phobos.
>>
>> This isn't really an argument against it. I just don't see any
>> argument for it, not that's supported by my own experience.
>
> I would like to echo this sentiment.
>
> I am developing a general-use library for D that is currently resting at
> around 50,000 lines.

Is the source code publicly available?

> I have never felt a need for a feature like this,
> and I can't imagine a reason to begin using it. Dependency management
> has just never presented an issue. Very nearly all modules in the
> library are fewer than 1,000 lines long and very nearly all symbols are
> selectively imported, and the approach has proven to be completely
> manageable.
>
> If it can be added without interfering with the existing patterns, I
> don't really have an argument against this feature. But I do think that
> what this DIP is meant to address is not really a problem experienced by
> all or even most who are working with D.

I recall Liran Zvibel repeatedly mentioned this as a big pain point at Weka.io. I will reach out to him.

On what basis do you extrapolate from personal experience to most or all who are working with D?

> It's a problem being
> experienced with Phobos, but there are very valid solutions to that
> problem that don't involve an addition to the language - only some
> refactoring. I think that makes the argument in favor somewhat weak.

How is Phobos special?

What kind of refactoring do you envision would improve dependency management and build times?


Thanks,

Andrei

On 12/18/16 6:47 AM, Jacob Carlborg wrote:
> On 2016-12-18 01:34, Andrei Alexandrescu wrote:
>
>> Yeah, std/datetime.d is a monster, from what I can tell owing to a rote
>> and redundant way of handling unittesting. I didn't look at its
>> dependencies, but I doubt they are special. I was quite vocal about
>> breaking it up, but I got mellower with time since (a) someone measured
>> its size without unittests and it was something like one order of
>> magnitude smaller, and (b) there was really no more trouble using or
>> maintaining it than with anything else in Phobos.
>
> Most other languages don't have inline unit tests, which saves a lot of
> lines of code.

How does the lack of the feature save lines? Doesn't it just move them elsewhere?

> Not sure if this is the case. But if we have unit tests that are more on
> the functional/integration side perhaps those should be moved to a
> separate file structure.
>
>> I should also add that each large project has a couple of outliers like
>> that. I even recall a switch of a couple thousand lines once :o).
>
> Just because another project is worse doesn't mean we're in a good
> position.

The point was just that it's par for the course.

> Rubocop, the major linter in the Ruby world, will complain if a class is
> more than 100 lines of code.

Does that include full documentation and unittests?

> I think that is on the extreme side but I
> think any module with more than 2000 lines of code is too big.

Phobos' average file size 2055 is lines and median file size is 903 lines. Do you find these appropriate for your preferences, considering this size includes documentation and unittests? Would you find it helpful to use a tool that collapses these?


Andrei

On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote:
> On 12/17/16 10:21 PM, pineapple wrote:
>> I am developing a general-use library for D that is currently resting at
>> around 50,000 lines.
>
> Is the source code publicly available?

https://github.com/pineapplemachine/mach.d

On 12/18/2016 5:38 AM, Andrei Alexandrescu wrote:
> On 12/18/16 6:47 AM, Jacob Carlborg wrote:
>> Rubocop, the major linter in the Ruby world, will complain if a class is
>> more than 100 lines of code.
>
> Does that include full documentation and unittests?

My complaint is more along the lines of aggregates that seem to have scores of member functions.

It brings to mind Scott Meyers' article about kitchen sink aggregates:

http://www.drdobbs.com/cpp/how-non-member-functions-improve-encapsu/184401197

On Thursday, 15 December 2016 at 22:56:42 UTC, Dmitry Olshansky wrote:
> On 12/13/16 11:33 PM, Andrei Alexandrescu wrote:
>> Destroy.
>>
>> https://github.com/dlang/DIPs/pull/51/files
>>
>>
>> Andrei
>
> On first it seems like an awesome idea. That solves ... but wait what? Thinking more  about the problem at hand - I fail to see what this DIP accomplishes that can't be done better today.
>
> 1. The benefit of placing import to each function is based on the untold assumption that we have:
> a) huge modules with many functions
> b) that constraints of these functions require different (large) modules
>
> The reality is far from this picture - if anything 99% of template constraints are dependent on std.range.primitives and std.traits with a bit of std.meta from time to time. So we'd have a boilerplate of
>
> auto foo(R)(R range)
> (import std.range.primitives)
> if(isInputRange!R){ ... }
>
> everywhere for no noticeable benefit - touch one of functions and you get full set of imports of these _small_ modules.
>
> 2. By itself the mechanism for delaying import even for constraint until the function is touched is moot as long as the module in question is huge and is not split in pieces. In other words:
>
> auto foo(R)(R range)
> (import std.range)
> if(isInputRange!R){ ... }
>
> Pulls in full std.range the moment foo is touched, compared to
>
> import std.range.primitives;
> ...
> auto foo(R)(R range)
> if(isInputRange!R){ ... }
>
> which is because it actually isolates the whole mess of complete std.range from the user of a template.
>
> All in all my practical response is split the modules at least in 2 parts: constraints + full functionality, then import the one with constraints at the top level. Works today and solves the practical issues unlike the proposal.
>
> ---
> Dmitry Olshansky

I largely agree with Dmitry. Ilya refactored several Phobos modules to use scoped, selective imports much more, and I pitched in for some remaining imports in the largest modules, so that only these module-level imports remain, ie those necessary for symbols imported in template constraints:

std.datetime - https://github.com/dlang/phobos/pull/4373/files
std.uni - https://github.com/dlang/phobos/pull/4365/files
std.string and std.traits - https://github.com/dlang/phobos/pull/4370/files

When I first saw this DIP, like Dmitry I was happy that we could get rid of those too, but the more I see these horrible syntax suggestions for what is really a minor convenience, I changed my mind.  std.datetime, the 35k line (17 kloc according to Dscanner) beast of phobos, only needs 20 or so symbols at module-scope. std.uni- 10k lines, 4.2 kloc- only needs 17 symbols, all from the three modules Dmitry mentioned.  I don't think his workaround of splitting up modules is even needed for such a low amount of module-level imports.

Maybe there are other issues having to do with symbol resolution and dependency encapsulation that are addressed by this DIP, ie the technical performance of the compiler rather than refactoring or code clarity, that I don't fully grasp, but from the first two points of the claimed benefits of DCDs, ie ease of reasoning about dependencies and refactoring code, I don't think this feature will come anywhere close to carrying its own weight.

As for the third benefit having to do with scalable template libraries, I'm not sure I completely understand all the details there, but I wonder if those problems aren't an artifact of the way dmd works now rather than something that can't be fixed without this DIP.

On 12/18/16 1:03 PM, Joakim wrote:
> I largely agree with Dmitry. Ilya refactored several Phobos modules to
> use scoped, selective imports much more, and I pitched in for some
> remaining imports in the largest modules, so that only these
> module-level imports remain, ie those necessary for symbols imported in
> template constraints:
>
> std.datetime - https://github.com/dlang/phobos/pull/4373/files
> std.uni - https://github.com/dlang/phobos/pull/4365/files
> std.string and std.traits - https://github.com/dlang/phobos/pull/4370/files

Yah, there's been a lot of good work (Jack Stouffer did a lot as well IIRC) on pushing imports inside. The following searches should be relevant:

git grep '^\(private \)\?import' | wc -l

yields about 426 top-level import declarations. The number of indented imports is 4605:

git grep '     *import\W' | wc -l

So we're looking at 10% of imports being problematic. Sadly, they turn out to make things quite difficult for Phobos. Last time I looked at the module dependency graph it wasn't a lot better than it used to before scoped imports. (I don't have it handy... could anyone please produce it?)

> When I first saw this DIP, like Dmitry I was happy that we could get rid
> of those too, but the more I see these horrible syntax suggestions for
> what is really a minor convenience, I changed my mind.  std.datetime,
> the 35k line (17 kloc according to Dscanner) beast of phobos, only needs
> 20 or so symbols at module-scope. std.uni- 10k lines, 4.2 kloc- only
> needs 17 symbols, all from the three modules Dmitry mentioned.  I don't
> think his workaround of splitting up modules is even needed for such a
> low amount of module-level imports.

This paragraph is a good example of a couple of counterarguments that I think point directly to flaws in the DIP:

(1) The DIP uses Phobos as an example, so it is applicable mostly to Phobos. In fact, Phobos is among the systems that would benefit least from the DIP: it has only druntime as dependency, and is distributed in its entirety. Many projects out there list multiple dependencies and may have various building and distribution policies. The converse is to believe that working around a problem in Phobos would render the DIP less useful in general.

(2) "I don't like the syntax, hence I don't like the feature." I see this as a good opportunity for tasteful design, not a downside of the feature.

> Maybe there are other issues having to do with symbol resolution and
> dependency encapsulation that are addressed by this DIP, ie the
> technical performance of the compiler rather than refactoring or code
> clarity, that I don't fully grasp, but from the first two points of the
> claimed benefits of DCDs, ie ease of reasoning about dependencies and
> refactoring code, I don't think this feature will come anywhere close to
> carrying its own weight.

Does the refactoring in https://github.com/dlang/phobos/pull/4962 make dependencies clearer? Are you e.g. clear on what you need in order to use Appender? Would you want to take this (or another) experiment to another module and see how it improves its dependency structure?

> As for the third benefit having to do with scalable template libraries,
> I'm not sure I completely understand all the details there, but I wonder
> if those problems aren't an artifact of the way dmd works now rather
> than something that can't be fixed without this DIP.

The DIP now dedicates an entire section to the pluses and minuses of lazy imports, and concludes that lazy imports would address scalability if carefully used.


Andrei

On 12/18/2016 10:01 AM, pineapple wrote:
> On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote:
>> On 12/17/16 10:21 PM, pineapple wrote:
>>> I am developing a general-use library for D that is currently resting at
>>> around 50,000 lines.
>>
>> Is the source code publicly available?
>
> https://github.com/pineapplemachine/mach.d

The code looks clean, congrats. With your permission I'd like to give this library as an example in the "Workaround: Increasing Granularity of Modules" section. Please advise, thanks. -- Andrei

On Sunday, 18 December 2016 at 21:09:46 UTC, Andrei Alexandrescu wrote:
> On 12/18/2016 10:01 AM, pineapple wrote:
>> On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote:
>>> Is the source code publicly available?
>>
>> https://github.com/pineapplemachine/mach.d
>
> The code looks clean, congrats. With your permission I'd like to give this library as an example in the "Workaround: Increasing Granularity of Modules" section. Please advise, thanks. -- Andrei

No problem, feel free. Thank you for asking.

On 12/18/2016 05:18 PM, pineapple wrote:
> On Sunday, 18 December 2016 at 21:09:46 UTC, Andrei Alexandrescu wrote:
>> On 12/18/2016 10:01 AM, pineapple wrote:
>>> On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote:
>>>> Is the source code publicly available?
>>>
>>> https://github.com/pineapplemachine/mach.d
>>
>> The code looks clean, congrats. With your permission I'd like to give
>> this library as an example in the "Workaround: Increasing Granularity
>> of Modules" section. Please advise, thanks. -- Andrei
>
> No problem, feel free. Thank you for asking.

Is there a simple command to e.g. unittest everything in the project? Also, is there a build process or it's all templated? -- Andrei