Following the discussion at https://github.com/dlang/phobos/pull/5916, we realized that there's quite some overhead in importing some Phobos modules. Ideally importing a Phobos module without using it, should be cost-free (i.e. pay for what you use).

Test case: consider a file with `import std.X` and compile it with `dmd -c -o-`.

A quick overview of the worst of the worst:

std.regex	1.03s
std.net.curl	0.28s
std.concurrency	0.23s
std.encoding	0.18s
std.zip	        0.16s
std.path	0.14s
std.file	0.14s
std.mmfile	0.14s
std.datetime	0.14s
std.socket	0.12s
std.string	0.10s
std.uni	        0.09s
std.process	0.05s

This following discussions have already brought a huge improvements to its light:

https://github.com/dlang/phobos/pull/5931

Can you do a another one?

On 12/15/2017 02:10 PM, Seb wrote:
> Following the discussion at https://github.com/dlang/phobos/pull/5916, we realized that there's quite some overhead in importing some Phobos modules. Ideally importing a Phobos module without using it, should be cost-free (i.e. pay for what you use).
> 
> Test case: consider a file with `import std.X` and compile it with `dmd -c -o-`.
> 
> A quick overview of the worst of the worst:
> 
> std.regex    1.03s
> std.net.curl    0.28s
> std.concurrency    0.23s
> std.encoding    0.18s
> std.zip            0.16s
> std.path    0.14s
> std.file    0.14s
> std.mmfile    0.14s
> std.datetime    0.14s
> std.socket    0.12s
> std.string    0.10s
> std.uni            0.09s
> std.process    0.05s
> 
> This following discussions have already brought a huge improvements to its light:
> 
> https://github.com/dlang/phobos/pull/5931
> 
> Can you do a another one?

Dmitry wrote a nice PR for that, and I wrote two:

https://github.com/dlang/phobos/pull/5942
https://github.com/dlang/phobos/pull/5931
https://github.com/dlang/phobos/pull/5948

My approach has been the following:

* First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability.

* Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest.

* Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.)

Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter.

A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit:

depend --explicit *.d

This run would push all imports down to the innermost scope of usage:

depend --pushdown *.d


Andrei

On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:
> On 12/15/2017 02:10 PM, Seb wrote:
>> Following the discussion at https://github.com/dlang/phobos/pull/5916, we realized that there's quite some overhead in importing some Phobos modules. Ideally importing a Phobos module without using it, should be cost-free (i.e. pay for what you use).
>> 
>> Test case: consider a file with `import std.X` and compile it with `dmd -c -o-`.
>> 
>> A quick overview of the worst of the worst:
>> 
>> std.regex    1.03s
>> std.net.curl    0.28s
>> std.concurrency    0.23s
>> std.encoding    0.18s
>> std.zip            0.16s
>> std.path    0.14s
>> std.file    0.14s
>> std.mmfile    0.14s
>> std.datetime    0.14s
>> std.socket    0.12s
>> std.string    0.10s
>> std.uni            0.09s
>> std.process    0.05s
>> 
>> This following discussions have already brought a huge improvements to its light:
>> 
>> https://github.com/dlang/phobos/pull/5931
>> 
>> Can you do a another one?
>
> Dmitry wrote a nice PR for that, and I wrote two:
>
> https://github.com/dlang/phobos/pull/5942
> https://github.com/dlang/phobos/pull/5931
> https://github.com/dlang/phobos/pull/5948
>
> My approach has been the following:
>
> * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability.
>
> * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest.
>
> * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.)
>
> Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter.
>
> A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit:
>
> depend --explicit *.d
>
> This run would push all imports down to the innermost scope of usage:
>
> depend --pushdown *.d
>
>
> Andrei

I'm on it.  I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:

https://github.com/dlang/dmd/pull/7425
https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47
https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f

Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.

On 12/20/2017 10:31 PM, Joakim wrote:
> On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:
>> A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit:
>>
>> depend --explicit *.d
>>
>> This run would push all imports down to the innermost scope of usage:
>>
>> depend --pushdown *.d
>>
>>
>> Andrei
> 
> I'm on it.  I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
> 
> https://github.com/dlang/dmd/pull/7425
> https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47
> https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f
> 
> Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.

That's an awesome start, thanks! -- Andrei

On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
> On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:
>> On 12/15/2017 02:10 PM, Seb wrote:
>>> [...]
>>
>> Dmitry wrote a nice PR for that, and I wrote two:
>>
>> https://github.com/dlang/phobos/pull/5942
>> https://github.com/dlang/phobos/pull/5931
>> https://github.com/dlang/phobos/pull/5948
>>
>> My approach has been the following:
>>
>> * First I tried to clarify which module imports what by means of replacing "import std.xxx;" with "import std.xxx : yyy, zzz, ...;" That has been of tremendous help in assessing each module's dependency liability.
>>
>> * Then I tried to figure which imports are most costly. I inserted __EOF__ in the module at different parts and measure how long it takes to just import that module. After a few attempts it becomes clear which imports are heaviest.
>>
>> * Then I pushed those heavy imports down where they're needed. An import placed in a function will not be opened while that function's module gets imported. Same goes about template types, but not structs or classes. (That should be fixed; see Seb's comment in my last PR.)
>>
>> Not only such work reduces import times, but it's illuminating for maintenance because it represents dependencies in a simple, fine-grained matter.
>>
>> A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit:
>>
>> depend --explicit *.d
>>
>> This run would push all imports down to the innermost scope of usage:
>>
>> depend --pushdown *.d
>>
>>
>> Andrei
>
> I'm on it.  I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:

I ran into the same issue. The problem was that currently the frontend
doesn't set the version identifiers. There's a PR which tries to fix it [1],
but if you want to go back at using the compiler library via Seb's interface
you just have to call this method [2] in the initialization step.

Anyway I am working on this sort of tool also, so maybe we can unite our efforts.
Ping me if you wish so,

[1] https://github.com/dlang/dmd/pull/7524
[2] https://github.com/dlang/dmd/blob/master/src/dmd/mars.d#L1314
>
> https://github.com/dlang/dmd/pull/7425
> https://github.com/dlang/dmd/blob/master/src/dmd/gluelayer.d#L47
> https://gist.github.com/joakim-noah/09cf49bee3d82b03a54f
>
> Once I have something basic working, I'll put it up on github, so others who are interested can play with it and pitch in to make it a real tool.

Best regards,
RazvanN

On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:
> On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
>> On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:
>>> A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit:
>>>
>>> depend --explicit *.d
>>>
>>> This run would push all imports down to the innermost scope of usage:
>>>
>>> depend --pushdown *.d
>>>
>>>
>>> Andrei
>>
>> I'm on it.  I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
>
> I ran into the same issue. The problem was that currently the frontend
> doesn't set the version identifiers. There's a PR which tries to fix it [1],
> but if you want to go back at using the compiler library via Seb's interface
> you just have to call this method [2] in the initialization step.

I doubt that's it, as I noted I was running the standard dmd main from mars.d.  More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.

> Anyway I am working on this sort of tool also, so maybe we can unite our efforts.

As in the `depend` import-scoping tool that Andrei wanted above?  If so, that'd be great, as you know the frontend much better than me.  I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd.

Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.

On Tuesday, 26 December 2017 at 15:38:14 UTC, Joakim wrote:
> On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:
>> On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
>>> On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:
>>>> A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit:
>>>>
>>>> depend --explicit *.d
>>>>
>>>> This run would push all imports down to the innermost scope of usage:
>>>>
>>>> depend --pushdown *.d
>>>>
>>>>
>>>> Andrei
>>>
>>> I'm on it.  I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
>>
>> I ran into the same issue. The problem was that currently the frontend
>> doesn't set the version identifiers. There's a PR which tries to fix it [1],
>> but if you want to go back at using the compiler library via Seb's interface
>> you just have to call this method [2] in the initialization step.
>
> I doubt that's it, as I noted I was running the standard dmd main from mars.d.  More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.
>
>> Anyway I am working on this sort of tool also, so maybe we can unite our efforts.
>
> As in the `depend` import-scoping tool that Andrei wanted above?  If so, that'd be great, as you know the frontend much better than me.  I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd.
>
> Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.

Yes, that was the tool I was talking about. I was thinking that the first
version of the tool just outputs for each import what the used symbols are;
for example : if you import std.stdio and use 3 symbols from it, the tool will output "std.stdio: sym1, sym2, sym2" and then the programmer can update the code accordingly. After developing this initial version we can think on how to improve it.

For now, my tool only identifies imported non-templated functions, but I plan on adding support variables, types etc.

We can chat on slack more about this if you are interested.

On Wednesday, 27 December 2017 at 10:40:38 UTC, RazvanN wrote:
> On Tuesday, 26 December 2017 at 15:38:14 UTC, Joakim wrote:
>> On Tuesday, 26 December 2017 at 13:27:38 UTC, RazvanN wrote:
>>> On Thursday, 21 December 2017 at 03:31:16 UTC, Joakim wrote:
>>>> On Wednesday, 20 December 2017 at 18:21:33 UTC, Andrei Alexandrescu wrote:
>>>>> A tool (call it depend - heh) to automate that would be awesome. For example, this run would make all imported names explicit:
>>>>>
>>>>> depend --explicit *.d
>>>>>
>>>>> This run would push all imports down to the innermost scope of usage:
>>>>>
>>>>> depend --pushdown *.d
>>>>>
>>>>>
>>>>> Andrei
>>>>
>>>> I'm on it.  I tried using a version of Seb's frontend library yesterday but with the standard dmd main and tried running it on std.stdio, but it asserted in asmSemantic because of an asm block somewhere in druntime, so I'm back to using a tweaked full dmd with the backend simply disabled again:
>>>
>>> I ran into the same issue. The problem was that currently the frontend
>>> doesn't set the version identifiers. There's a PR which tries to fix it [1],
>>> but if you want to go back at using the compiler library via Seb's interface
>>> you just have to call this method [2] in the initialization step.
>>
>> I doubt that's it, as I noted I was running the standard dmd main from mars.d.  More likely it tries to run asmSemantic on an asm block in core.atomic and asserts because that function's stubbed out when there's no backend.
>>
>>> Anyway I am working on this sort of tool also, so maybe we can unite our efforts.
>>
>> As in the `depend` import-scoping tool that Andrei wanted above?  If so, that'd be great, as you know the frontend much better than me.  I've been meaning to investigate how to extract a scope tree from the frontend, as I've never done much with dmd.
>>
>> Let me know how far you've gotten and what you want it to do initially, and maybe I can pitch in with your effort.
>
> Yes, that was the tool I was talking about. I was thinking that the first
> version of the tool just outputs for each import what the used symbols are;
> for example : if you import std.stdio and use 3 symbols from it, the tool will output "std.stdio: sym1, sym2, sym2" and then the programmer can update the code accordingly. After developing this initial version we can think on how to improve it.
>
> For now, my tool only identifies imported non-templated functions, but I plan on adding support variables, types etc.

I think I did all this with a dead-simple patch to dmd three years ago:

http://forum.dlang.org/thread/pbpckzwmfglzgwqveoza@forum.dlang.org

I used that tweaked dmd to scope imports for some modules in phobos last summer:

https://github.com/dlang/phobos/pulls?utf8=✓&q=is%3Apr+author%3Ajoakim-noah+is%3Aclosed+"selective+imports"

That's much better than the alternative of tracking the imported symbols down yourself, but it's still a pain to edit the files manually.  I'd rather it were integrated with Brian's libdparse to modify the source for you:

https://github.com/dlang-community/libdparse/

> We can chat on slack more about this if you are interested.

I don't use slack or any other chat platform.  Probably best to stick to the forum or github, so that anyone else who's interested can follow along.

On Wednesday, 27 December 2017 at 15:12:28 UTC, Joakim wrote:
> On Wednesday, 27 December 2017 at 10:40:38 UTC, RazvanN wrote:
>> [...]
>
> I think I did all this with a dead-simple patch to dmd three years ago:
>
> [...]

Hey Razvan, are you doing anything with this? Never heard back after my last post, so figured you were going your own way with this tool.