View mode: basic / threaded / horizontal-split · Log in · Help
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
Le 13/06/2012 11:37, Iain Buclaw a écrit :
> On 13 June 2012 09:07, Don Clugston<dac@nospam.com>  wrote:
>> On 12/06/12 18:46, Walter Bright wrote:
>>>
>>> On 6/12/2012 2:07 AM, timotheecour wrote:
>>>>
>>>> There's a current pull request to improve di file generation
>>>> (https://github.com/D-Programming-Language/dmd/pull/945); I'd like to
>>>> suggest
>>>> further ideas.
>>>> As far as I understand, di interface files try to achieve these
>>>> conflicting goals:
>>>>
>>>> 1) speed up compilation by avoiding having to reparse large files over
>>>> and over.
>>>> 2) hide implementation details for proprietary reasons
>>>> 3) still maintain source code in some form to allow inlining and CTFE
>>>> 4) be human readable
>>>
>>>
>>> (4) was not a goal.
>>>
>>> A .di file could very well be a binary file, but making it look like D
>>> source enabled them to be loaded with no additional implementation work
>>> in the compiler.
>>
>>
>> I don't understand (1) actually.
>>
>> For two reasons:
>> (a) Is lexing + parsing really a significant part of the compilation time?
>> Has anyone done some solid profiling?
>>
>
> Lexing and Parsing are miniscule tasks in comparison to the three
> semantic runs done on the code.
>
> I added speed counters into the glue code of GDC some time ago.
> http://iainbuclaw.wordpress.com/2010/09/18/implementing-speed-counters-in-gdc/
>
> And here is the relavent report to go with it.
> http://iainbuclaw.files.wordpress.com/2010/09/d2-time-report2.pdf
>
>
> Example: std/xml.d
> Module::parse : 0.01 ( 0%)
> Module::semantic : 0.50 ( 9%)
> Module::semantic2 : 0.02 ( 0%)
> Module::semantic3 : 0.04 ( 1%)
> Module::genobjfile : 0.10 ( 2%)
>
> For the entire time it took to compile the one file (5.22 seconds) -
> it spent almost 10% of it's time running the first semantic analysis.
>
>
> But that was the D2 frontend / phobos as of September 2010.  I should
> re-run a report on updated times and draw some comparisons. :~)
>
>
> Regards

Nice numbers ! It also show that the slowest part is the backend.

Can you get some number on a recent version of D ? And in some different 
D codes (ie, template intensive or not for instance is nice to compare).
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 13 June 2012 10:45, Dmitry Olshansky <dmitry.olsh@gmail.com> wrote:
> On 13.06.2012 13:37, Iain Buclaw wrote:
>>
>> On 13 June 2012 09:07, Don Clugston<dac@nospam.com>  wrote:
>>>
>>> On 12/06/12 18:46, Walter Bright wrote:
>>>
>>>>
>>>> On 6/12/2012 2:07 AM, timotheecour wrote:
>>>>>
>>>>>
>>>>> There's a current pull request to improve di file generation
>>>>> (https://github.com/D-Programming-Language/dmd/pull/945); I'd like to
>>>>> suggest
>>>>> further ideas.
>>>>> As far as I understand, di interface files try to achieve these
>>>>> conflicting goals:
>>>>>
>>>>> 1) speed up compilation by avoiding having to reparse large files over
>>>>> and over.
>>>>> 2) hide implementation details for proprietary reasons
>>>>> 3) still maintain source code in some form to allow inlining and CTFE
>>>>> 4) be human readable
>>>>
>>>>
>>>>
>>>> (4) was not a goal.
>>>>
>>>> A .di file could very well be a binary file, but making it look like D
>>>> source enabled them to be loaded with no additional implementation work
>>>> in the compiler.
>>>
>>>
>>>
>>> I don't understand (1) actually.
>>>
>>> For two reasons:
>>> (a) Is lexing + parsing really a significant part of the compilation
>>> time?
>>> Has anyone done some solid profiling?
>>>
>>
>> Lexing and Parsing are miniscule tasks in comparison to the three
>> semantic runs done on the code.
>>
>> I added speed counters into the glue code of GDC some time ago.
>>
>> http://iainbuclaw.wordpress.com/2010/09/18/implementing-speed-counters-in-gdc/
>>
>> And here is the relavent report to go with it.
>> http://iainbuclaw.files.wordpress.com/2010/09/d2-time-report2.pdf
>>
>>
>> Example: std/xml.d
>> Module::parse : 0.01 ( 0%)
>> Module::semantic : 0.50 ( 9%)
>> Module::semantic2 : 0.02 ( 0%)
>> Module::semantic3 : 0.04 ( 1%)
>> Module::genobjfile : 0.10 ( 2%)
>>
>> For the entire time it took to compile the one file (5.22 seconds) -
>> it spent almost 10% of it's time running the first semantic analysis.
>>
>>
>> But that was the D2 frontend / phobos as of September 2010.  I should
>> re-run a report on updated times and draw some comparisons. :~)
>>
>
> Is time spent on I/O accounted for in the parse step? And where is the rest
> spent :)
>

It would be, the counter starts before the files are even touched, and
ends after they are closed.

The rest of the time spent is in the GCC backend, going through the
some 60+ code passes and outputting the assembly to file.


-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 13.06.2012 14:16, Iain Buclaw wrote:
> On 13 June 2012 10:45, Dmitry Olshansky<dmitry.olsh@gmail.com>  wrote:
>> On 13.06.2012 13:37, Iain Buclaw wrote:
>>>
>>> On 13 June 2012 09:07, Don Clugston<dac@nospam.com>    wrote:
>>>>
>>>> On 12/06/12 18:46, Walter Bright wrote:
>>>>
>>>>>
>>>>> On 6/12/2012 2:07 AM, timotheecour wrote:
>>>>>>
>>>>>>
>>>>>> There's a current pull request to improve di file generation
>>>>>> (https://github.com/D-Programming-Language/dmd/pull/945); I'd like to
>>>>>> suggest
>>>>>> further ideas.
>>>>>> As far as I understand, di interface files try to achieve these
>>>>>> conflicting goals:
>>>>>>
>>>>>> 1) speed up compilation by avoiding having to reparse large files over
>>>>>> and over.
>>>>>> 2) hide implementation details for proprietary reasons
>>>>>> 3) still maintain source code in some form to allow inlining and CTFE
>>>>>> 4) be human readable
>>>>>
>>>>>
>>>>>
>>>>> (4) was not a goal.
>>>>>
>>>>> A .di file could very well be a binary file, but making it look like D
>>>>> source enabled them to be loaded with no additional implementation work
>>>>> in the compiler.
>>>>
>>>>
>>>>
>>>> I don't understand (1) actually.
>>>>
>>>> For two reasons:
>>>> (a) Is lexing + parsing really a significant part of the compilation
>>>> time?
>>>> Has anyone done some solid profiling?
>>>>
>>>
>>> Lexing and Parsing are miniscule tasks in comparison to the three
>>> semantic runs done on the code.
>>>
>>> I added speed counters into the glue code of GDC some time ago.
>>>
>>> http://iainbuclaw.wordpress.com/2010/09/18/implementing-speed-counters-in-gdc/
>>>
>>> And here is the relavent report to go with it.
>>> http://iainbuclaw.files.wordpress.com/2010/09/d2-time-report2.pdf
>>>
>>>
>>> Example: std/xml.d
>>> Module::parse : 0.01 ( 0%)
>>> Module::semantic : 0.50 ( 9%)
>>> Module::semantic2 : 0.02 ( 0%)
>>> Module::semantic3 : 0.04 ( 1%)
>>> Module::genobjfile : 0.10 ( 2%)
>>>
>>> For the entire time it took to compile the one file (5.22 seconds) -
>>> it spent almost 10% of it's time running the first semantic analysis.
>>>
>>>
>>> But that was the D2 frontend / phobos as of September 2010.  I should
>>> re-run a report on updated times and draw some comparisons. :~)
>>>
>>
>> Is time spent on I/O accounted for in the parse step? And where is the rest
>> spent :)
>>
>
> It would be, the counter starts before the files are even touched, and
> ends after they are closed.

Ok, then parsing is indistinguishable from I/O and together are only 
tiny fraction of the whole. Great info, thanks.

>
> The rest of the time spent is in the GCC backend, going through the
> some 60+ code passes and outputting the assembly to file.
>

Damn, I like DMD :)



-- 
Dmitry Olshansky
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
The measurements should be done for modules being imported, not 
the module being compiled.
Something like this.
---
import std.algorithm;
import std.stdio;
import std.typecons;
import std.datetime;

int ok;
---
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On Wednesday, 13 June 2012 at 11:29:45 UTC, Kagamin wrote:
> The measurements should be done for modules being imported, not 
> the module being compiled.
> Something like this.
> ---
> import std.algorithm;
> import std.stdio;
> import std.typecons;
> import std.datetime;
>
> int ok;
> ---

Oh and let it import .d files, not .di
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 13 June 2012 12:33, Kagamin <spam@here.lot> wrote:
> On Wednesday, 13 June 2012 at 11:29:45 UTC, Kagamin wrote:
>>
>> The measurements should be done for modules being imported, not the module
>> being compiled.
>> Something like this.
>> ---
>> import std.algorithm;
>> import std.stdio;
>> import std.typecons;
>> import std.datetime;
>>
>> int ok;
>> ---
>
>
> Oh and let it import .d files, not .di

std.datetime is one reason for me to run it again. I can imagine that
*that* module will have an impact on parse times.  But I'm still
persistent that the majority of the compile time in the frontend is
done in the first semantic pass, and not the read/parser stage. :~)


-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On Wednesday, 13 June 2012 at 11:47:31 UTC, Iain Buclaw wrote:
> std.datetime is one reason for me to run it again. I can 
> imagine that
> *that* module will have an impact on parse times.  But I'm still
> persistent that the majority of the compile time in the 
> frontend is
> done in the first semantic pass, and not the read/parser stage. 
> :~)

Probably. Also test with -fsyntax-only is it works and runs 
semantic passes.
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 6/13/2012 1:07 AM, Don Clugston wrote:
> On 12/06/12 18:46, Walter Bright wrote:
>> On 6/12/2012 2:07 AM, timotheecour wrote:
>>> There's a current pull request to improve di file generation
>>> (https://github.com/D-Programming-Language/dmd/pull/945); I'd like to
>>> suggest
>>> further ideas.
>>> As far as I understand, di interface files try to achieve these
>>> conflicting goals:
>>>
>>> 1) speed up compilation by avoiding having to reparse large files over
>>> and over.
>>> 2) hide implementation details for proprietary reasons
>>> 3) still maintain source code in some form to allow inlining and CTFE
>>> 4) be human readable
>>
>> (4) was not a goal.
>>
>> A .di file could very well be a binary file, but making it look like D
>> source enabled them to be loaded with no additional implementation work
>> in the compiler.
>
> I don't understand (1) actually.
>
> For two reasons:
> (a) Is lexing + parsing really a significant part of the compilation time? Has
> anyone done some solid profiling?

It is for debug builds.


> (b) Wasn't one of the goals of D's module system supposed to be that you could
> import a symbol table? Why not just implement that? Seems like that would be
> much faster than .di files can ever be.

Yes, it is designed so you could just import a symbol table. It is done as 
source code, however, because it's trivial to implement.
June 13, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 2012-06-13 13:47, Iain Buclaw wrote:

> std.datetime is one reason for me to run it again. I can imagine that
> *that* module will have an impact on parse times.  But I'm still
> persistent that the majority of the compile time in the frontend is
> done in the first semantic pass, and not the read/parser stage. :~)

You should try the Objective-C/D bridge, that took quite a while to 
compile. Although it will probably not compile any more, haven't been 
update. I think it was only for D1 as well. I think that was most 
templates so I guess that would mean the some of the semantic passes.

-- 
/Jacob Carlborg
June 14, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 13/06/12 16:29, Walter Bright wrote:
> On 6/13/2012 1:07 AM, Don Clugston wrote:
>> On 12/06/12 18:46, Walter Bright wrote:
>>> On 6/12/2012 2:07 AM, timotheecour wrote:
>>>> There's a current pull request to improve di file generation
>>>> (https://github.com/D-Programming-Language/dmd/pull/945); I'd like to
>>>> suggest
>>>> further ideas.
>>>> As far as I understand, di interface files try to achieve these
>>>> conflicting goals:
>>>>
>>>> 1) speed up compilation by avoiding having to reparse large files over
>>>> and over.
>>>> 2) hide implementation details for proprietary reasons
>>>> 3) still maintain source code in some form to allow inlining and CTFE
>>>> 4) be human readable
>>>
>>> (4) was not a goal.
>>>
>>> A .di file could very well be a binary file, but making it look like D
>>> source enabled them to be loaded with no additional implementation work
>>> in the compiler.
>>
>> I don't understand (1) actually.
>>
>> For two reasons:
>> (a) Is lexing + parsing really a significant part of the compilation
>> time? Has
>> anyone done some solid profiling?
>
> It is for debug builds.

Iain's data indicates that it's only a few % of the time taken on 
semantic1().
Do you have data that shows otherwise?

It seems to me, that slow parsing is a C++ problem which D already solved.

>
>> (b) Wasn't one of the goals of D's module system supposed to be that
>> you could
>> import a symbol table? Why not just implement that? Seems like that
>> would be
>> much faster than .di files can ever be.
>
> Yes, it is designed so you could just import a symbol table. It is done
> as source code, however, because it's trivial to implement.

It has those nasty side-effects listed under (3) though.
1 2 3 4 5 6
Top | Discussion index | About this forum | D home