View mode: basic / threaded / horizontal-split · Log in · Help
June 18, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 6/18/2012 6:07 AM, Don Clugston wrote:
> On 17/06/12 00:37, Walter Bright wrote:
>> On 6/14/2012 1:03 AM, Don Clugston wrote:
>>>> It is for debug builds.
>>> Iain's data indicates that it's only a few % of the time taken on
>>> semantic1().
>>> Do you have data that shows otherwise?
>>
>> Nothing recent, it's mostly from my C++ compiler testing.
>
> But you argued in your blog that C++ parsing is inherently slow, and you've
> fixed those problems in the design of D.
> And as far as I can tell, you were extremely successful!
> Parsing in D is very, very fast.

Yeah, but I can't escape that lingering feeling that lexing is slow.

I was fairly disappointed that asynchronously reading the source files didn't 
have a measurable effect most of the time.
June 18, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On Monday, 18 June 2012 at 17:54:40 UTC, Walter Bright wrote:
> On 6/18/2012 6:07 AM, Don Clugston wrote:
>> On 17/06/12 00:37, Walter Bright wrote:
>>> On 6/14/2012 1:03 AM, Don Clugston wrote:
>>>>> It is for debug builds.
>>>> Iain's data indicates that it's only a few % of the time 
>>>> taken on
>>>> semantic1().
>>>> Do you have data that shows otherwise?
>>>
>>> Nothing recent, it's mostly from my C++ compiler testing.
>>
>> But you argued in your blog that C++ parsing is inherently 
>> slow, and you've
>> fixed those problems in the design of D.
>> And as far as I can tell, you were extremely successful!
>> Parsing in D is very, very fast.
>
> Yeah, but I can't escape that lingering feeling that lexing is 
> slow.
>
> I was fairly disappointed that asynchronously reading the 
> source files didn't have a measurable effect most of the time.

Same here, I wish there were a standardized pre-lexed-token 
"binary" file-format, would benefit all text editors also, as 
they need to lex it anyway to perform color syntax highlighting.
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On Monday, 18 June 2012 at 18:05:59 UTC, Daniel wrote:
> Same here, I wish there were a standardized pre-lexed-token 
> "binary" file-format, would benefit all text editors also, as 
> they need to lex it anyway to perform color syntax highlighting.

If I were to make my own language, I'd forego a human-readable 
format and just have the "language" be defined as a big 
machine-readable AST. You'd have to have an IDE, but it could 
display the code in just about any way the person wants (syntax, 
style, etc).

Syntax highlighting would be instantaneous and there would be 
fewer errors made by programmers (maybe ...). Plus it'd be 
unbelievably easy to implement things like auto-completion.
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 06/19/2012 02:47 AM, Chris Cain wrote:
> On Monday, 18 June 2012 at 18:05:59 UTC, Daniel wrote:
>> Same here, I wish there were a standardized pre-lexed-token "binary"
>> file-format, would benefit all text editors also, as they need to lex
>> it anyway to perform color syntax highlighting.
>
> If I were to make my own language, I'd forego a human-readable format
> and just have the "language" be defined as a big machine-readable AST.

http://de.wikipedia.org/wiki/Lisp ?

> You'd have to have an IDE, but it could display the code in just about
> any way the person wants (syntax, style, etc).
>

This could be done even if the language's source code storage format is 
human-readable.

> Syntax highlighting would be instantaneous and there would be fewer
> errors made by programmers (maybe ...). Plus it'd be unbelievably easy
> to implement things like auto-completion.

Parsing is not a huge issue. Depending on how powerful the language is, 
auto-completion may depend on full code analysis.
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On Monday, 18 June 2012 at 17:54:40 UTC, Walter Bright wrote:
> Yeah, but I can't escape that lingering feeling that lexing is 
> slow.
>
> I was fairly disappointed that asynchronously reading the 
> source files didn't have a measurable effect most of the time.

I don't even understand all this rage about asynchronicity, if 
the program has nothing to do until it reads the data, 
asynchronicity won't help you in the slightest. Anyway everything 
is stuck while the device performs DMA.
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On Tuesday, 19 June 2012 at 01:47:27 UTC, Timon Gehr wrote:
> Parsing is not a huge issue. Depending on how powerful the 
> language is, auto-completion may depend on full code analysis.

Yep, pegged runs at compile time.
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
Am 19.06.2012 09:43, schrieb Kagamin:
> On Monday, 18 June 2012 at 17:54:40 UTC, Walter Bright wrote:
>> Yeah, but I can't escape that lingering feeling that lexing is
>> slow.
>>
>> I was fairly disappointed that asynchronously reading the
>> source files didn't have a measurable effect most of the time.

> I don't even understand all this rage about asynchronicity, if
> the program has nothing to do until it reads the data,

the lexing and parsing process can be asynchron - i will be faster on 
multiple cores because there is no dependency between seperated 
lexing-parsing threads - why to lex/parse in sequence then?

> asynchronicity won't help you in the slightest. Anyway everything
> is stuck while the device performs DMA.

yea down to the hardware level - but there are caches etc. out there - 
its not like "multithreaded-file-reading-is-always-fast-like-synchron", 
and also not "asynchron-file-reading-is-always-faster" - more somewere 
in between :)
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
Am 18.06.2012 19:53, schrieb Walter Bright:
> On 6/18/2012 6:07 AM, Don Clugston wrote:
>> On 17/06/12 00:37, Walter Bright wrote:
>>> On 6/14/2012 1:03 AM, Don Clugston wrote:
>>>>> It is for debug builds.
>>>> Iain's data indicates that it's only a few % of the time taken on
>>>> semantic1().
>>>> Do you have data that shows otherwise?
>>>
>>> Nothing recent, it's mostly from my C++ compiler testing.
>>
>> But you argued in your blog that C++ parsing is inherently slow, and you've
>> fixed those problems in the design of D.
>> And as far as I can tell, you were extremely successful!
>> Parsing in D is very, very fast.
>
> Yeah, but I can't escape that lingering feeling that lexing is slow.
>
> I was fairly disappointed that asynchronously reading the source files didn't
> have a measurable effect most of the time.

so you started you lexing, parsing in seperated threads for each file - 
where was synchronization needed, have you measured what parts of the 
code makes it like synchron reading - or is it the file reading itself?
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
Le 16/06/2012 11:18, Iain Buclaw a écrit :
> On 13 June 2012 12:47, Iain Buclaw<ibuclaw@ubuntu.com>  wrote:
>> On 13 June 2012 12:33, Kagamin<spam@here.lot>  wrote:
>>> On Wednesday, 13 June 2012 at 11:29:45 UTC, Kagamin wrote:
>>>>
>>>> The measurements should be done for modules being imported, not the module
>>>> being compiled.
>>>> Something like this.
>>>> ---
>>>> import std.algorithm;
>>>> import std.stdio;
>>>> import std.typecons;
>>>> import std.datetime;
>>>>
>>>> int ok;
>>>> ---
>>>
>>>
>>> Oh and let it import .d files, not .di
>>
>> std.datetime is one reason for me to run it again. I can imagine that
>> *that* module will have an impact on parse times.  But I'm still
>> persistent that the majority of the compile time in the frontend is
>> done in the first semantic pass, and not the read/parser stage. :~)
>>
>>
>
> Rebuilt a compile log with latest gdc as of writing on the 2.059
> frontend / library.
>
> http://iainbuclaw.files.wordpress.com/2012/06/d2time_report32_2059.pdf
> http://iainbuclaw.files.wordpress.com/2012/06/d2time_report64_2059.pdf
>
>
> Notes about it:
> - GCC has 4 new time counters
>    -  phase setup  (time spent loading the compile time environment)
>    -  phase parsing  (time spent in the frontend)
>    -  phase generate (time spent in the backend)
>    -  phase finalize  (time spent cleaning up and exiting)
>
> - Of the phase parsing stage, it is broken down into 5 components
>    -  Module::parse
>    -  Module::semantic
>    -  Module::semantic2
>    -  Module::semantic3
>    -  Module::genobjfile
>
> - Module::read, Module::parse and Module::importAll in the one I did 2
> years ago are now counted as part of just the one parsing stage,
> rather than separate just to make it a little bit more balanced. :-)
>
>
> I'll post a tl;dr later on it.
>

Thank you very much for your work.
June 19, 2012
Re: AST files instead of DI interface files for faster compilation and easier distribution
On 16 June 2012 22:17, Guillaume Chatelet <chatelet.guillaume@gmail.com> wrote:
>> So parsing time has taken quite a hit since I last did any reports on
>> compilation speed of building phobos.
>
> So maybe my post about "keeping import clean" wasn't as irrelevant as I
> thought.
>
> http://www.digitalmars.com/d/archives/digitalmars/D/Keeping_imports_clean_162890.html#N162890
>
> --
> Guillaume


I think it's relevancy is only geared towards projects that are
compiling one file at a time - ie: I'd expect all gdc users to be
compiling in this way as whole program compilation using gdc still
needs some rigourous testing first.  If there is a particular large
module, or set of large modules that are persistantly being
importanted, then you will see a notable constant slowdown on
compilation of each file.


-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
1 2 3 4 5 6
Top | Discussion index | About this forum | D home