On Thursday, 11 December 2014 at 09:07:18 UTC, ketmar via Digitalmars-d wrote:
> On Thu, 11 Dec 2014 08:57:56 +0000
> Tobias Pankrath via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
>> >> 
>> >> Storing it as body IR accomplishes nothing practical over storing it as source file, i.e. .di files.
>> > except that there's no need to parse source code over and over again,
>> > which is good for other tools (like completion suggesting, intelligent
>> > code browsing and so on).
>> 
>> Which usually hold an AST in memory anyway. We have a fast parser, parsing even a big codebase once is really not a problem, see DCD for example.
>> 
>> If the only advantage is to skip a parsing stage here or there, it does not justify the work that would be needed.
> as we have a fast compiler too, i can't see any sense in producing
> machine code files at all. the only advantage is to skip a parsing and
> compiling stages here or there, it does not justify the work that would
> be needed.

Parsing is so fast it's not worth spending huge numbers of man-hours building an effective cacheing system for it. The rest of compilation is comparatively much slower and is therefore more important to cache.

You're being sarcastic to a straw-man.

On Wednesday, 10 December 2014 at 23:23:50 UTC, Walter Bright
wrote:
> On 12/10/2014 4:15 AM, Paulo Pinto wrote:
>> I prefer the model used by the referred languages, where binary libraries and
>> metadata is used, instead of the C toolchain model.
>>
>> For example, just shipping the .TPU/.DCU libraries in the Object Pascal world.
>
> If the metadata had enough info in it to do inlining, it might as well be the source code.

Er ... but you're the guy who stresses that lexers cannot be fast
enough because they need to look at every input char. (And you're
right.) You can at least cache the lexing step. Not that D's
compiler is not fast enough anyway, I'm just saying that your
statement is really weird.

>> Come on, that is not even a half decent analogy.
> it is. you can't see any uses of (semi)compiled module files (and i
> can; it's essential for component framework, for example). i can't see
> any uses of compiled binaries (i don't need that in component
> framework).

Actually I asked in this thread what the benefits are and the only one that come up was improved compilation speed due to caching of the lexing/parsing stage.

If you think it is a good idea for a component framework, would you please explain how? Honest question.

On Thu, 11 Dec 2014 09:44:49 +0000
John Colvin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> Parsing is so fast it's not worth spending huge numbers of man-hours building an effective cacheing system for it.
and generating machine code is useless at all, it's enough to simply improve CTFE.

> The rest of compilation is comparatively much slower and is therefore more important to cache.
what does "the rest of compilation" mean? there are alot of things you can do with AST before writing it to disk. ah, just writing compressed AST to disk is good enough, as reading it back is *way* *faster* than parsing the source. and any other tool -- like lint, or completion tool, or documentation generators can use that compressed AST without reparsing the sources.

you can't see how this can help 'cause we don't have such AST-companions yet. i can see how this will help 'cause i have alot of expirience with turbo/borland pascal and BlackBox Component Builder. think a-la BCB can be a killer app for D, but it's very hard to build it without good AST-companions.

On Thu, 11 Dec 2014 10:51:21 +0000
Tobias Pankrath via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> >> Come on, that is not even a half decent analogy.
> > it is. you can't see any uses of (semi)compiled module files
> > (and i
> > can; it's essential for component framework, for example). i
> > can't see
> > any uses of compiled binaries (i don't need that in component
> > framework).
> 
> Actually I asked in this thread what the benefits are and the only one that come up was improved compilation speed due to caching of the lexing/parsing stage.
> 
> If you think it is a good idea for a component framework, would you please explain how? Honest question.
the core of the component framework a-la BlackBox Component Builder is dynamic module system. this requires dynamic linker, and the linker must know alot about framework internals to be fast and usable. with precompiled modules which keeps symbolic information and ASTs for templates such linker can be written as independend module. you don't need to add hacks to runtime, to care about correct .so building and loading order and so on.

it's too long to explain in NG post. if you really interested you can take a look at BlackBox Component Builder itself, it's open-source. having ".sym" and ".cod" files are necessary to make such system usable.

D has a great foundation to build component framework a-la BCB. there are *no* competitors for D here, and having such system can boost D popularity to the skies. BCB failed due to two strategic errors: choosing Component Pascal as the system language (CP is great language, but the reality is that you cannot with with "pascal") and resisting to open-source the system until it was too late.

with "AST-companions" D is in position to occupy that niche. D is c-like, D has great metaprogramming abilities, D is open-source. it's doomed to win.

On Thursday, 11 December 2014 at 11:46:50 UTC, ketmar via Digitalmars-d wrote:
>
> you can't see how this can help 'cause we don't have such
> AST-companions yet. i can see how this will help 'cause i have alot of
> expirience with turbo/borland pascal and BlackBox Component Builder.
> think a-la BCB can be a killer app for D, but it's very hard to build
> it without good AST-companions.

And we're back to parsing speed. Starting DCD takes 1 (one) second on my pc and is parsing the hole of phobos and druntime at startup. And it only starts once a day. The code I'm constantly changing needs to be reparsed anyway. I never to wait for the tooltip though. So, using an preparsed ast yields me nothing today, but might brake all the tools that work with object files.

Giving the amount of manpower available, I'd say we have better things to do. But I guess no one would mind if you just make it happen.

On Thursday, 11 December 2014 at 11:46:50 UTC, ketmar via Digitalmars-d wrote:
> On Thu, 11 Dec 2014 09:44:49 +0000
> John Colvin via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
>> Parsing is so fast it's not worth spending huge numbers of man-hours building an effective cacheing system for it.
> and generating machine code is useless at all, it's enough to simply
> improve CTFE.
>
>> The rest of compilation is comparatively much slower and is therefore more important to cache.
> what does "the rest of compilation" mean? there are alot of things you
> can do with AST before writing it to disk. ah, just writing compressed
> AST to disk is good enough, as reading it back is *way* *faster* than
> parsing the source. and any other tool -- like lint, or completion
> tool, or documentation generators can use that compressed AST without
> reparsing the sources.
>
> you can't see how this can help 'cause we don't have such
> AST-companions yet. i can see how this will help 'cause i have alot of
> expirience with turbo/borland pascal and BlackBox Component Builder.
> think a-la BCB can be a killer app for D, but it's very hard to build
> it without good AST-companions.

BlackBox! A fellow user. :)

Another example, the Oberon operating system, specially the System 3 Gadgets framework.

--
Paulo

On Thursday, 11 December 2014 at 12:00:25 UTC, ketmar via Digitalmars-d wrote:
> On Thu, 11 Dec 2014 10:51:21 +0000
> Tobias Pankrath via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>
>> >> Come on, that is not even a half decent analogy.
>> > it is. you can't see any uses of (semi)compiled module files (and i
>> > can; it's essential for component framework, for example). i can't see
>> > any uses of compiled binaries (i don't need that in component
>> > framework).
>> 
>> Actually I asked in this thread what the benefits are and the only one that come up was improved compilation speed due to caching of the lexing/parsing stage.
>> 
>> If you think it is a good idea for a component framework, would you please explain how? Honest question.
> the core of the component framework a-la BlackBox Component Builder is
> dynamic module system. this requires dynamic linker, and the linker
> must know alot about framework internals to be fast and usable. with
> precompiled modules which keeps symbolic information and ASTs for
> templates such linker can be written as independend module. you don't
> need to add hacks to runtime, to care about correct .so building and
> loading order and so on.
>
> it's too long to explain in NG post. if you really interested you can
> take a look at BlackBox Component Builder itself, it's open-source.
> having ".sym" and ".cod" files are necessary to make such system
> usable.
>
> D has a great foundation to build component framework a-la BCB. there
> are *no* competitors for D here, and having such system can boost D
> popularity to the skies. BCB failed due to two strategic errors:
> choosing Component Pascal as the system language (CP is great language,
> but the reality is that you cannot with with "pascal") and resisting to
> open-source the system until it was too late.
>
> with "AST-companions" D is in position to occupy that niche. D is
> c-like, D has great metaprogramming abilities, D is open-source. it's
> doomed to win.

To be honest, with .NET Native and OpenJDK getting an AOT compiler around the corner (Java 9 or 10 not yet decided) this opportunity is already lost.

--
Paulo

> the core of the component framework a-la BlackBox Component Builder is
> dynamic module system. this requires dynamic linker, and the linker
> must know alot about framework internals to be fast and usable. with
> precompiled modules which keeps symbolic information and ASTs for
> templates such linker can be written as independend module.

You'll still need to compile everything that is a template, so you'd have
to provide a compiler as an independent module. That would be quite need, but
I don't see how this couldn't work with current object files. There is a REPL
using compilation to .so files and dynamical linking, after all.

If what you have in mind is indeed impossible with current object files, it may
be worthwhile to create our own. But as I see it, the only benefit of storing an AST is compilation speed, which currently is not dominated by parsing.

How would your precompiled modules differ from ELF except that they'd contain an AST for things that didn't emit the machine code yet?

On Thu, 11 Dec 2014 12:04:28 +0000
Paulo  Pinto via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> BlackBox! A fellow user. :)
yeah! i miss BCB almost every day i'm doing any coding.

> Another example, the Oberon operating system, specially the System 3 Gadgets framework.
yep, they have the same roots. i didn't mention Oberon as it's not "commercial programming environment", yet i still dreaming about Oberon as my primary OS... ;-)