On Sunday, December 16, 2012 23:32:38 Andrej Mitrovic wrote:
> On 12/16/12, Paulo Pinto <pjmlp@progtools.org> wrote:
> > If modules are used correctly, a .di should be created with the public interface and everything else is already in binary format, thus the compiler is not really parsing everything all the time.
> 
> A lot of D code tends to be templated code, .di files don't help you in that case.

And .di files don't work with CTFE or inlining. In general, .di files are a horrible idea.

I tend to be of the opinion that they shouldn't even exist, but some corporate types require that sort of thing when distributing libraries to 3rd parties, so we need some sort of header solution. A better one probably would have been a binary format where the code is partially compiled with documentation providing a human-readable API, but that's something that we'll have to look into in the future. For now, we're stuck with .di files.

- Jonathan M Davis

On 12/16/2012 2:13 PM, Simen Kjaeraas wrote:
> What's a good solution to the issue, then? Leave it as is?

Figure out the cases where it happens and fix those cases.

On Sunday, 16 December 2012 at 21:34:53 UTC, Walter Bright wrote:
> On 12/16/2012 5:36 AM, Jacob Carlborg wrote:
>> Walter doesn't seem to want to implement a flag that output the template
>> instantiations to all object files.
>
> No "seem" about it!
>
> I think it is a bad solution to the issue.

Is no solution better than a bad solution ?

On 12/16/2012 3:30 PM, deadalnix wrote:
> On Sunday, 16 December 2012 at 21:34:53 UTC, Walter Bright wrote:
>> I think it is a bad solution to the issue.
>
> Is no solution better than a bad solution ?

Solutions based on not understanding the issue rarely turn out well.

On Monday, December 17, 2012 00:30:34 deadalnix wrote:
> On Sunday, 16 December 2012 at 21:34:53 UTC, Walter Bright wrote:
> > On 12/16/2012 5:36 AM, Jacob Carlborg wrote:
> >> Walter doesn't seem to want to implement a flag that output
> >> the template
> >> instantiations to all object files.
> > 
> > No "seem" about it!
> > 
> > I think it is a bad solution to the issue.
> 
> Is no solution better than a bad solution ?

Sometimes. It depends on the problem and how bad the solution is (especially if we risk being stuck with the bad solution). But in general, the thing to do is to figure out a good solution, not to simply implement the bad solution because it's the only thing that's been suggested. It wouldn't surprise me though if Walter knew a better solution and just hasn't gotten around to implementing it. That happens often enough given how busy he is and how much there is to do.

- Jonathan M Davis

On Sunday, 16 December 2012 at 23:51:16 UTC, Walter Bright wrote:
> On 12/16/2012 3:30 PM, deadalnix wrote:
>> On Sunday, 16 December 2012 at 21:34:53 UTC, Walter Bright wrote:
>>> I think it is a bad solution to the issue.
>>
>> Is no solution better than a bad solution ?
>
> Solutions based on not understanding the issue rarely turn out well.

Assuming that other don't understand the issue is not going to help. The issue is pretty clear and simple.

Emitting template symbols clearly slow down the compilation process and increase the size of object files. That is a given. But that allow for separate compilation.

Note that separate compilation is the only path that make sense considering that the GC do not work.

On 12/16/2012 4:04 PM, deadalnix wrote:
> Assuming that other don't understand the issue is not going to help. The issue
> is pretty clear and simple.

Why it happens in one circumstance and not another is not nearly so clear and simple.

On Sun, Dec 16, 2012 at 03:06:16PM -0800, Jonathan M Davis wrote:
> On Sunday, December 16, 2012 23:32:38 Andrej Mitrovic wrote:
> > On 12/16/12, Paulo Pinto <pjmlp@progtools.org> wrote:
> > > If modules are used correctly, a .di should be created with the public interface and everything else is already in binary format, thus the compiler is not really parsing everything all the time.
> > 
> > A lot of D code tends to be templated code, .di files don't help you in that case.
> 
> And .di files don't work with CTFE or inlining. In general, .di files are a horrible idea.

Yeah, that's another thing that C++ did wrong, that unfortunately we sorta inherited in a partial way.


> I tend to be of the opinion that they shouldn't even exist, but some corporate types require that sort of thing when distributing libraries to 3rd parties, so we need some sort of header solution. A better one probably would have been a binary format where the code is partially compiled with documentation providing a human-readable API, but that's something that we'll have to look into in the future. For now, we're stuck with .di files.
[...]

I've proposed this before: when compiling a library, the compiler should process everything into an intermediate form (perhaps even the IR that's used internally to integrate with the codegen), then save that as a sort of partially-compiled object format. This can either be in the form of an object file, or perhaps a custom D-specific format. Everything, including templates, function bodies, etc., is included.  Then when you import something from the library, the compiler simply loads the IR from the intermediate format and use that info to compile the code.

This eliminates the wasted time relexing and reparsing imported files every single time they're used, and also suitably "hides" the implementation details of the library in a format that isn't easily read. (I mean, let's face it, if someone is desperate enough, he can reverse engineer *anything*, so imagining that shipping a binary is somehow "safer" than shipping a straightforward IR is nonsense. But in the reasonable case, the library being in IR should be disincentive enough for people to not try to break encapsulation.)

This intermediate format can be in the form of some kind of hash table that the compiler can quickly look up referenced symbols from, so there will be almost no overhead.


T


-- 
They say that "guns don't kill people, people kill people." Well I think the gun helps. If you just stood there and yelled BANG, I don't think you'd kill too many people. -- Eddie Izzard, Dressed to Kill

On 12/16/2012 4:48 PM, H. S. Teoh wrote:
> I've proposed this before: when compiling a library, the compiler should
> process everything into an intermediate form (perhaps even the IR that's
> used internally to integrate with the codegen), then save that as a sort
> of partially-compiled object format. This can either be in the form of
> an object file, or perhaps a custom D-specific format. Everything,
> including templates, function bodies, etc., is included.  Then when you
> import something from the library, the compiler simply loads the IR from
> the intermediate format and use that info to compile the code.

This offers nothing over a .di file.


> This eliminates the wasted time relexing and reparsing imported files
> every single time they're used,

Frankly, it's not much faster than parsing the .di file.


> and also suitably "hides" the
> implementation details of the library in a format that isn't easily
> read. (I mean, let's face it, if someone is desperate enough, he can
> reverse engineer *anything*, so imagining that shipping a binary is
> somehow "safer" than shipping a straightforward IR is nonsense. But in
> the reasonable case, the library being in IR should be disincentive
> enough for people to not try to break encapsulation.)

It doesn't hide the source in any effective way. There are enough Java byte code => source translators around to prove that. It only takes one such tool to exist (and it's especially easy to create such a tool given D being open source).

An effective way to hide your implementation is to use PIMPL, which D supports well.


> This intermediate format can be in the form of some kind of hash table
> that the compiler can quickly look up referenced symbols from, so there
> will be almost no overhead.

I've done such (precompiled headers for C++), I've done .di files, and I've done Java bytecode. .di files are superior in nearly every way.

On Sunday, December 16, 2012 16:57:31 Walter Bright wrote:
> I've done such (precompiled headers for C++), I've done .di files, and I've done Java bytecode. .di files are superior in nearly every way.

Given that .di don't work with inlining or CTFE, I'd consider them to be a very poor solution. You're seriously impairing yourself if you use them. It's pretty much BS that corporations insist on header files to hide implementation, since it really doesn't work, but if we're going to be forced to a have a solution which tries to hide implementation to make folks like that happy, we could at least have one that doesn't cripple the language like .di files do. It may not truly hide the implementation any better than .di files do, but at least it would allow us to still use the language properly.

I'm not expecting this problem to be fixed any time soon (we have far higher priorites), but I really do think that in the long run .di files should be deprecated in favor of a binary solution which doesn't stop things like inlining or CTFE from working.

- Jonathan M Davis