Why does druntime use .di files reather than .d files? What do we gain by doing that? Phobos uses .d files. I don't understand why we use .di files in druntime. If they don't give us a real benefit, I would think that it would be better to just use .d files. Of course, if we get a real benefit from them, then we should probably use them, but the .di files are pointless from what I can see.

Apparently, there have been CTFE-related bugs where CTFE doesn't work because the compiler doesn't have druntime's full source when compiling Phobos. I don't know if the use of .di files is really the whole issue here or if it's something that needs to be fixed with regards to CTFE, but if using .di files for druntime is really causing a problem for CTFE, that would seem to be a good reason to stop using .di files for druntime.

So, my question is why we're using .di files for druntime. From what I can tell, we gain nothing from it, and it's causing us problems.

- Jonathan M Davis

.di files are used when they are generated from .d files.

On 7/12/2011 11:49 PM, Jonathan M Davis wrote:
> So, my question is why we're using .di files for druntime. From what I can tell, we gain nothing from it, and it's causing us problems.
>

On 2011-07-13 13:36, Walter Bright wrote:
> .di files are used when they are generated from .d files.

Yes. I get that. But why are .di files generated in the first place? Phobos just uses .d files. It doesn't generate any .di files. druntime, on the other hand, generates .di files, and that's what you end up importing when you import core.*. I don't understand why the .di files are being used at all.

- Jonathan M Davis

On 7/13/2011 2:21 PM, Jonathan M Davis wrote:
> On 2011-07-13 13:36, Walter Bright wrote:
>> .di files are used when they are generated from .d files.
> Yes. I get that. But why are .di files generated in the first place?

To 1. hide implementation details and 2. speed up compilation.

>   Phobos just uses .d files.

It'll eventually migrate to .di files.

>   It doesn't generate any .di files. druntime, on the other
> hand, generates .di files, and that's what you end up importing when you
> import core.*. I don't understand why the .di files are being used at all.
>

On 2011-07-13 14:58, Walter Bright wrote:
> On 7/13/2011 2:21 PM, Jonathan M Davis wrote:
> > On 2011-07-13 13:36, Walter Bright wrote:
> >> .di files are used when they are generated from .d files.
> > 
> > Yes. I get that. But why are .di files generated in the first place?
> 
> To 1. hide implementation details and 2. speed up compilation.
> 
> > Phobos just uses .d files.
> 
> It'll eventually migrate to .di files.

Given its heavy use of templates, I'm not sure how much that'll buy us in terms of compilation speed, and since it's all open source, I don't know why it would matter about hiding implementation details. The documentation gives the API. You look at the source if you want more than that, in which case you probably want the .d file and not a .di file. But I guess that even incremental improvements are worth something as far as compilation speed goes, and there is a fair bit of functionality which isn't templated (though increasingly, I think that it's in the minority).

Regardless, if it's making it so that stuff isn't CTFEable (which at least seems to be the case with druntime at the moment - I'm not quite sure what's going on there, but people have been getting errors about it), then I would think that that would make using .di files a really bad idea in general, since you effectively risk killing a language feature - and in the standard library no less.

But anyway, thanks for the reasoning on why druntime uses .di files. I've always found it odd (and honestly, I still find it odd, but at least understandable now). I've never seen much point to .di files at all. But if they speed up compilation, then that's definitely of some value.

- Jonathan M Davis

On 7/13/2011 3:19 PM, Jonathan M Davis wrote:
>
> Given its heavy use of templates, I'm not sure how much that'll buy us in terms of compilation speed, and since it's all open source, I don't know why it would matter about hiding implementation details.

Consider the garbage collector. Should every compilation have to recompile that, too?

On 2011-07-13 15:41, Walter Bright wrote:
> On 7/13/2011 3:19 PM, Jonathan M Davis wrote:
> > Given its heavy use of templates, I'm not sure how much that'll buy us in terms of compilation speed, and since it's all open source, I don't know why it would matter about hiding implementation details.
> 
> Consider the garbage collector. Should every compilation have to recompile that, too?

Since it doesn't take very long to compile, I really don't care much personally, and I think that it's just simpler to only deal with .d files. However, I'm not against the use of .di files if it speeds up compilation and doesn't have any major drawbacks. My primary concern at the moment (and why I brought it up in the first place) is that it seems to be affecting CTFE. For instance, this program

====
import std.stdio;
import std.string;

enum a = strip(" hello world");

void main() {}
====

currently fails to compile, giving these errors:

====
/home/jmdavis/dmd2/linux/bin/../../src/phobos/std/string.d(1468): Error:
_aApplycd2 cannot be interpreted at compile time, because it has no available
source code
/home/jmdavis/dmd2/linux/bin/../../src/phobos/std/string.d(1521): Error:
cannot evaluate stripLeft(s) at compile time
/home/jmdavis/dmd2/linux/bin/../../src/phobos/std/string.d(1521): Error:
cannot evaluate stripRight(stripLeft(s)) at compile time
q.d(4): Error: cannot evaluate strip(" hello world") at compile time
====

It used to compile prior to 2.054. I don't know whether it's really a CTFE bug or if the lack of actual source is really the cause of the problem. But if the lack of source _is_ the problem, then that would mean that using .di files would tend to make CTFE not work, in which case, I'd argue that we shouldn't be using .di files. However, if it's really just a CTFE bug, and .di files don't cause any problems for CTFE, then it's fine with me if we use .di files. I doubt that I'll ever choose to use .di files in one of my projects, but stuff like compilation speed matters that much more when you're dealing with the standard library, particularly since it affects everyone using the language.

- Jonathan M Davis

Right.? I would think the most important reason to use .di files is to hide private imports.? The GC is a good example, you don't want to import the GC implementation when importing core.memory.? But in fact, core.memory doesn't import anything from rt, it links to the GC implementation via extern(C) functions, so the implementation of core.memory is not a good example.


I think to a point, using .di files has merit.? To be certain, anything that imports anything from rt should be hidden (I'm not sure if there are any cases of that).? But we should consider the cost of disabling CTFE on druntime functions, especially those in core that really aren't managing the actual runtime of the compiler, but are just tools used by both druntime and phobos (core.time and core.math come to mind).

-Steve




>________________________________
>From: Walter Bright <walter at digitalmars.com>
>To: d-runtime at puremagic.com
>Sent: Wednesday, July 13, 2011 6:41 PM
>Subject: Re: [D-runtime] Why does druntime us .di files instead of .d?
>
>
>
>On 7/13/2011 3:19 PM, Jonathan M Davis wrote:
>>
>> Given its heavy use of templates, I'm not sure how much that'll buy us in terms of compilation speed, and since it's all open source, I don't know why it would matter about hiding implementation details.
>
>Consider the garbage collector. Should every compilation have to recompile that, too?
>
>_______________________________________________
>D-runtime mailing list
>D-runtime at puremagic.com
>http://lists.puremagic.com/mailman/listinfo/d-runtime
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/d-runtime/attachments/20110714/911bdbaa/attachment.html>

On Thu, Jul 14, 2011 at 4:01 AM, Steve Schveighoffer <schveiguy at yahoo.com>wrote:

> Right.  I would think the most important reason to use .di files is to hide private imports.  The GC is a good example, you don't want to import the GC implementation when importing core.memory.  But in fact, core.memory doesn't import anything from rt, it links to the GC implementation via extern(C) functions, so the implementation of core.memory is not a good example.
>
> I think to a point, using .di files has merit.  To be certain, anything that imports anything from rt should be hidden (I'm not sure if there are any cases of that).  But we should consider the cost of disabling CTFE on druntime functions, especially those in core that really aren't managing the actual runtime of the compiler, but are just tools used by both druntime and phobos (core.time and core.math come to mind).
>
> -Steve
>

It seems to me that a way to fix all this and guarantee CTFE-ability (which
seems to be the end goal) would be to add some sort of modifier or
annotation to designate that a function is CTFE-able. This could be verified
by the compiler at build-time and used as a flag to emit full function
source when generating a di file. Things get more complicated because it
would have to be transitive, so it wouldn't be a simple addition, and might
be heavier than what we're looking for.
On the other hand, it would provide a strong guarantee about a D feature
that's somewhat nebulous right now.
Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/d-runtime/attachments/20110714/336c91b2/attachment.html>

On 2011-07-14 14:40, Andrew Wiley wrote:
> On Thu, Jul 14, 2011 at 4:01 AM, Steve Schveighoffer
<schveiguy at yahoo.com>wrote:
> > Right. I would think the most important reason to use .di files is to hide private imports. The GC is a good example, you don't want to import the GC implementation when importing core.memory. But in fact, core.memory doesn't import anything from rt, it links to the GC implementation via extern(C) functions, so the implementation of core.memory is not a good example.
> > 
> > I think to a point, using .di files has merit. To be certain, anything that imports anything from rt should be hidden (I'm not sure if there are any cases of that). But we should consider the cost of disabling CTFE on druntime functions, especially those in core that really aren't managing the actual runtime of the compiler, but are just tools used by both druntime and phobos (core.time and core.math come to mind).
> > 
> > -Steve
> 
> It seems to me that a way to fix all this and guarantee CTFE-ability (which
> seems to be the end goal) would be to add some sort of modifier or
> annotation to designate that a function is CTFE-able. This could be
> verified by the compiler at build-time and used as a flag to emit full
> function source when generating a di file. Things get more complicated
> because it would have to be transitive, so it wouldn't be a simple
> addition, and might be heavier than what we're looking for.
> On the other hand, it would provide a strong guarantee about a D feature
> that's somewhat nebulous right now.
> Thoughts?

It's not altogether a bad idea, but at least in theory, one of the main features of CTFE is that it's supposed to just work without treating a function specially. Some functions just can't be run at compile time due to their nature, but in theory, pretty much anything is supposed to be CTFEable. So, on some level, CTFE is a bit of a failure if a function requires an attribute to be CTFEable. On the other hand, having an attribute whose purpose was purely to make it so that the function has its full source put in a .di file (which would ensure that it was CTFEable) but which was otherwise ignored and completely unnecessary for CTFE wouldn't be as bad. It might even have some value beyond CTFE. For instance, wouldn't the lack of source in a .di file make a function un-inlinable? As such, there are a lot of functions which you would want the full source for in a .di file. In the inline case at least, I can see the compiler being smart enough to realize that it's probably going to want to inline a function, so it then puts the source for that function in the .di file. But CTFE functions can be arbitrarily complex, so that wouldn't work for them.

I don't know. In C/C++, at least it's totally up to the programmer to separate the header and source files as they deem appropriate. In D, the .di is likely to be generated, so while you could edit it by hand after generation, you then have problems keeping it in sync with the .d file (or you have to regenerate the .di file every time and patch it). And since the .d file contains everything in it while the .di file has a subset, it's not quite the same situation as you get with a header and source file in C/C++.

As long as you can generate a .di file from a .d file without losing any of the source that you actually need (for inlining or CTFE or whatever), then you're fine. But beyond that, it seems like you either give up those capabilities, or you have to hand edit the .di file every time that a change is made to the .d file which would affect the .di file. Neither option seems particularly pleasant to me.

If modules were organized such that those which could afford to not be inlined or CTFEed at all were .di files, and the rest were .d files, then that essentially solves the problem, but depending on the modules, that could negatively affect module organization. I suppose that you could effectively organize modules as either internal or external and make the internal ones have the .di files and the external ones just .d files. But that's not necessarily reasonable. Pretty soon, you're effectively trying to have .d files which are akin to C/C++ source files, and .d files which are akin to C/++ header files and have .di files. Overall, it just seems simpler to use .d files, but we don't necessarily want to lose the compilation gains from .di files either. So, I don't know what the best way to handle this is. But I'd be very concerned about using .di files if it negatively affects CTFE and inlinability. If it doesn't, then they're fine. But from the looks of it, it does.

- Jonathan M Davis