On Saturday 16 July 2011 05:01:53 Sean Kelly wrote:
> On Jul 13, 2011, at 4:01 PM, Jonathan M Davis wrote:
> > On 2011-07-13 15:41, Walter Bright wrote:
> >> On 7/13/2011 3:19 PM, Jonathan M Davis wrote:
> >>> Given its heavy use of templates, I'm not sure how much that'll buy us in terms of compilation speed, and since it's all open source, I don't know why it would matter about hiding implementation details.
> >> 
> >> Consider the garbage collector. Should every compilation have to recompile that, too?
> > 
> > Since it doesn't take very long to compile, I really don't care much personally, and I think that it's just simpler to only deal with .d files. However, I'm not against the use of .di files if it speeds up compilation and doesn't have any major drawbacks. My primary concern at the moment (and why I brought it up in the first place) is that it seems to be affecting CTFE. For instance, this program
> > 
> > ====
> > import std.stdio;
> > import std.string;
> > 
> > enum a = strip(" hello world");
> > 
> > void main() {}
> > ====
> > 
> > currently fails to compile, giving these errors:
> > 
> > ====
> > /home/jmdavis/dmd2/linux/bin/../../src/phobos/std/string.d(1468): Error:
> > _aApplycd2 cannot be interpreted at compile time, because it has no
> > available source code
> 
> This function is in src/rt, which isn't publicly visible code (much like the GC).  Is there a similar example involving a module in core.*?

There may be, but I'd have to play around to find one. I only know about the one example, because it was brought up on the newsgroup. But regardless of whether there's another example, the fact that that one doesn't work is definitely a problem.

- Jonathan M Davis

On Thursday 21 July 2011 06:54:17 Don Clugston wrote:
> On 21 July 2011 01:55, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> > On 2011-07-20 16:06, Don Clugston wrote:
> >> On 13 July 2011 23:58, Walter Bright <walter at digitalmars.com> wrote:
> >> > On 7/13/2011 2:21 PM, Jonathan M Davis wrote:
> >> >> On 2011-07-13 13:36, Walter Bright wrote:
> >> >>> .di files are used when they are generated from .d files.
> >> >> 
> >> >> Yes. I get that. But why are .di files generated in the first place?
> >> > 
> >> > To 1. hide implementation details and 2. speed up compilation.
> >> > 
> >> >> Phobos just uses .d files.
> >> > 
> >> > It'll eventually migrate to .di files.
> >> 
> >> That will indeed prevent the use of CTFE.
> >> I'm not sure how to get around that. The basic problem is that CTFE
> >> requires a semantically-analyzed function, in order to be able to run
> >> it. If the source is provided, then you've entirely lost the benefits
> >> of .di files. If it's not provided, you can't do CTFE.
> >> 
> >> There are a couple of intermediate options:
> >> Then there's the option of adding the ability to run external
> >> functions from CTFE. A huge amount of work to implement (among other
> >> things, you need to dynamically link to a .obj file, and you have
> >> problems with initialization and gc use).
> >> A deeper problem is cross-compilation: the target CPU may not the same
> >> as the one the compiler is running on. (In fact, that's already the
> >> case with the 64-bit compiler).
> >> Really, you would need to specifically compile a separate version of
> >> each function, for use by CTFE.
> >> If you provide the function in an intermediate form, you add a host of
> >> complications (reminds me a bit of 'export' in C++, maybe not as bad).
> >> 
> >> Sounds like a D3 feature to me.
> > 
> > That may be, but many consider CTFE to be a big enough feature that disallowing it in the standard library (which would happen if we switched it all to .di files) would be unacceptable. So, while there may be a better solution which we could implement later, the question of how to handle it now remains.
> 
> I mean shipping Phobos only as .di files sounds like a D3 feature for this reason.

Ah. Okay. Though that raises the question as to whether druntime should really be being shipped only as .di files. Large portions of it would be fine that way (particularly the stuff that's just a bunch of C bindings), but some of it would likely not.

- Jonathan M Davis

On Jul 22, 2011, at 2:53 AM, Jonathan M Davis wrote:

> On Saturday 16 July 2011 05:01:53 Sean Kelly wrote:
>> On Jul 13, 2011, at 4:01 PM, Jonathan M Davis wrote:
>>> On 2011-07-13 15:41, Walter Bright wrote:
>>>> On 7/13/2011 3:19 PM, Jonathan M Davis wrote:
>>>>> Given its heavy use of templates, I'm not sure how much that'll buy us in terms of compilation speed, and since it's all open source, I don't know why it would matter about hiding implementation details.
>>>> 
>>>> Consider the garbage collector. Should every compilation have to recompile that, too?
>>> 
>>> Since it doesn't take very long to compile, I really don't care much personally, and I think that it's just simpler to only deal with .d files. However, I'm not against the use of .di files if it speeds up compilation and doesn't have any major drawbacks. My primary concern at the moment (and why I brought it up in the first place) is that it seems to be affecting CTFE. For instance, this program
>>> 
>>> ====
>>> import std.stdio;
>>> import std.string;
>>> 
>>> enum a = strip(" hello world");
>>> 
>>> void main() {}
>>> ====
>>> 
>>> currently fails to compile, giving these errors:
>>> 
>>> ====
>>> /home/jmdavis/dmd2/linux/bin/../../src/phobos/std/string.d(1468): Error:
>>> _aApplycd2 cannot be interpreted at compile time, because it has no
>>> available source code
>> 
>> This function is in src/rt, which isn't publicly visible code (much like the GC).  Is there a similar example involving a module in core.*?
> 
> There may be, but I'd have to play around to find one. I only know about the one example, because it was brought up on the newsgroup. But regardless of whether there's another example, the fact that that one doesn't work is definitely a problem.

I disagree, but mostly because this is an issue of how the runtime is designed than whether .d or .di files are generated.  Even if druntime were shipped with all .d files the issue would remain because the rt/* modules are imported by user code.  This is actually kind of an interesting issue because the fix would be a fundamentally different approach to the runtime design, and one that may be a much better design (because the resulting code could be optimized much more heavily) but may also turn out to be a bear to deal with.

In essence, the fix would be to have every function that is called by compiler-generated code be exposed by object.di.  I suspect that this would expose a good part of the now hidden compiler runtime code to the user, which not every compiler writer may like doing.  Which raises an interesting point.  Should CTFE support be consistent across compilers?  I'd be inclined to say so, but this appears to have interesting implications in terms of how code generation occurs regarding language features (array operations in particular).

On Sunday 24 July 2011 22:02:08 Sean Kelly wrote:
> On Jul 22, 2011, at 2:53 AM, Jonathan M Davis wrote:
> > On Saturday 16 July 2011 05:01:53 Sean Kelly wrote:
> >> On Jul 13, 2011, at 4:01 PM, Jonathan M Davis wrote:
> >>> On 2011-07-13 15:41, Walter Bright wrote:
> >>>> On 7/13/2011 3:19 PM, Jonathan M Davis wrote:
> >>>>> Given its heavy use of templates, I'm not sure how much that'll
> >>>>> buy
> >>>>> us in terms of compilation speed, and since it's all open
> >>>>> source, I
> >>>>> don't know why it would matter about hiding implementation
> >>>>> details.
> >>>> 
> >>>> Consider the garbage collector. Should every compilation have to recompile that, too?
> >>> 
> >>> Since it doesn't take very long to compile, I really don't care much
> >>> personally, and I think that it's just simpler to only deal with .d
> >>> files. However, I'm not against the use of .di files if it speeds up
> >>> compilation and doesn't have any major drawbacks. My primary concern
> >>> at
> >>> the moment (and why I brought it up in the first place) is that it
> >>> seems to be affecting CTFE. For instance, this program
> >>> 
> >>> ====
> >>> import std.stdio;
> >>> import std.string;
> >>> 
> >>> enum a = strip(" hello world");
> >>> 
> >>> void main() {}
> >>> ====
> >>> 
> >>> currently fails to compile, giving these errors:
> >>> 
> >>> ==== /home/jmdavis/dmd2/linux/bin/../../src/phobos/std/string.d(1468): Error: _aApplycd2 cannot be interpreted at compile time, because it has no available source code
> >> 
> >> This function is in src/rt, which isn't publicly visible code (much like the GC).  Is there a similar example involving a module in core.*?
> > 
> > There may be, but I'd have to play around to find one. I only know about the one example, because it was brought up on the newsgroup. But regardless of whether there's another example, the fact that that one doesn't work is definitely a problem.
> 
> I disagree, but mostly because this is an issue of how the runtime is designed than whether .d or .di files are generated.  Even if druntime were shipped with all .d files the issue would remain because the rt/* modules are imported by user code.  This is actually kind of an interesting issue because the fix would be a fundamentally different approach to the runtime design, and one that may be a much better design (because the resulting code could be optimized much more heavily) but may also turn out to be a bear to deal with.
> 
> In essence, the fix would be to have every function that is called by compiler-generated code be exposed by object.di.  I suspect that this would expose a good part of the now hidden compiler runtime code to the user, which not every compiler writer may like doing.  Which raises an interesting point.  Should CTFE support be consistent across compilers? I'd be inclined to say so, but this appears to have interesting implications in terms of how code generation occurs regarding language features (array operations in particular).

>From the user's perspective, the fact that you can't iterate over a string
with CTFE is a big problem ( http://d.puremagic.com/issues/show_bug.cgi?id=3512 ) regardless of what the reasons for it might be. It's hugely limiting.

Personally, I don't understand all of the issues involved, so I have no idea what the correct fix is. But most programmers are going to want it to "just work," and right now it doesn't. It may be that the problem is big enough that it's going to take a while to come up with an appropriate fix, but I don't think that it's really acceptable to disallow iteration over strings in CTFE unless it just really can't be done in any sane way. It's just too limiting (especially when you consider stuff like string mixins).

And yes, I think that CTFE should be consistent across compilers. It's going to be a problem if it isn't. There seem to be a number of people who rely on CTFE quite a bit on their code, and it's supposed to be one of D's selling points, so it needs to be appropriately portable. That could be problematic depending on what all of the implementation issues are, but it's also going to be problematic if CTFE doesn't work the same way across compilers.

So, obviously, there are some issues which need to be sorted out here, and that may take time, but the situation with regards to CTFE and strings definitely needs to be improved.

- Jonathan M Davis

On 25 July 2011 07:48, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> On Sunday 24 July 2011 22:02:08 Sean Kelly wrote:
>> In essence, the fix would be to have every function that is called by compiler-generated code be exposed by object.di. ?I suspect that this would expose a good part of the now hidden compiler runtime code to the user, which not every compiler writer may like doing. ?Which raises an interesting point. ?Should CTFE support be consistent across compilers? I'd be inclined to say so, but this appears to have interesting implications in terms of how code generation occurs regarding language features (array operations in particular).
>
> From the user's perspective, the fact that you can't iterate over a string with CTFE is a big problem ( http://d.puremagic.com/issues/show_bug.cgi?id=3512 ) regardless of what the reasons for it might be. It's hugely limiting.

It's a trivial issue. Don't worry about it. It'll be fixed in the next release.
The issue Sean raises about array operations is interesting. I think
DMD currently does array operations far too early; they probably
shouldn't be converted into function calls until all semantic passes
have completed.

On Monday 25 July 2011 09:00:45 Don Clugston wrote:
> On 25 July 2011 07:48, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> > On Sunday 24 July 2011 22:02:08 Sean Kelly wrote:
> >> In essence, the fix would be to have every function that is called by compiler-generated code be exposed by object.di.  I suspect that this would expose a good part of the now hidden compiler runtime code to the user, which not every compiler writer may like doing.  Which raises an interesting point.  Should CTFE support be consistent across compilers? I'd be inclined to say so, but this appears to have interesting implications in terms of how code generation occurs regarding language features (array operations in particular).
> > 
> > From the user's perspective, the fact that you can't iterate over a string with CTFE is a big problem ( http://d.puremagic.com/issues/show_bug.cgi?id=3512 ) regardless of what the reasons for it might be. It's hugely limiting.
> 
> It's a trivial issue. Don't worry about it. It'll be fixed in the next release. The issue Sean raises about array operations is interesting. I think DMD currently does array operations far too early; they probably shouldn't be converted into function calls until all semantic passes have completed.

That's good to hear. As I said, I don't really understand all of the issues here, since I'm not familiar enough with the compiler and druntime and this level, but it _is_ an issue that needs to be addressed. If it's an easy fix, all the better.

- Jonathan M Davis

On Monday 25 July 2011 00:10:14 Jonathan M Davis wrote:
> On Monday 25 July 2011 09:00:45 Don Clugston wrote:
> > On 25 July 2011 07:48, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> > > On Sunday 24 July 2011 22:02:08 Sean Kelly wrote:
> > >> In essence, the fix would be to have every function that is called
> > >> by
> > >> compiler-generated code be exposed by object.di.  I suspect that
> > >> this
> > >> would expose a good part of the now hidden compiler runtime code
> > >> to
> > >> the user, which not every compiler writer may like doing.  Which
> > >> raises an interesting point.  Should CTFE support be consistent
> > >> across compilers? I'd be inclined to say so, but this appears to
> > >> have
> > >> interesting implications in terms of how code generation occurs
> > >> regarding language features (array operations in particular).
> > > 
> > > From the user's perspective, the fact that you can't iterate over a
> > > string with CTFE is a big problem (
> > > http://d.puremagic.com/issues/show_bug.cgi?id=3512 ) regardless of
> > > what
> > > the reasons for it might be. It's hugely limiting.
> > 
> > It's a trivial issue. Don't worry about it. It'll be fixed in the next release. The issue Sean raises about array operations is interesting. I think DMD currently does array operations far too early; they probably shouldn't be converted into function calls until all semantic passes have completed.
> 
> That's good to hear. As I said, I don't really understand all of the issues here, since I'm not familiar enough with the compiler and druntime and this level, but it _is_ an issue that needs to be addressed. If it's an easy fix, all the better.

By the way, is the CTFE problem in http://d.puremagic.com/issues/show_bug.cgi?id=6374 likely to be fixed by the next release as well? I believe that those 2 issues are the two biggest blockers for using CTFE with string functions (though it sounds like there is a workaround that we could implement in std.array.appender if we have to).

- Jonathan M Davis

On 25 July 2011 09:16, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> By the way, is the CTFE problem in http://d.puremagic.com/issues/show_bug.cgi?id=6374 likely to be fixed by the next release as well? I believe that those 2 issues are the two biggest blockers for using CTFE with string functions (though it sounds like there is a workaround that we could implement in std.array.appender if we have to).

Definitely. And all the similar CTFE pointer bugs. They all involve special cases which I missed, rather than being difficult or fundamental issues.

On Monday 25 July 2011 09:49:12 Don Clugston wrote:
> On 25 July 2011 09:16, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> > By the way, is the CTFE problem in http://d.puremagic.com/issues/show_bug.cgi?id=6374 likely to be fixed by the next release as well? I believe that those 2 issues are the two biggest blockers for using CTFE with string functions (though it sounds like there is a workaround that we could implement in std.array.appender if we have to).
> 
> Definitely. And all the similar CTFE pointer bugs. They all involve special cases which I missed, rather than being difficult or fundamental issues.

Good to here!

- Jonathan M Davis

On Monday 25 July 2011 00:51:04 Jonathan M Davis wrote:
> On Monday 25 July 2011 09:49:12 Don Clugston wrote:
> > On 25 July 2011 09:16, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> > > By the way, is the CTFE problem in
> > > http://d.puremagic.com/issues/show_bug.cgi?id=6374 likely to be
> > > fixed by the next release as well? I believe that those 2 issues
> > > are the two biggest blockers for using CTFE with string functions
> > > (though it sounds like there is a workaround that we could
> > > implement in
> > > std.array.appender if we have to).
> > 
> > Definitely. And all the similar CTFE pointer bugs. They all involve special cases which I missed, rather than being difficult or fundamental issues.
> 
> Good to here!

to _hear_ that is. I really should be getting to bed... :)

- Jonathan M Davis