John Reimer wrote:
> Ivan Senji wrote:
> 
>> bobef wrote:
>>
>>> Walter Bright wrote:
>>>
>>>> This incorporates a new 'header' generator capability, written by Dave
>>>> Fladebo, now working!
>>>>
>>>> http://www.digitalmars.com/d/changelog.html
>>>>
>>>>
>>>
>>>
>>> I play OpenTTD all night and in the morning see what happens :)
>>
>>
>> Wow! It is one of the *best* games I ever played. It is great to see there are more fans.
> 
> 
> Woah!  I didn't realize that I needed the original game graphics in order to make this work.  So much for that.
> 

You can find patched original version of the game, it is practically the same as OpenTTD and has original graphics. But you have to be warned that some people really don't like this type of games. I spent countless hours playing it, but some people just don't get it. :)

"Walter Bright" <newshound@digitalmars.com> wrote in message news:dp1s4k$12ev$2@digitaldaemon.com...
>
> "Derek Parnell" <derek@psych.ward> wrote in message news:1uxk73kt2jabq.1w7j11urctxbb.dlg@40tude.net...
>> On Thu, 29 Dec 2005 13:38:48 -0800, Walter Bright wrote:
>>> The main advantage of a D interface file is compile performance.
>> This is the *only* advantage
>
> No, it does wind up hiding the implementation details of most functions.
>
>> and that's not such a problem anyhow.
>
> Build speed isn't a problem now, but it will become one once D projects exceed a certain size.
>
>> It seems then that I won't be needing the -H switch anytime soon.
>
> It's certainly a forward looking capability.
>
>> Excuse me for sounding impertinent and selfish, but are you spending your time on D wisely? We get something of little value and still not receive things that are higher on the priority list.

Maybe it isn't a big step, at least at this stage in D's lifetime, but if so it may have cost me a bunch of time, not Walter (AFAIK). He integrated (and improved) this over the Holiday season and I reckon he deserves a break once in a while <g>

It looks like there may be a few issues right now, but once -H is stable, build the phobos and DWT .di's and then compare and perhaps you'll see more value: 1) lots less code for the compiler to cut through and 2) the inlined functions really don't give all that much away, IMO.

During my testing, for phobos the code reduction was ~50% and for DWT ~80%. Not to mention that skipping the semantic analysis for large function bodies should save a good chunk of compiler time.

Once I saw how DWT was put together, one of the things I thought this would be useful for would be things like RAD IDE's. RAD IDE's should be a lot easier to build if you can import a couple of 'master headers' that then import the rest. If you can reduce the avg. 'header' by 80% for a GUI library (like DWT), the edit-compile-debug cycles should be much shorter, plus you simplify the IDE with respect to maintaining imports and the like.

>
> It is regularly asked for. The specific impetus in this case was that Dave Fladebo went ahead and implemented it.
>

The -H stuff was built using dmdfe(*) and then I wrote Walter regarding what it did and how it was tested to see if it made sense to include in the compiler before posting it as a seperate tool. It was intentionally built to be "easy" to plug-in to the dmd front end, but I also found out that it pretty much needs a full-blown compiler to do things right.

The original dmdfe implementation was tested by producing 'headers' for phobos, DWT and several other test cases covering all of the examples in the documentation.

I experimented with the "implementation hiding" details previously mentioned in other posts, but that was abandoned for the reasons also previously mentioned in other posts (see: http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29883).

* dmdfe: http://home.comcast.net/~benhinkle/dmdfe/

>> How can I help? I know I can /donate/ library code and documentation, but the core language is still where the pre-1.0 issues lie. And even when I have given you library fixes, they are still to appear. I guess I don't really understand your organization's development processes.
>
> It's mainly driven by 1) what people feel like contributing and 2) things
> that are blocking use of D.
>

"bobef" <bobef@lessequal.com> wrote in message news:dp3132$1r9i$1@digitaldaemon.com...
> So I just wanted to support Walter and say that this is a very useful feature for me. Thank you, Walter (with much help from Dave Fladebo ;).

You're welcome!

> P.S. I didn't undestand how you control if the compiler is importing .d or .di . I looked in the docs but I didn't find any info on that. I mean if they have the same name?

It looks at each directory in the import path, first for a .di file, then a .d file.

"Derek Parnell" <derek@psych.ward> wrote in message news:op.s2le7jga6b8z09@ginger.vic.bigpond.net.au...
> On Fri, 30 Dec 2005 21:10:52 +1100, bobef <bobef@lessequal.com> wrote:
>> P.S. I didn't undestand how you control if the compiler is importing .d or .di . I looked in the docs but I didn't find any info on that. I mean if they have the same name?
>
> If you code "import xyz" the compiler looks first for xyz.d, and if it can't find that it tries to find xyz.di and if that's not found it errors.

No, first it looks for .di, then .d.

"J C Calvarese" <technocrat7@gmail.com> wrote in message news:dp3vqv$2jkt$1@digitaldaemon.com...
> But unfortunately, when these eager D citizens tried out the new compiler
> function they realized something was terribly wrong. The compiler wouldn't
> hide
> things that one would expect it to hide. "Hey, Walter, I think it's
> broken,"
> they say.
>
> "No," says Walter. "That's how it's supposed to work. Unless it crashes --
> I'll
> fix the things that make it crash."

It winds up putting the same things in the 'header' that you have to put in in C++, for the same reasons.

"Kris" <fu@bar.com> wrote in message news:dp3sve$2hm8$1@digitaldaemon.com...
> Just to clarify a couple of minor things:
>
> As John noted, implementation hiding is not about encryption ~ it's about supporting closed-source development.
>
> D /does/ support implementation hiding, and has done for a long time. What's been missing is a nice, simple, automated, standard, and compiler-supported way to generate a bridge between the "hidden" implementation, and client usage thereof. It's always been done manually, and it's easy to get confused over which file is which since there's common module names involved etc.
>
> One way to approach information hiding is to use Interfaces in conjunction with a Factory pattern. There's been a couple of trivial examples posted in this thread, but the general idea is that Interfaces are used to abstract away all implementation details, and one or more public functions are exposed in a "bridging module" to create instances on a clients behalf.
>
> Another approach to implementation hiding is to simply strip out existing method bodies and leave the bare declarations, along with any enums and/or typedefs they may use. Many requests have been made for an automated tool to do exactly this: which would also work well for the Interface approach noted above. The end result would be the same in both cases ~ a secondary

This is basically what -H does, except that inlinable function bodies are emitted. As pointed out earlier, private members often need to be there too, with or without function bodies (offsets, const initializers, templates), otherwise it gets pretty complicated, fast.

I experimented with most/all the implementation hiding stuff and got most of it working, I think, but in the end after testing it would have taken a whole heck of a lot more work (and time maintaining it) to make and keep the thing bulletproof (see: http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29883).

Why don't you give -H a try by using it against phobos, DWT and Mango (once the integration issues are taken care of)? Then see if it still really makes the implementation details all that apparent (or at least more apparent than C/++ library header files do now)?

- Dave

> "declaration only" file would be produced ~ which would be used by a client as a means to get the appropriate code linked. In all cases, the client is expected to know nothing of, nor care about, the implementation under the hood.
>
> This is where a misunderstanding arose with -H which, as it turns out, has no real value with respect to the above. Thus, D does not have a tool to help produce a client-binding from an implementation ~ you /still/ have to do that manually. That doesn't mean it can't be done, but it can certainly be a chore, and it would be good to get some standardization in the mechanics of it (such as a standard file-extension, for example, such that the conflict over module names can be better managed).
>
> So ~
>
> 1) D still needs a much more solid *story* for closed-source development. Without it, one could argue it deserves to remain a niche language. Note that this is not a technical hurdle, but it's one that has repeatedly failed to be passed. All it needs is a little TLC and attention.
>
> 2) The -H flag is apparently for something else entirely.
>
>
> Hope that helps?
>
> - Kris
>
>
>
> "John Reimer" <terminal.node@gmail.com> wrote
>> Kyle Furlong wrote:
>>> John Reimer wrote:
>>
>>>>
>>>> Okay, now I see why there is so much confusion going on with the addition of this feature.  There are two problems to be solved: one is the issue of implementation hiding; the other is the issue of compiler performance.  I think most of us mistook the di file generator to be a solution for implementation hiding for closed source projects. Obviously we got the wrong idea about the whole thing.
>>>>
>>>> If di files serve the same purpose as "precompiled headers," obviously there is no consideration whatsoever for implementation hiding (and there never was any such intention).
>>>>
>>>> Good to know.  Personally, I was less interested in compile performance and more interested in implementation hiding.  The -H flag, therefore, will not be of much use to me.
>>>>
>>>> -JJR
>>>
>>> Right on John. Walter, what is your answer for closed source implementation hiding?
>>
>> I think Walter already answered this question in a previous post in this thread.  If I understand correctly, he basically detailed how implementation hiding is nigh to impossible to do effectively.  I'm not sure if he meant that it's not worth trying to program a solution because data can't be effectively hidden or that the difficultly is not worth the effort.
>>
>> I just want to clarify that my idea of implementation-hiding really didn't entail encrypted libraries.  I know somebody could find out implementation details if they really wanted to.  The idea, much like the Modula 2 and 3 interface definition files, is merely to separate the details: (1) so that it isn't /easy/ to see implementation details and (2) so that we have a summary of the features a library provides.  A nifty side effect is that there's probably increase compiler performance.
>>
>> In the end though, maybe something like DDL (www.dsource.org) will solve all our problems in some way.  Eliminating the need for the compiler to parse a header altogether may be a big step forward.
>>
>> Also, while I feel that the -H option seems to be more of distraction from fixing higher priority items in D, it may do well for us at some point; I'm just hoping, though, that all the bugs it introduces don't take up Walter's time.
>>
>> A tool like this could theoretically be separate from the compiler. That may be the best solution in the end if it's too much work to maintain. I vaguely remember arguments to this end being made months to a year or so before.
>>
>> -JJR
>
>

Why dose the order of the function in a class define the order of there entries in the vtbl? This seems to me like an unnecessary complication that could create some vary difficult to find bugs.

Example:
Say an imported file gets changed but never gets recompiled (maybe it's a header file to a closed library)

1) compile main so that it imports new a.
2) compile old a
3) link

This won't have any errors but will end up calling the wrong fn at run time.
----------------
module a;	// old a

class C
{
	int fn1() {return 2;}
	int fn2() {return 3;}
}
----------------
module a;	// new a

class C
{
	int fn2() {return 3;}	// note: order reversed
	int fn1() {return 2;}
}
----------------
module main;
import a;
import std.stdio;

void main()
{
	auto c = new C;
	writef("hello world",\n);
	int i = c.fn1();
	writef("hello world ", i,\n);
}
----------------


I know next to nothing about COFF/ELF/etc. so this suggestion may be impossible to implement but here goes.

How about only calculate the offsets for the vtbl once, when you compile the class they are in. Then in the generated object file you store some symbols that are the offsets. When another module imports the file, the code is generated to make the virtual function call but the offset is left as an unresolved external that must be resolved at link time. Furthermore, this would solve many of the problems about code hiding in much the same way as James Dunne's suggestion



James Dunne wrote:
[...]
> 
> For the closed source community, one could also invent a specially formatted D import source file which declares only the non-private methods of classes *and* at which offsets they can be found at in the vtable.  Of course include the public fields and at which offsets they are found at as well.

> Maybe it isn't a big step, at least at this stage in D's lifetime, but if so it may have cost me a bunch of time, not Walter (AFAIK). He integrated (and improved) this over the Holiday season and I reckon he deserves a break once in a while <g>
> 
> It looks like there may be a few issues right now, but once -H is stable, build the phobos and DWT .di's and then compare and perhaps you'll see more value: 1) lots less code for the compiler to cut through and 2) the inlined functions really don't give all that much away, IMO.
> 
> During my testing, for phobos the code reduction was ~50% and for DWT ~80%. Not to mention that skipping the semantic analysis for large function bodies should save a good chunk of compiler time.
> 
> Once I saw how DWT was put together, one of the things I thought this would be useful for would be things like RAD IDE's. RAD IDE's should be a lot easier to build if you can import a couple of 'master headers' that then import the rest. If you can reduce the avg. 'header' by 80% for a GUI library (like DWT), the edit-compile-debug cycles should be much shorter, plus you simplify the IDE with respect to maintaining imports and the like.
> 
>> It is regularly asked for. The specific impetus in this case was that Dave
>> Fladebo went ahead and implemented it.
>>
> 
> The -H stuff was built using dmdfe(*) and then I wrote Walter regarding what it did and how it was tested to see if it made sense to include in the
> compiler before posting it as a seperate tool. It was intentionally built to be "easy" to plug-in to the dmd front end, but I also found out that it
> pretty much needs a full-blown compiler to do things right.
> 
> The original dmdfe implementation was tested by producing 'headers' for phobos, DWT and several other test cases covering all of the examples in the documentation.
> 
> I experimented with the "implementation hiding" details previously mentioned in other posts, but that was abandoned for the reasons also previously
> mentioned in other posts (see: http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29883).
> 
> * dmdfe: http://home.comcast.net/~benhinkle/dmdfe/


Walter is not always the best at "selling" these things to us.  You've done a lot better job here. :)

Still, I wonder how these headers look for DWT, which I'm pretty sure has lots of opportunities for inlining and private data.

-JJR

By the way, this bug is still around.

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/5671

Perhaps there's further misunderstanding here:

"Dave" <Dave_member@pathlink.com> wrote in
>
> "Kris" <fu@bar.com> wrote ...
>> Another approach to implementation hiding is to simply strip out existing
>> method bodies and leave the bare declarations, along with any enums
>> and/or
>> typedefs they may use. Many requests have been made for an automated tool
>> to do exactly this: which would also work well for the Interface approach
>> noted above. The end result would be the same in both cases ~ a secondary
>
> This is basically what -H does, except that inlinable function bodies are emitted. As pointed out earlier, private members often need to be there too, with or without function bodies (offsets, const initializers, templates), otherwise it gets pretty complicated, fast.


And because of that, it really has no value for implementation hiding. I'd go as far as to say it simply complicates things instead. Do imports get exposed also? I'd try this myself, but the compiler GPFs. Further, exposing public attributes in this manner seems to defy logic, and appears to breaks every visibility rule in the book. Such declarations were made private because they were intended to be, /ahem/, private?


> I experimented with most/all the implementation hiding stuff and got most of it working, I think, but in the end after testing it would have taken a whole heck of a lot more work (and time maintaining it) to make and keep the thing bulletproof (see: http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29883).
>
> Why don't you give -H a try by using it against phobos, DWT and Mango (once the integration issues are taken care of)? Then see if it still really makes the implementation details all that apparent (or at least more apparent than C/++ library header files do now)?


Forgive me Dave, but I just can't see how that's relevant. What you describe is not what, for example, any Fortune-500 company would consider usable. Walter apparently agrees. In reference to software contract-exposure: ".di files aren't necessarilly going to help you here."

Then there's the question of what this feature thinks it addresses. Walter seems to feel it's about compiler performance: "di files' main advantage is improving build speeds for large projects", and this is borne out in the flurry of posts. You also intimated this in a recent post. Yet you're also saying it can be used for implementation-hiding? Pardon me, but I have to disagree on all points:

<rant>

1) -H has little or no benefit for implementation hiding. It may look as though it could suffice, but I'll be happy to bet the current incarnation is a questionable toy in this regard. If I could actually get it to work, then I'd be happy to prove the point. Perhaps surprisingly, Walter appears to agree.

2) -H certainly ought to cut down on import size. So, just how much does that affect the compiler performance? It's already blazingly fast. I'll be a monkey's uncle if -H makes true /real-world/ differences to more than 5% of users. On the other hand, a /binary/ symbolic representation (as discussed several times) could make a really notable dent for certain environments, if such speedups were to become a necessity. If -H is all about performance, then why isn't it based around such a representation?

3) You speculatively mentioned DWT and RAD IDEs. There's a few things to consider there: (a) if the API is stable, the headers can be produced by a standalone tool ~ it doesn't need to be part of the compiler; (b) it the headers in question are being revised, they need to be regenerated on the fly; and (c) a RAD tool would likely prefer to have a library made available that hides all the details of creating and managing such headers, and would likely prefer that happened at the binary-symbolic level, explicitly for reasons of performance. Especially in case (a). Partial source-code munging does little to help such a tool ~ it's just not going far enough.

I've no intent on causing discomfort to you or anyone here, but I really have to say this feature appears to have been given not /nearly/ sufficient thought. It fails to support industrial-level implementation-hiding (the one place it is needed) and it doesn't push the performance envelope very hard at all.

You can call me an ass if you like, Dave, but this is surely feature-creep of the worst kind ~ there's apparently no meat. Can you, or anyone, please help me grasp where the real value lies? It's not that I'm somehow against new functionality. Part of what bugs me about this is the reference compiler sets an unwritten "standard" that others must somehow comply with. Whatever those standards are, they surely ought to be well considered.

</rant>

I can point to one side-effect that actually does help: the compiler now looks for "di" files, which means that hand-coded "implementation bridging" files can (at last!) live side-by-side with the real implementation module, eliminating some prior grief for a developer. However, this conflicts with -H rather badly, and therefore is likely to be an unintended and somewhat precarious benefit. Was that aspect considered?

Respectfully;

- Kris