"Kris" <fu@bar.com> wrote in message news:dp6onr$1m6u$1@digitaldaemon.com...
> Okay. I think there's an opportunity to clear this up via a few short questions. These refer back to the SearchEngine example posted previously (repeated below):
>
> a) The example provided, where the SearchFactory.di file was hand-crafted? Will usage of that "di" module operate correctly?
>
> b) You say below " it just isn't practical". Do you mean it is not technically feasible to mechanically extract the example SearchFactory.di from the given SearchFactory.d?

It isn't practical to determine that a declaration has *no* dependencies on any private class members or anything imported from an embedded import statement. It *is* practical to elide the bodies of functions that don't appear in templates, because there's no way for another declaration to look inside it, so -H does that. static this's are also elided, as nothing else can refer to them anyway.

Think of it in these terms, and I think it'll become clear. -H can elide something *only* if it can guarantee there is nothing that can look inside it.

Also, .di files have to be independent of changes in files the .di file imports. Thus, the import declarations have to remain, even if they are private imports.

It is technically feasible to analyse the source code to the point where one can prove it depends on no private data, no private imports, etc., but that is a huge project, and would likely not yield much as few source files are written this way. Hence, I said it was impractical.

"Sean Kelly" <sean@f4.ca> wrote in message news:dp6j8o$1ibk$1@digitaldaemon.com...
> There's been a lot of confusion about the real purpose of "export" in C++, much like the -H option here ;-)  All the feature really provides is a way to speed compilation--implementation hiding is not really possible (as you've pointed out).

I don't believe it speeds up compilation either. None of the EDG compilers have been willing to take up my challenge on DMC++ with precompiled headers vs EDG and exported templates <g>.

> Whether potential benefit of "export" is worth the development cost is an issue of much contention.

The people who argue it is worth it are either the ones for which it is a sunk cost or are not the ones who'll have to pay for it. Although I've pointed out that the ones who think they don't have to pay for it will actually have to pay for it, in terms that they won't get other features/support they probably need much more.

> The folks at EDG argued against the feature before it was made standard, but now they're arguing for it... possibly because they've spent so much time implementing it.

For EDG, it's a sunk cost. Their cost for the feature going forward is minimal, so once it is a sunk cost, the cost/benefit ratio suddenly becomes very favorable. It's in EDG's economic best interest at this point to strongly support export.

> I don't think Microsoft has any plans of implementing "export," though I haven't heard anything from other vendors.

I haven't heard anyone else planning to implement it, either.

"Walter Bright" <newshound@digitalmars.com> wrote in message news:dp7459$1stn$1@digitaldaemon.com...
>
> "Kris" <fu@bar.com> wrote in message news:dp6onr$1m6u$1@digitaldaemon.com...
>> Okay. I think there's an opportunity to clear this up via a few short questions. These refer back to the SearchEngine example posted previously (repeated below):
>>
>> a) The example provided, where the SearchFactory.di file was hand-crafted? Will usage of that "di" module operate correctly?
>>
>> b) You say below " it just isn't practical". Do you mean it is not technically feasible to mechanically extract the example SearchFactory.di from the given SearchFactory.d?
>
> It isn't practical to determine that a declaration has *no* dependencies on any private class members or anything imported from an embedded import statement. It *is* practical to elide the bodies of functions that don't appear in templates, because there's no way for another declaration to look inside it, so -H does that. static this's are also elided, as nothing else can refer to them anyway.


Okay. And what about question (a) above, please?

"John Reimer" <terminal.node@gmail.com> wrote in message news:dp4lhd$4m1$1@digitaldaemon.com...

>> During my testing, for phobos the code reduction was ~50% and for DWT ~80%.

>
> Still, I wonder how these headers look for DWT, which I'm pretty sure has lots of opportunities for inlining and private data.
>

I just re-checked my numbers and it looks to be >= 80% ;)

But, it is pretty hard to do really accurately with a quick grep -v | wc type of thing because of different whitespace delimiters, etc. To do that really well you'd almost need to add it to the compiler.. hmmm... <JUST KIDDING <g>>

What I've noticed is that there are fewer opportunities for inlining then you might think in DWT and in general for anything non-trivial... For example, any fbody with a loop in it is not a candidate, which generally is a real good thing, I think. OTOH, Phobos std.math[2] has just about everything in it inlined, which is also a good thing, since most of those are return statements basically codifying what the name of the function implies.

There was a great article a while back in Dr. Dobbs with an inlining algorithm that made a lot of sense, and I think the DMD frontend heuristics and the article coincide quite a bit (for the record, DMD had implemented the inlining long before the article came out ;). The reason I mention this is because I think compiler developers are starting to nail down what should be inlined pretty well, so the inlining is good to have in the reference compiler and reasonable to have in the -H tool too.

> -JJR

Walter Bright wrote:
> This incorporates a new 'header' generator capability, written by Dave
> Fladebo, now working!
> 
> http://www.digitalmars.com/d/changelog.html
> 
> 

Jeez, what a mess of a thread.

Before of all, as for this pre-parse header feature, since it is (primarily?) a performance feature, is it planned that in the future (after debugging) this option will be enabled by default?

Now, what is the appropriate way to create and use in D an external library, in terms of module importation?
Is it to use the whole library source and import it's modules (but not compile them of course)?, Or to hand-craft header modules?
But what if I want to use different versions of said library with the same aplication *executable* (say, like updating a buggy DLL)? This makes the first option (use full source) inviable, right? (because the the bugfix may be in a inlined function?)

Thus, if hand-crafting is the only solution, wouldn't it be useful to have a tool that would automatically generate this interface(the set of headers)?

However, what is the interface of a module and what is it's implemention? I think that it is in this question that many of the confusion of this thread originated.
For Walter, this interface is everything that might be need by the importing module, even on a binary level, so this includes privates and inlinable function bodies.
This, is the standard C++ point of view and pratice, as Walter pointed out and exemplified.

However, since in many cases the conceptual interface may be smaller than this standard interface, some people "sugested" that the generated interface(header) should be closer to this conceptual interface, particularly: it should not contain privates and function bodies.

However, for this to be done(implemented), it has to be *correct* and *clear* on how it would work. People said that privates shouldn't appear on the generated headers, but "private" and protection attributes in general are (again I say this) not made for code hiding, but just for protection. And so, when using them for information hiding (like removing private code from generated headers), it doesn't work, as Walter pointed out in many exemples (used internally, problems with the binary interface, etc.).
So if one wants to complain about not having this feature implemented, then it should first of all be well tought out and defined how the feature works. (and unfortunately that didn't happen)
That means extending and clarifying the semantics of private for such purpose. For instance when a public function has a private parameter. (I wonder what happens in Java or C# , but I'm not yet back home with my PC, so I can't try it out)

In article <dp4hdq$31jl$3@digitaldaemon.com>, Walter Bright says...
>
>
>"J C Calvarese" <technocrat7@gmail.com> wrote in message news:dp3vqv$2jkt$1@digitaldaemon.com...
>> But unfortunately, when these eager D citizens tried out the new compiler
>> function they realized something was terribly wrong. The compiler wouldn't
>> hide
>> things that one would expect it to hide. "Hey, Walter, I think it's
>> broken,"
>> they say.
>>
>> "No," says Walter. "That's how it's supposed to work. Unless it crashes --
>> I'll
>> fix the things that make it crash."
>
>It winds up putting the same things in the 'header' that you have to put in in C++, for the same reasons.

After reading some of the most recent posts, I think I understand better than I did before what some of these issues boil down to.

Apparently, implementation-hiding is _a lot_ more complicated than I realized. Part of why I thought that it wouldn't be that complicated is because dig's strip.d is a pretty short program. I didn't really test it, but I thought the dig's strip.d worked pretty good. Now, I'm pretty sure it could work only because it wasn't thoroughly tested with complicated enough code to make it choke.

jcc7

In article <dp50iu$bna$1@digitaldaemon.com>, Dave says...
>
>In article <dp4oq5$6qt$1@digitaldaemon.com>, Kris says...
>>
>>Perhaps there's further misunderstanding here:
>>
>>"Dave" <Dave_member@pathlink.com> wrote in
>>>

..

>>I've no intent on causing discomfort to you or anyone here, but I really have to say this feature appears to have been given not /nearly/ sufficient thought. It fails to support industrial-level implementation-hiding (the one place it is needed) and it doesn't push the performance envelope very hard at all.
>>
>
>No discomfort at all - Everyone has their opinions. This obviously doesn't have to be the be-all and end-all to the 'header', symbol file, what-have-you functionality, and I don't think it was ever intended as such, certainly not by me.
>
>At the very least it brought out the discussion -- included in my earlier post
>was a link to an older request for functionality and no-one replied, so I
>implemented what seemed to make the most sense and what I had time for, at >least as a *first step*.
>
>I would applaud anyone (Kris?, Derek?) who would take what's there and provide >a better solution -- plain-and-simple fact is I've spent all the time I can on >it.
>
>- Dave

Dave, thanks for taking all of the negative criticism so well. (I'm sure I wouldn't be able to maintain my calm in the face of so many surprisingly severe posts.)

I think the main issue might be that one or more people are jealous that you not only implemented something that you find useful, but also convinced Walter to add it to the compiler. I once posted a bug report (including the simple fix) that went unacknowleged for months. ;)

I don't see the real harm in the "-H" switch. Since I guess .di files are read first, I can see that people will need to make sure that their .di files stay in synch with their .d files, but that could be solved with a batch file that negates any possible speed-up benefits are the "-H" switch. It's the same issue that happens when I install lib's into \dmd\lib and need to remember to copy new libs to reap the benefits from the changes I've tried.

On the other hand, this new switch will give us a new way to analyze D code with. We could see what is a candidate for inlining and what isn't. That seems like it could be pretty useful to me.

I just hope that if people feel so strongly about the need to have implementation-hiding in D that they'll spend a little more time trying to hack up the front-end to reach their goals (and a little less time ranting about it in the newsgroup).

jcc7

In article <dp9bci$jcn$1@digitaldaemon.com>, J C Calvarese says...
>
>I think the main issue might be that one or more people are jealous that you not only implemented something that you find useful, but also convinced Walter to add it to the compiler. I once posted a bug report (including the simple fix) that went unacknowleged for months. ;)
>

I don't think I convinced Walter of anything here <g> More of "coinciding opinion" probably. I think he saw a need for something offered as a contribution already implemented and tested, so he could spend the time on other more important stuff.

Hey, it was enjoyable and instructive for me <g>

>On the other hand, this new switch will give us a new way to analyze D code with. We could see what is a candidate for inlining and what isn't. That seems like it could be pretty useful to me.
>

Hopefully it will eventually be seen as more useful than just inline feedback (although that part is kind of cool too) <g>

As mentioned elsewhere, it cuts the code for the compiler to process by 80% for a library like DWT. I agree with Walter that for large projects and such the value of this should become more apparent as D is used in those environments more.

Other people have mentioned binary representation, symbol files, etc... IMHO, the problem there is that you'll need a tool to read (and debug) those and/or the 'symbol' representation will be implemented differently between compilers. Unless, of course, you first specify an "intermediate language" which really wouldn't hide anything anyway. It's taken 5+ years to specify D already - how long for the IL ;)

At least this is legal and readable D code, is automatically generated, and really should not give away any more implementation details than comparable C/++ header files do now.

>I just hope that if people feel so strongly about the need to have implementation-hiding in D that they'll spend a little more time trying to hack up the front-end to reach their goals (and a little less time ranting about it in the newsgroup).
>

I'm not saying this to discourage anyone from the effort, but AFAIK most/all of the things mentioned in this thread, have been tried and abandoned for various reasons (see: http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29883). It was a circuitous route. <g>

It'd be nice if someone out there can come up with something better since obviously it is very important to some people and probably somewhat important to many others.

>jcc7

In article <dpa8f8$195s$1@digitaldaemon.com>, Dave says...

.snip...

>Hopefully it will eventually be seen as more useful than just inline feedback (although that part is kind of cool too) <g>

I wasn't try to put it down. I get that it's cool, but I think most of its benefits are somewhat over my head. ;)

..snip...

>>I just hope that if people feel so strongly about the need to have
>>implementation-hiding in D that they'll spend a little more time trying to >>hack up the front-end to reach their goals (and a little less time ranting >>about it in the newsgroup).
>
>I'm not saying this to discourage anyone from the effort, but AFAIK most/all of the things mentioned in this thread, have been tried and abandoned for various reasons (see: http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29883). It was a circuitous route. <g>

I've read that post a few times now, but I don't know if it's sunk in yet. It's not that don't believe you. The thing is I don't really need implementation hiding for my own purposes, so I'm not trying to get in the middle of this argument.

From reading the posts on this topics, I think that more implementation can be hidden. It's more a question of "Do the people that want it want it enough to get it done?" No, not everything should or could be hidden. Templates would have to be visible. If someone doesn't care about inlining, couldn't he drop a few function bodies that would be inlineable otherwise? And hiding the implementation wouldn't prevent reverse engineering (though reverse engineering might be more difficult). Or maybe I'm wrong about this, and anything else is infeasible. Sometimes, I just have to convince myself that something is impossible before I can admit what's obvious to someone else.

>It'd be nice if someone out there can come up with something better since
>obviously it is very important to some people and probably somewhat important >to many others.

Sure.

jcc7

Dave ~ a couple of comments:

"Dave" <Dave_member@pathlink.com> wrote
> Other people have mentioned binary representation, symbol files, etc...
> IMHO,
> the problem there is that you'll need a tool to read (and debug) those
> and/or
> the 'symbol' representation will be implemented differently between
> compilers.
> Unless, of course, you first specify an "intermediate language" which
> really
> wouldn't hide anything anyway. It's taken 5+ years to specify D already -
> how
> long for the IL ;)

I fully agree with your assertion on the time-scale of things. While I feel that this aspect of the compiler is premature-optimization, that's just my opinion ... not that it makes a blind bit of difference anyway. As noted before, and fwiw, I truly commend you for taking the initiative and actually /doing something/!

Just for the record, I recall it was Walter who said he would be doing a binary-symbolic representation.


> At least this is legal and readable D code, is automatically generated,
> and
> really should not give away any more implementation details than
> comparable C/++
> header files do now.

Walter has now noted that he feels it's "impractical" to mechanically perform the kind of implementation-hiding we'd been asking for, and thus we have to still do that manually. That's cool, but sometimes it's like pulling-teeth just trying to get some clarity around here. I'd like to refer back to the original posting on this subject-line (branch), since I think the perspective is relevant there? This branch is not about the "optimization" aspect, although it apparantly got mixed in.


> I'm not saying this to discourage anyone from the effort, but AFAIK
> most/all of
> the things mentioned in this thread, have been tried and abandoned for
> various
> reasons (see:
> http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29883).
> It was a circuitous route. <g>

I wish I'd seen that post on Nov 12th ~ would have immediately noted (regarding the last paragraph of the main thrust) that emitting source code for 'inlineable' would mean potentially exposing private import modules. That's not a "good thing"tm for the kind of implementation-hiding we'd been asking for (for a long time). It also seems from your post that you had more in mind than what's currently exposed.


> obviously it is very important to some people and probably somewhat
> important to
> many others.

It can be acutely important from a commercial standpoint. Because most people are simply tinkering with the language, you won't see much in the way of opinion there. Nor much from the open-source community.

We should all understand that there's a lot of inertia against change in IT. To get a foot into the commercial segment, D needs as much help as it can possibly get in order to avoid being marginalized ~ even just keeping the darned lawyers out of the way. Implementation-hiding is one such aspect that helps. I can tell you that D is /not/ being considered where I work. Hence, I have a beef with certain weak attributes of the language and tools.

I hope this clarifies certain aspects further; and thank you for being open-minded on the subject.