On Fri, May 04, 2012 at 03:13:21PM -0400, Steven Schveighoffer wrote: [...]
> I think he meant that object (and library) binary files would be augmented by API segments that provide what di files provide now -- an interface-only version of the code.  It doesn't have to be text, it can be binary (maybe even partially compiled).
> 
> The really nice thing you get from this is, the compiler now would use this object file instead of .d files for importing.  So not only do you eliminate errors from having two possibly separately maintained files, but the compiler can build *extra* details into the .dobj file.  For example, it could put in metadata that would allow for full escape analysis.  Or tag that a function is implied pure (without actually having to tag the function with the pure attribute).
[...]

+1. It's about time we moved on from 30+ year old outdated linker technology, to something more powerful. Full escape analysis, compiler deduced function attributes like pureness, all the stuff that's impractical to implement in the current system, can all be done in a reasonable way if we stuck this information into the object files. The linker doesn't have to care what's in those extra sections; the compiler reads the info and does what it needs to do. The linker can omit the extra info from the final executable. (Or make use of it, if we implement a smarter linker. Like do cross-module string optimization, or something.)


T

-- 
Кто везде - тот нигде.

On Fri, 04 May 2012 14:12:16 -0700, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:

> On Fri, May 04, 2012 at 07:54:38PM +0200, Andrej Mitrovic wrote:
>> On 5/4/12, foobar <foo@bar.com> wrote:
>> > How about augmenting the object format so that libraries would be
>> > self contained and would not require additional .di files? Is
>> > this possible optlink by e.g. adding special sections that would
>> > be otherwise ignored?
>>
>> How would you use a library you don't even have the interface to? I
>> mean if you can't even look at the API in your editor.. that'd be
>> insane.
>
> Exactly. And while we're at it, *really* strip unnecessary stuff from
> .di files, like function bodies, template bodies, etc.. That stuff is
> required by the compiler, not the user, so stick that in the object
> files and let the compiler deal with it. The .di file should be ONLY
> what's needed for the user to understand how to use the library.
>
>
> T

I've written code to do this, but apparently it breaks Phobos in the autotester. I can't get it to break Phobos on my local machine so I'm at a loss as how to fix it. Maybe you can help? The code is here: https://github.com/LightBender/dmd.git

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/

On Friday, 4 May 2012 at 21:11:22 UTC, H. S. Teoh wrote:
> Exactly. And while we're at it, *really* strip unnecessary stuff from
> .di files, like function bodies, template bodies, etc.. That stuff is
> required by the compiler, not the user, so stick that in the object
> files and let the compiler deal with it. The .di file should be ONLY
> what's needed for the user to understand how to use the library.
>
>
> T

You contradict yourself.
The purpose of di files *is* to provide the compiler the required info to use the binary object/library. If you want human readable docs we already have DDoc (and other 3rd party tools) for that.
If you don't like the default HTML output (I can't fathom why) you can easily define appropriate macros for other output types such as TeX (and PDF via external converter), text based, etc..

On Sat, May 05, 2012 at 12:07:16AM +0200, foobar wrote:
> On Friday, 4 May 2012 at 21:11:22 UTC, H. S. Teoh wrote:
> >Exactly. And while we're at it, *really* strip unnecessary stuff from .di files, like function bodies, template bodies, etc.. That stuff is required by the compiler, not the user, so stick that in the object files and let the compiler deal with it. The .di file should be ONLY what's needed for the user to understand how to use the library.
> >
> >
> >T
> 
> You contradict yourself.
> The purpose of di files *is* to provide the compiler the required
> info to use the binary object/library. If you want human readable
> docs we already have DDoc (and other 3rd party tools) for that.
> If you don't like the default HTML output (I can't fathom why) you
> can easily define appropriate macros for other output types such as
> TeX (and PDF via external converter), text based, etc..

HTML is a stupid format, and ddoc output is not very navigable, but that's beside the point. I prefer to be reading actual code to be 100% sure that ddoc isn't leaving out some stuff that I should know about. All it takes is for somebody to leave out a doc comment and a particular declaration becomes invisible. (For example, std.uni was next to useless before I discovered that it actually had functions that I needed, but they didn't show up in dlang.org 'cos somebody failed to write doc comments for them.) I've seen too many commercial projects to believe for a moment that documentation is ever up-to-date. It depends on the library authors to provide ddoc output formats in a sane, usable format. Whereas if the compiler had a standardized, uniform, understandable format in well-known code syntax, that's a lot more dependable.

It's often impossible to debug something if you don't get to see what the compiler sees. I suppose you could argue that leaving out function bodies and stuff amounts to the same thing, but at least the language's interface for a function is the function's signature. When you have a .di file, you're guaranteed that all public declarations are there, and you can see exactly what they are. Of course, IF ddoc can be guaranteed to produce exactly what's in a .di file, then I concede that it is sufficient this purpose.


T

-- 
Recently, our IT department hired a bug-fix engineer. He used to work for Volkswagen.

On Friday 04 May 2012 08:56 PM, Andrew Wiley wrote:
> On Fri, May 4, 2012 at 1:46 PM, foobar <foo@bar.com
> <mailto:foo@bar.com>> wrote:
>
>     On Friday, 4 May 2012 at 18:30:32 UTC, Andrej Mitrovic wrote:
>
>         On 5/4/12, foobar <foo@bar.com <mailto:foo@bar.com>> wrote:
>
>             The di files are mostly meant to be machine read (e.g. the
>             compiler) and this belongs as part of the library file in order
>             to provide ease of use and maintain the relationship between the
>             binary code and it's interface.
>
>             maintaining two sets of files that could easily get out of sync
>             and *not* using the docs is way more insane.
>
>
>         I'd say the docs are more likely to be out of sync than .di code. If
>         the .di code is really out of sync you'll likely even get linker
>         errors. And not everything ends up being documented.
>
>         And then what about existing tools like IDEs and editors. E.g.
>         autocomplete wouldn't work anymore.
>
>
>     I'd say you'd be wrong.
>     Both di and docs are auto-generated from the same source.
>     As I said docs are designed for human consumption. This includes all
>     sorts of features such as a table of contents, a symbol index, the
>     symbols should have links, the docs provide usage examples, etc, etc.
>     Docs can be put online thus ensuring they're always up-to-date.
>
>     Tools should either read the data from the lib file or retrieve it
>     from the web. Keeping separate local di files is simply insane.
>
>     And really, have you never heard of Java? How about Pascal?
>     Should I continue back in history to all the languages that
>     implemented this feature decades ago?
>     C/C++ is a huge PITA with their nonsense compilation model which we
>     shouldn't have copied verbatim in D.
>
>
> I like the idea, but what about templates? For them, you'd basically be
> stuffing source code into the object files (unless you came up with a
> way to store the AST, but that seems like the effort/benefit ratio
> wouldn't be worth it since we currently have no way to preserve an AST
> tree between compiler runs).
> Otherwise, I find this idea very compelling. I'm sure there are probably
> other issues, though.
>

Storing the AST would basically equal storing the source code except 'trivia' like white space and unneeded tokens. At that point, you may as well ship the source.

-- 
- Alex

On Friday 04 May 2012 11:17 PM, H. S. Teoh wrote:
> On Fri, May 04, 2012 at 03:13:21PM -0400, Steven Schveighoffer wrote:
> [...]
>> I think he meant that object (and library) binary files would be
>> augmented by API segments that provide what di files provide now --
>> an interface-only version of the code.  It doesn't have to be text,
>> it can be binary (maybe even partially compiled).
>>
>> The really nice thing you get from this is, the compiler now would
>> use this object file instead of .d files for importing.  So not only
>> do you eliminate errors from having two possibly separately
>> maintained files, but the compiler can build *extra* details into
>> the .dobj file.  For example, it could put in metadata that would
>> allow for full escape analysis.  Or tag that a function is implied
>> pure (without actually having to tag the function with the pure
>> attribute).
> [...]
>
> +1. It's about time we moved on from 30+ year old outdated linker
> technology, to something more powerful. Full escape analysis, compiler
> deduced function attributes like pureness, all the stuff that's
> impractical to implement in the current system, can all be done in a
> reasonable way if we stuck this information into the object files. The
> linker doesn't have to care what's in those extra sections; the compiler
> reads the info and does what it needs to do. The linker can omit the
> extra info from the final executable. (Or make use of it, if we
> implement a smarter linker. Like do cross-module string optimization, or
> something.)
>
>
> T
>

Purity inference won't happen either way. Purity is part of your API and also meant to help you reason about your code. If the compiler just infers purity in a function and you later change the implementation so it's no longer pure, you break your users' code. Also, purity would no longer help you reason about your code if it's not explicit.

-- 
- Alex

Am 04.05.2012 20:26, schrieb Steven Schveighoffer:
> On Fri, 04 May 2012 13:54:38 -0400, Andrej Mitrovic
> <andrej.mitrovich@gmail.com>  wrote:
>
>>  On 5/4/12, foobar<foo@bar.com>  wrote:
>>>  How about augmenting the object format so that libraries would be
>>>  self contained and would not require additional .di files? Is
>>>  this possible optlink by e.g. adding special sections that would
>>>  be otherwise ignored?
>>
>>  How would you use a library you don't even have the interface to? I
>>  mean if you can't even look at the API in your editor.. that'd be
>>  insane.
>
> Ever heard of Java?
>
> -Steve

ever heard about Turbo Pascal (and delphi) got this feature since turbo pascal 4 around 1987

and turbo pascal and delphi are extremely fast native compilers without any Java, .Net magic

Am 05.05.2012 09:06, schrieb dennis luehring:
> Am 04.05.2012 20:26, schrieb Steven Schveighoffer:
>>  On Fri, 04 May 2012 13:54:38 -0400, Andrej Mitrovic
>>  <andrej.mitrovich@gmail.com>   wrote:
>>
>>>   On 5/4/12, foobar<foo@bar.com>   wrote:
>>>>   How about augmenting the object format so that libraries would be
>>>>   self contained and would not require additional .di files? Is
>>>>   this possible optlink by e.g. adding special sections that would
>>>>   be otherwise ignored?
>>>
>>>   How would you use a library you don't even have the interface to? I
>>>   mean if you can't even look at the API in your editor.. that'd be
>>>   insane.
>>
>>  Ever heard of Java?
>>
>>  -Steve
>
> ever heard about Turbo Pascal (and delphi) got this feature since turbo
> pascal 4 around 1987
>
> and turbo pascal and delphi are extremely fast native compilers without
> any Java, .Net magic

an more up-to-date example can be seen using the freepascal compiler and
its ppdump tool: http://www.freepascal.org/tools/ppudump.var

and turbo pascal gots even since 1987 a very good package system like a Java Jar file - you can just integrate compiled pascal sources (.pas -> .tpu) into something called .tpl file (turbo pascal library)

the freepascal compiler got something similar called .ppl

these "technologies" are damn good and invented so long before - but sometimes totaly unknown to all the obj-file-linker-guys

On 2012-05-05 00:39, H. S. Teoh wrote:

> It's often impossible to debug something if you don't get to see what
> the compiler sees. I suppose you could argue that leaving out function
> bodies and stuff amounts to the same thing, but at least the language's
> interface for a function is the function's signature. When you have a
> .di file, you're guaranteed that all public declarations are there, and
> you can see exactly what they are. Of course, IF ddoc can be guaranteed
> to produce exactly what's in a .di file, then I concede that it is
> sufficient this purpose.

If the compiler can extract the .di files from an object file so can other tools. I don't see the problem.

-- 
/Jacob Carlborg

On Friday, 4 May 2012 at 22:38:27 UTC, H. S. Teoh wrote:
> On Sat, May 05, 2012 at 12:07:16AM +0200, foobar wrote:
>> On Friday, 4 May 2012 at 21:11:22 UTC, H. S. Teoh wrote:
>> >Exactly. And while we're at it, *really* strip unnecessary stuff from
>> >.di files, like function bodies, template bodies, etc.. That stuff is
>> >required by the compiler, not the user, so stick that in the object
>> >files and let the compiler deal with it. The .di file should be ONLY
>> >what's needed for the user to understand how to use the library.
>> >
>> >
>> >T
>> 
>> You contradict yourself.
>> The purpose of di files *is* to provide the compiler the required
>> info to use the binary object/library. If you want human readable
>> docs we already have DDoc (and other 3rd party tools) for that.
>> If you don't like the default HTML output (I can't fathom why) you
>> can easily define appropriate macros for other output types such as
>> TeX (and PDF via external converter), text based, etc..
>
> HTML is a stupid format, and ddoc output is not very navigable, but
> that's beside the point. I prefer to be reading actual code to be 100%
> sure that ddoc isn't leaving out some stuff that I should know about.
> All it takes is for somebody to leave out a doc comment and a particular
> declaration becomes invisible. (For example, std.uni was next to useless
> before I discovered that it actually had functions that I needed, but
> they didn't show up in dlang.org 'cos somebody failed to write doc
> comments for them.) I've seen too many commercial projects to believe
> for a moment that documentation is ever up-to-date. It depends on the
> library authors to provide ddoc output formats in a sane, usable format.
> Whereas if the compiler had a standardized, uniform, understandable
> format in well-known code syntax, that's a lot more dependable.
>
> It's often impossible to debug something if you don't get to see what
> the compiler sees. I suppose you could argue that leaving out function
> bodies and stuff amounts to the same thing, but at least the language's
> interface for a function is the function's signature. When you have a
> .di file, you're guaranteed that all public declarations are there, and
> you can see exactly what they are. Of course, IF ddoc can be guaranteed
> to produce exactly what's in a .di file, then I concede that it is
> sufficient this purpose.
>
>
> T

This all amounts to the issues you have with the current implementation of DDoc which I agree needs more work.
The solution then is to fix/enhance DDoc.
Doxygen for instance has a setting to output all declarations whether documented or not, thus addressing your main point.

The projects you speak of I assume are written in C/C++? Those tend to have poor documentation precisely because people assume the header files are enough.
C/C++ requires you to install a 3rd party doc tool and learn that tool's doc syntax - effort that people are too lazy to invest.

In the Java world the syntax is standardized, the tool comes bundled with the compiler, all tools speak it and IDEs will even insert empty doc comment for you automatically. Frankly it takes effort to *not* document your code in this setting.
D provides DDoc precisely because it strives to provide the same doc friendly setting as Java.