On Fri, 04 May 2012 14:48:04 -0400, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> On 5/4/12, Steven Schveighoffer <schveiguy@yahoo.com> wrote:
>> What I'm saying is, it's completely possible to store the API in binary
>> format *in* the object files, and use documentation generators to document
>
> Yes but then you need to *modify* existing tools in order to add a new
> feature that extracts information from object files. Either that, or
> you'd have to somehow extract the .di files back from the object
> files. How else can you see the interface in your text editor without
> the source files? :)

Current tools:  read .di files and extract API
new tools: read .dobj files and extract API.

I'm not really seeing the difficulty here...

-Steve

On 5/4/12, Steven Schveighoffer <schveiguy@yahoo.com> wrote:
> Current tools:  read .di files and extract API
> new tools: read .dobj files and extract API.
>
> I'm not really seeing the difficulty here...

I thought he meant libraries that are only distributed in binary form. So no .di files anywhere. Maybe I misunderstood..

On Fri, 04 May 2012 15:07:43 -0400, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> On 5/4/12, Steven Schveighoffer <schveiguy@yahoo.com> wrote:
>> Current tools:  read .di files and extract API
>> new tools: read .dobj files and extract API.
>>
>> I'm not really seeing the difficulty here...
>
> I thought he meant libraries that are only distributed in binary form.
> So no .di files anywhere. Maybe I misunderstood..

No reason for .di files if the object file already serves as the interface file.

I think he meant that object (and library) binary files would be augmented by API segments that provide what di files provide now -- an interface-only version of the code.  It doesn't have to be text, it can be binary (maybe even partially compiled).

The really nice thing you get from this is, the compiler now would use this object file instead of .d files for importing.  So not only do you eliminate errors from having two possibly separately maintained files, but the compiler can build *extra* details into the .dobj file.  For example, it could put in metadata that would allow for full escape analysis.  Or tag that a function is implied pure (without actually having to tag the function with the pure attribute).

-Steve

On Fri, May 4, 2012 at 1:46 PM, foobar <foo@bar.com> wrote:

> On Friday, 4 May 2012 at 18:30:32 UTC, Andrej Mitrovic wrote:
>
>> On 5/4/12, foobar <foo@bar.com> wrote:
>>
>>> The di files are mostly meant to be machine read (e.g. the compiler) and this belongs as part of the library file in order to provide ease of use and maintain the relationship between the binary code and it's interface.
>>>
>>> maintaining two sets of files that could easily get out of sync and *not* using the docs is way more insane.
>>>
>>>
>> I'd say the docs are more likely to be out of sync than .di code. If the .di code is really out of sync you'll likely even get linker errors. And not everything ends up being documented.
>>
>> And then what about existing tools like IDEs and editors. E.g. autocomplete wouldn't work anymore.
>>
>
> I'd say you'd be wrong.
> Both di and docs are auto-generated from the same source.
> As I said docs are designed for human consumption. This includes all sorts
> of features such as a table of contents, a symbol index, the symbols should
> have links, the docs provide usage examples, etc, etc.
> Docs can be put online thus ensuring they're always up-to-date.
>
> Tools should either read the data from the lib file or retrieve it from the web. Keeping separate local di files is simply insane.
>
> And really, have you never heard of Java? How about Pascal?
> Should I continue back in history to all the languages that implemented
> this feature decades ago?
> C/C++ is a huge PITA with their nonsense compilation model which we
> shouldn't have copied verbatim in D.
>

I like the idea, but what about templates? For them, you'd basically be
stuffing source code into the object files (unless you came up with a way
to store the AST, but that seems like the effort/benefit ratio wouldn't be
worth it since we currently have no way to preserve an AST tree between
compiler runs).
Otherwise, I find this idea very compelling. I'm sure there are probably
other issues, though.

On 2012-05-04 18:57, foobar wrote:

> How about augmenting the object format so that libraries would be self
> contained and would not require additional .di files? Is this possible
> optlink by e.g. adding special sections that would be otherwise ignored?

That would be nice. I guess that would mean that compiler needs to be changed as well to be able to read the .di files from the library.

-- 
/Jacob Carlborg

On 2012-05-04 20:30, Andrej Mitrovic wrote:

> I'd say the docs are more likely to be out of sync than .di code. If
> the .di code is really out of sync you'll likely even get linker
> errors. And not everything ends up being documented.

Then you need to manage your docs better.

> And then what about existing tools like IDEs and editors. E.g.
> autocomplete wouldn't work anymore.

They would need to be able to read the library and extract the .di files. Isn't this basically just how Java works?

-- 
/Jacob Carlborg

On Friday, 4 May 2012 at 19:13:21 UTC, Steven Schveighoffer wrote:
> On Fri, 04 May 2012 15:07:43 -0400, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:
>
>> On 5/4/12, Steven Schveighoffer <schveiguy@yahoo.com> wrote:
>>> Current tools:  read .di files and extract API
>>> new tools: read .dobj files and extract API.
>>>
>>> I'm not really seeing the difficulty here...
>>
>> I thought he meant libraries that are only distributed in binary form.
>> So no .di files anywhere. Maybe I misunderstood..
>
> No reason for .di files if the object file already serves as the interface file.
>
> I think he meant that object (and library) binary files would be augmented by API segments that provide what di files provide now -- an interface-only version of the code.  It doesn't have to be text, it can be binary (maybe even partially compiled).
>
> The really nice thing you get from this is, the compiler now would use this object file instead of .d files for importing.  So not only do you eliminate errors from having two possibly separately maintained files, but the compiler can build *extra* details into the .dobj file.  For example, it could put in metadata that would allow for full escape analysis.  Or tag that a function is implied pure (without actually having to tag the function with the pure attribute).
>
> -Steve

Exactly :)

On Friday, 4 May 2012 at 19:21:02 UTC, Andrew Wiley wrote:
> On Fri, May 4, 2012 at 1:46 PM, foobar <foo@bar.com> wrote:
>
>> On Friday, 4 May 2012 at 18:30:32 UTC, Andrej Mitrovic wrote:
>>
>>> On 5/4/12, foobar <foo@bar.com> wrote:
>>>
>>>> The di files are mostly meant to be machine read (e.g. the
>>>> compiler) and this belongs as part of the library file in order
>>>> to provide ease of use and maintain the relationship between the
>>>> binary code and it's interface.
>>>>
>>>> maintaining two sets of files that could easily get out of sync
>>>> and *not* using the docs is way more insane.
>>>>
>>>>
>>> I'd say the docs are more likely to be out of sync than .di code. If
>>> the .di code is really out of sync you'll likely even get linker
>>> errors. And not everything ends up being documented.
>>>
>>> And then what about existing tools like IDEs and editors. E.g.
>>> autocomplete wouldn't work anymore.
>>>
>>
>> I'd say you'd be wrong.
>> Both di and docs are auto-generated from the same source.
>> As I said docs are designed for human consumption. This includes all sorts
>> of features such as a table of contents, a symbol index, the symbols should
>> have links, the docs provide usage examples, etc, etc.
>> Docs can be put online thus ensuring they're always up-to-date.
>>
>> Tools should either read the data from the lib file or retrieve it from
>> the web. Keeping separate local di files is simply insane.
>>
>> And really, have you never heard of Java? How about Pascal?
>> Should I continue back in history to all the languages that implemented
>> this feature decades ago?
>> C/C++ is a huge PITA with their nonsense compilation model which we
>> shouldn't have copied verbatim in D.
>>
>
> I like the idea, but what about templates? For them, you'd basically be
> stuffing source code into the object files (unless you came up with a way
> to store the AST, but that seems like the effort/benefit ratio wouldn't be
> worth it since we currently have no way to preserve an AST tree between
> compiler runs).
> Otherwise, I find this idea very compelling. I'm sure there are probably
> other issues, though.

C++ has pre-compiled header files (.pch) which speedup compilation time for projects with lots'o'templates. The same kind of info could be stored inside the object files, for example by serializing the AST as you said yourself.

There are many uses for this kind of technology. we can store additional info that currently isn't available for all sorts of link-time optimizations.

On Fri, 04 May 2012 14:56:56 -0400, Andrew Wiley <wiley.andrew.j@gmail.com> wrote:

> I like the idea, but what about templates? For them, you'd basically be
> stuffing source code into the object files

Nothing wrong with this.  There is still a gain here -- object code compiled from the source containing the original source is tightly coupled with the template.  You can be sure that this object file will link against one that you build based on the template contained in it.

-Steve

On Fri, May 04, 2012 at 07:54:38PM +0200, Andrej Mitrovic wrote:
> On 5/4/12, foobar <foo@bar.com> wrote:
> > How about augmenting the object format so that libraries would be self contained and would not require additional .di files? Is this possible optlink by e.g. adding special sections that would be otherwise ignored?
> 
> How would you use a library you don't even have the interface to? I mean if you can't even look at the API in your editor.. that'd be insane.

Exactly. And while we're at it, *really* strip unnecessary stuff from .di files, like function bodies, template bodies, etc.. That stuff is required by the compiler, not the user, so stick that in the object files and let the compiler deal with it. The .di file should be ONLY what's needed for the user to understand how to use the library.


T

-- 
There are three kinds of people in the world: those who can count, and those who can't.