A chat in IRC revealed a couple of possible improvements to the development scenario in which the interface file (.di) is managed manually and separately from the implementation file (.d).

After discussing a few ideas with Walter, the following language improvement came about. Consider a very simple scenario in which we have files a.d, a.di, and client.d, all situated in the same directory, with the following contents:

// a.di
class A { private int x; int foo(); }

// a.d
import a;
int A.foo() { return x + 1; }

// client.d
import a;
void main() { (new A).foo(); }

To compile:

dmd -c a.d
dmd -c client.d
dmd client.o a.o

Currently, in order for a program with separately-implemented methods to work properly, there must be TWO different files for the same class, and the .di file and the .d file MUST specify the same exact field layout. Ironically, the .d file is forbidden from including the .di file, which makes any checks and balances impossible. Any change in the layout (e.g. swapping, inserting, or removing fields) is undetectable and has undefined behavior.

I see this as a source of problems going forward, and I propose the following changes to the language:

1. The compiler shall accept method definitions outside a class.

2. A method cannot be implemented unless it was declared with the same signature in the class definition.

Walter is reluctantly on board with a change in this direction, with the note that he'd just recommend interfaces for this kind of separation. My stance in this matter is that we shouldn't constrain without necessity the ability of programmers to organize the physical design of their large projects.

Please discuss here. I should mention that Walter has his hands too full to embark on this, so implementation should be approached by one of our other dmd contributors (after of course there's a shared view on the design).


Andrei

On 7/22/11 5:06 PM, Andrei Alexandrescu wrote:
[snip]

I almost forgot - we should also have a means to allow a class to import its own incomplete declaration, as follows:

// a.di
class A { private int x; int foo(); }

// a.d
import a;
class A { private int x; int foo() { return x + 1; }

The compiler should accept this because that's what the .di generator outputs. The import will be used as a means to verify that the layout declared in the .di file is identical to the one in the .d file.

Without this feature, it would be impossible for the compiler to ensure that no inadvertent change was made to one of the files.


Andrei

On Fri, 22 Jul 2011, Andrei Alexandrescu wrote:

> On 7/22/11 5:06 PM, Andrei Alexandrescu wrote:
> [snip]
> 
> I almost forgot - we should also have a means to allow a class to import its own incomplete declaration, as follows:
> 
> // a.di
> class A { private int x; int foo(); }
> 
> // a.d
> import a;
> class A { private int x; int foo() { return x + 1; }

}

> 
> The compiler should accept this because that's what the .di generator outputs. The import will be used as a means to verify that the layout declared in the .di file is identical to the one in the .d file.
> 
> Without this feature, it would be impossible for the compiler to ensure that no inadvertent change was made to one of the files.
> 
> 
> Andrei

How about another case:

// a.di
class A { private int; int foo() { return x+1; } }

// a.d
imoprt a;
class A { private int; int foo() { return x+1; } }

ie, a case where the .di file has a function body (be it due to manual creation or automatic).  How closely do they need to match?

Later,
Brad

Andrei:

> Currently, in order for a program with separately-implemented methods to work properly, there must be TWO different files for the same class, and the .di file and the .d file MUST specify the same exact field layout. Ironically, the .d file is forbidden from including the .di file, which makes any checks and balances impossible. Any change in the layout (e.g. swapping, inserting, or removing fields) is undetectable and has undefined behavior.

Aren't .di files automatically generated? So isn't the job of tools like IDEs too keep .di files in sync with their .d file? .di files are better left as computer-generated things.

And if you really want to be sure, why you don't add something like this, that asks the compiler to verify they are in sync (most times a 128 bit hash of the .d module copied inside the .di file is enough to be sure the .d is in sync with its .di file:
pragma(di_sync);
There's also the possibility of a more fine grained hashing, usable to assert that the things you care for in a struct/class (like its layout) have not changed compared to the .di file.

Bye,
bearophile

On Sat, 23 Jul 2011 01:06:19 +0300, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> A chat in IRC revealed a couple of possible improvements to the development scenario in which the interface file (.di) is managed manually and separately from the implementation file (.d).

I might be forgetting bits of our discussions at the moment, but has it been considered that instead of facilitating use of manually-maintained .di files, to instead improve auto-generated .di files to the point that manual maintenance is not necessary? Some ideas:

1) As discussed, @ctfeable (include function body in .di to allow using in CTFE)
2) Removing imports for modules containing symbols not mentioned in .di code (avoid pulling in dependencies of the implementation)
3) Perhaps, adding opaque objects ("class X;") to the language and emitting those as appropriate? (semi-automatic Pimpls)

Did I miss anything?

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On 7/22/11 9:29 PM, Vladimir Panteleev wrote:
> On Sat, 23 Jul 2011 01:06:19 +0300, Andrei Alexandrescu
> <SeeWebsiteForEmail@erdani.org> wrote:
>
>> A chat in IRC revealed a couple of possible improvements to the
>> development scenario in which the interface file (.di) is managed
>> manually and separately from the implementation file (.d).
>
> I might be forgetting bits of our discussions at the moment, but has it
> been considered that instead of facilitating use of manually-maintained
> .di files, to instead improve auto-generated .di files to the point that
> manual maintenance is not necessary? Some ideas:
>
> 1) As discussed, @ctfeable (include function body in .di to allow using
> in CTFE)
> 2) Removing imports for modules containing symbols not mentioned in .di
> code (avoid pulling in dependencies of the implementation)
> 3) Perhaps, adding opaque objects ("class X;") to the language and
> emitting those as appropriate? (semi-automatic Pimpls)
>
> Did I miss anything?

I don't think it's an either-or situation. For a variety of reasons, some organizations want separate control of the "declaration" and "definition" files. Inability to do so is a common criticism leveled against Java and one of the reasons for the proliferation of XML configuration files and dynamic loading in that language.

Andrei

On Sat, 23 Jul 2011 05:52:12 +0300, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> I don't think it's an either-or situation. For a variety of reasons, some organizations want separate control of the "declaration" and "definition" files. Inability to do so is a common criticism leveled against Java and one of the reasons for the proliferation of XML configuration files and dynamic loading in that language.

Now I'm curious, what are those reasons? Can we improve .di generation to accommodate everyone, even if we'd need to add attributes or pragmas to the language or frontend?

It just seems to me like this path kills two birds with one stone, and is less work overall than doing both.

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On 7/23/11, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> Currently, in order for a program with separately-implemented methods to work properly, there must be TWO different files for the same class

Can we get a chat log of this discussion to figure out what is trying to be solved here?

interface IFoo
{
    int foo();
}

class Foo : IFoo
{
    private int x;

    version(one)
    {
        int foo() { return x; }
    }
    else
    version(two)
    {
        int foo() { return x++; }
    }
}

$ dmd -c foo.d -version=one

or:

class Foo : IFoo
{
    mixin(import("FooImpl.d"));
}

$ dmd -c foo.d -J./my/foo/

There are probably other ways to do this within the existing framework.

My vote goes to Walter and other contributors to work on fixing existing bugs so we can have not a perfect but a working language.

On Saturday 23 July 2011 07:34:43 Andrej Mitrovic wrote:
> On 7/23/11, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:
> > Currently, in order for a program with separately-implemented methods to work properly, there must be TWO different files for the same class
> 
> Can we get a chat log of this discussion to figure out what is trying to be solved here?
> 
> interface IFoo
> {
>     int foo();
> }
> 
> class Foo : IFoo
> {
>     private int x;
> 
>     version(one)
>     {
>         int foo() { return x; }
>     }
>     else
>     version(two)
>     {
>         int foo() { return x++; }
>     }
> }
> 
> $ dmd -c foo.d -version=one
> 
> or:
> 
> class Foo : IFoo
> {
>     mixin(import("FooImpl.d"));
> }
> 
> $ dmd -c foo.d -J./my/foo/
> 
> There are probably other ways to do this within the existing framework.
> 
> My vote goes to Walter and other contributors to work on fixing existing bugs so we can have not a perfect but a working language.

One of the key issues at hand is the fact that if you have a .di file and no actual implementation for a particular function, then you can't use that function in CTFE.

- Jonathan M Davis

Andrei:

> I don't think it's an either-or situation.

OK, but first it's better to work as much as possible on the automatic/compiler side of things, and see how much far it goes. Turning .di files into hand-written things has to be left as very last chance.


> For a variety of reasons, some organizations want separate control of the "declaration" and "definition" files.

Example: to not give full source code to clients. Even Python (that today is far more used in large and medium organizations than D) has received similar pressures. But each of those pressures have disadvantages, sometimes for the community of the programmers. So Python has chosen to resist to some of those pressures.


> Inability to do so is a common criticism leveled against Java and one of the reasons for the proliferation of XML configuration files and dynamic loading in that language.

There is JSON too today :-) Dynamic loading is a source of many troubles, it seems hard to make it safe.

Bye,
bearophile