December 19, 2011
On Mon, 19 Dec 2011 03:53:27 -0800, so <so@so.so> wrote:

> Good work Adam!
>
> After class/struct/interface level "export" we shouldn't need it again.
> It already implies that everything (if not stated otherwise via private) will be exported.

Well, I tried it that way and it didn't work on Windows. Without the export in front of the function declaration the function won't get exported to Windows in a shared library situation. I have verified this behavior with Walter and he assures me that, under Windows, this is the correct behavior. On other OS'es public works just fine and D treats export as public, but I work on Windows so I have to use export. I am working on an article for the site documenting my experiences with dynamic libraries and the pitfalls you can run into, including this one. Expect to see the initial draft after I finish DI generation.

> Removing them also would automagicaly solve the inconsistency below.
>
>> export class List : IList
>> {
>>      export void RemoveAt(ulong Index);
>>      protected int ProtectedTest(int d);
>> }
>
> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:
>
>> As you may all be aware, I've been trying to improve the automated generation of .di files and I now have something that I feel is testable. Currently the new code only makes the following changes.
>>
>> 1.	Function Implementations are removed
>> 2.	Private Function Declarations are removed.
>> 3.	Variable Initializers, except for const, are removed.
>>
>> Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.
>>
>> drey_ from IRC was kind enough to test build Derelict with the changes and has discovered a potential issue around private imports. Derelict uses private imports that contain types which are used in function alias declarations. As one would expect, this caused many compiler errors.  Currently, I feel that private imports should be stripped from the DI file as they are intended to be internal to the module. However, I want to put it to the community to decide, and I would especially appreciate Mr. Bright's opinion on private imports in DI files.
>>
>> If anyone wishes to test my changes against their code, you can download them from my git account here: https://LightBender@github.com/LightBender/dmd.git
>> I only ask that you inform me of anything broken and provide a test case that reproduces the error.
>>
>> The following is my current test case, it comes from a module in my project that has been modified to include the various tests I and others have been able to think of. I know for a fact that it is missing a great number of features available in D, and I would really appreciate any tests you might wish to add to this case.
>>
>> module Horizon.Collections.Basic;
>>
>> export void TestFunc(int a) {}
>> private void TestFunc2(int b) {}
>>
>> const gss = "__gshared";
>>
>> export interface IEnumerator
>> {
>> 	export @property Object Current();
>>
>> 	export bool MoveNext();
>> 	export void Reset();
>> }
>>
>> export interface ICollection
>> {
>> 	export @property uint Count();
>>
>> 	export int opApply(int delegate(ref Object) dg);
>> 	export IEnumerator GetEnumerator();
>> }
>>
>> export interface IList : ICollection
>> {
>> 	export @property bool IsFixedSize();
>> 	export @property bool IsReadOnly();
>>
>> 	export void Add(Object Value);
>> 	export void Clear();
>> 	export bool Contains(Object Value);
>> 	export Object opIndex(ulong Index);
>> 	export void opIndexAssign(Object Value, ulong Index);
>> 	export int IndexOf(Object Value);
>> 	export void Insert(ulong Index, Object Value);
>> 	export void Remove(Object Value);
>> 	export void RemoveAt(ulong Index);
>> }
>>
>> export struct StructTesting
>> {
>> 	private int i = 0;
>> 	protected int j = 1;
>>
>> 	export void ExportedStructFunc() {}
>> 	private void PrivateStructFunc() {}
>> }
>>
>> export class List : IList
>> {
>> 	export this(uint Capacity)
>> 	{
>> 		vCapacity = Capacity;
>> 		internal = new Object[4];
>> 	}
>>
>> 	private Object[] internal;
>>
>> 	private uint vCapacity = 0;
>> 	export @property uint Capacity()
>> 	{
>> 		return vCapacity;
>> 	}
>>
>> 	private uint vCount = 0;
>> 	export @property uint Count()
>> 	{
>> 		return vCount;
>> 	}
>>
>> 	export void Add(Object Value)
>> 	{
>> 		if(vCount == vCapacity) internal.length *= 2;
>>
>> 		internal[++vCount] = Value;
>> 	}
>>
>> 	export int opApply(int delegate(ref Object) dg) { return 0; }
>> 	export IEnumerator GetEnumerator() {return null;}
>>
>> 	export @property bool IsFixedSize() {return false;}
>> 	export @property bool IsReadOnly() {return false;}
>>
>> 	export void Clear() { return; }
>> 	export bool Contains(Object Value) {return false;}
>> 	export Object opIndex(ulong Index) {return null;}
>> 	export void opIndexAssign(Object Value, ulong Index) {}
>> 	export int IndexOf(Object Value) {return 0;}
>> 	export void Insert(ulong Index, Object Value) {}
>> 	export void Remove(Object Value) {}
>> 	export void RemoveAt(ulong Index) {}
>> 	private int FooTest(int c) { return c*2; }
>> 	protected int ProtectedTest(int d) { return d*3; }
>> }
>>
>>
>>
>> This is the DI file as exported with the generation changes:
>>
>> // D import file generated from 'Basic.d'
>> module Horizon.Collections.Basic;
>> export void TestFunc(int a);
>> const gss = "__gshared";
>> export interface IEnumerator
>> {
>>      export @property Object Current();
>>
>>      export bool MoveNext();
>>      export void Reset();
>> }
>>
>> export interface ICollection
>> {
>>      export @property uint Count();
>>
>>      export int opApply(int delegate(ref Object) dg);
>>      export IEnumerator GetEnumerator();
>> }
>>
>> export interface IList : ICollection
>> {
>>      export @property bool IsFixedSize();
>>
>>      export @property bool IsReadOnly();
>>
>>      export void Add(Object Value);
>>      export void Clear();
>>      export bool Contains(Object Value);
>>      export Object opIndex(ulong Index);
>>      export void opIndexAssign(Object Value, ulong Index);
>>      export int IndexOf(Object Value);
>>      export void Insert(ulong Index, Object Value);
>>      export void Remove(Object Value);
>>      export void RemoveAt(ulong Index);
>> }
>>
>> export struct StructTesting
>> {
>>      private int i;
>>
>>      protected int j;
>>
>>      export void ExportedStructFunc();
>> }
>>
>> export class List : IList
>> {
>>      export this(uint Capacity);
>>      private Object[] internal;
>>
>>      private uint vCapacity;
>>
>>      export @property uint Capacity();
>>
>>      private uint vCount;
>>
>>      export @property uint Count();
>>
>>      export void Add(Object Value);
>>      export int opApply(int delegate(ref Object) dg);
>>      export IEnumerator GetEnumerator();
>>      export @property bool IsFixedSize();
>>
>>      export @property bool IsReadOnly();
>>
>>      export void Clear();
>>      export bool Contains(Object Value);
>>      export Object opIndex(ulong Index);
>>      export void opIndexAssign(Object Value, ulong Index);
>>      export int IndexOf(Object Value);
>>      export void Insert(ulong Index, Object Value);
>>      export void Remove(Object Value);
>>      export void RemoveAt(ulong Index);
>>      protected int ProtectedTest(int d);
>> }
>>
>> Let me know what you think!


-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
December 19, 2011
On Mon, 19 Dec 2011 08:53:21 -0800, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> On 12/19/11 2:11 AM, Adam Wilson wrote:
>> As you may all be aware, I've been trying to improve the automated
>> generation of .di files and I now have something that I feel is
>> testable. Currently the new code only makes the following changes.
>>
>> 1. Function Implementations are removed
>> 2. Private Function Declarations are removed.
>> 3. Variable Initializers, except for const, are removed.
>
> Don't forget immutable.

Added to my list. :-)

>> Everything else is left alone. Templates and mixins are not addressed
>> with this code and *should* not be modified. That's where I need your
>> help, the test cases I have written cover some basic scenarios but I
>> don't have the capability to test these changes with the diverse code
>> base that the community has created.
>>
>> drey_ from IRC was kind enough to test build Derelict with the changes
>> and has discovered a potential issue around private imports. Derelict
>> uses private imports that contain types which are used in function alias
>> declarations. As one would expect, this caused many compiler errors.
>> Currently, I feel that private imports should be stripped from the DI
>> file as they are intended to be internal to the module. However, I want
>> to put it to the community to decide, and I would especially appreciate
>> Mr. Bright's opinion on private imports in DI files.
>
> I suspect you'd still need the private imports because template code may use them.

Ok, this is a good reason, I'd still like to hear from Walter on the subject, but I think he'll probably agree. I'll add it to my list.

> This is great work. It's almost a textbook example of how one can make a great positive impact on D's development by finding an area of improvement and working on it. Congratulations!

Thank you! I have to admit that reason I took it on is because it was annoying the daylights out of me on my project, but then I guess that's how most open-source work gets done. :-)

> You may want to generate DIs for Phobos and take a look at them. Phobos uses a vast array of D's capabilities so it's an effective unittest for DI generation.

Now that is a good idea. I may need some help getting the test setup as I've never built anything as big as phobos before, but it would be a fantastic way to nail down the DI behavior and maybe help push Phobos to become a shared library.

> Thanks,
>
> Andrei


-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
December 19, 2011
Btw Adam, a long long time ago (well, few years ago) there was an effort for better DLL support in D: http://dsource.org/projects/ddl/

There was also a presentation here: http://vimeo.com/2264486 and
slides: http://replay.waybackmachine.org/20081203030930/http://team0xf.com/conference/DDL.pdf

Anyway, it might be a useful resource for what seem to be D's DLL problems.
December 19, 2011
On Monday, December 19, 2011 10:53:21 Andrei Alexandrescu wrote:
> You may want to generate DIs for Phobos and take a look at them. Phobos uses a vast array of D's capabilities so it's an effective unittest for DI generation.

It's a great test ground, but I wouldn't want to see Phobos actually be released with .di files instead of d files. It would seriously harm inlining and CTFE.

- Jonathan M Davis
December 19, 2011
On Mon, 19 Dec 2011 09:36:27 -0800, Andrej Mitrovic <andrej.mitrovich@gmail.com> wrote:

> Btw Adam, a long long time ago (well, few years ago) there was an
> effort for better DLL support in D: http://dsource.org/projects/ddl/
>
> There was also a presentation here: http://vimeo.com/2264486 and
> slides: http://replay.waybackmachine.org/20081203030930/http://team0xf.com/conference/DDL.pdf
>
> Anyway, it might be a useful resource for what seem to be D's DLL problems.

Those are awesome links. Thanks Andrej!

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
December 19, 2011
On Mon, 19 Dec 2011 06:27:20 -0800, Martin Nowak <dawg@dawgfoto.de> wrote:

> On Mon, 19 Dec 2011 14:22:10 +0100, Rainer Schuetze <r.sagitario@gmx.de> wrote:
>
>>
>>
>> On 19.12.2011 09:11, Adam Wilson wrote:
>>> Currently, I feel that private imports should be stripped from the DI
>>> file as they are intended to be internal to the module. However, I want
>>> to put it to the community to decide, and I would especially appreciate
>>> Mr. Bright's opinion on private imports in DI files.
>>
>> I'd very much like the dependencies to be shrinked in the di file, but I think you can only remove private imports if you can guarantee that no symbols are referenced by the generated di file (e.g. function signatures, variable types, templates).
>>
>> IIRC removing imports might also have an influence on generated static ctor execution order, because there are some optimizations that remove dependencies if no imports with static ctors are found in a referenced module.
>>
> No it should not, an import will not actually create the module info that
> contains the import dependencies. So the initialization order will still rely
> solely on the compiled module.
> OTOH not declaring private static ctors can lead to unreferenced module infos
> that would otherwise end up in the final executable.
>
>> Otherwise, having a version of the di file without any implementation sounds like a good idea.

At this point it sounds like I need to keep all private members and imports. No problem, that's easy enough.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
December 19, 2011
On Mon, 19 Dec 2011 04:57:54 -0800, so <so@so.so> wrote:

> On Mon, 19 Dec 2011 14:43:07 +0200, Jakob Ovrum <jakobovrum@gmail.com> wrote:
>
>> On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:
>>> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:
>>>
>>>> Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.
>>>
>>> I am not exactly sure about your problem with templates and mixins but i'll give it a try regardless :)
>>> Since with templates there is no distinction between definition and decleration,
>>> exposing them IMO should be solely based on thier module access signatures.
>>>
>>> private struct A() // hide
>>> public struct B()  // expose
>>>
>>> Now if B or some another exposed structure in ".di" should call A,
>>> compiler will take care of it by outputting an error as usual.
>>
>> And if the public template tries to access the private one?
>>
>> Private module members must be treated like any other unless the compiler can prove it removed all references to the private member.
>
> You are right it was obscure, i'll try to clarify.
> By "B or some another exposed structure" i meant things resides "completely" in ".di" file.
> Things like Templates, mixins and if we get them one day inlined functions.
> In this scenario a "original.d" file should be interpreted as:
>
> original.d
>    public void A()()
>    private void B()()
>
> whatitwasmeant.di
>    public void A()()
>
> whatitwasmeant.d
>    private void B()()
>
> Here B can call A but not the opposite.
> A simple doesn't see B here. Things that resides completely in ".di" file have no way to access the things in its ".d" file unless it wasn't "exported".
> Imagine the final/generated ".di" file IS the user and the original ".d" file is the library designer. While i think this is the right way, it probably not that easy to implement.

So the consensus is to keep all private members/imports/etc?

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
December 19, 2011
On Mon, 19 Dec 2011 22:24:18 +0200, Adam Wilson <flyboynw@gmail.com> wrote:

> So the consensus is to keep all private members/imports/etc?

Ideally to get most out of the auto generation of di files (by definition of module system) we need to strip ALL private stuff.
Only this way we can say the automation is on par with handcrafting.
But if it does not look doable somehow, we can always handcraft them (which i'll generally prefer),
December 19, 2011
On Mon, 19 Dec 2011 23:04:27 +0200, so <so@so.so> wrote:

> On Mon, 19 Dec 2011 22:24:18 +0200, Adam Wilson <flyboynw@gmail.com> wrote:
>
>> So the consensus is to keep all private members/imports/etc?
>
> Ideally to get most out of the auto generation of di files (by definition of module system) we need to strip ALL private stuff.
> Only this way we can say the automation is on par with handcrafting.
> But if it does not look doable somehow, we can always handcraft them (which i'll generally prefer),

ALL "except" imports :)
December 19, 2011
On Monday, 19 December 2011 at 21:04:28 UTC, so wrote:
> On Mon, 19 Dec 2011 22:24:18 +0200, Adam Wilson <flyboynw@gmail.com> wrote:
>
>> So the consensus is to keep all private members/imports/etc?
>
> Ideally to get most out of the auto generation of di files (by definition of module system) we need to strip ALL private stuff.
> Only this way we can say the automation is on par with handcrafting.
> But if it does not look doable somehow, we can always handcraft them (which i'll generally prefer),

Private members can still be used from public templates and whatnot. You can't take them out unless you can statically proved all references within the same .di to that private member was removed in the stripping process.

foo.d
------

private i = 42;

// Is a public template, has to be left in
void bar()()
{
   /* Uses 'i' */
}

void baz()
{
   /* Also uses 'i' */
}
------

foo.di
------

void bar()()
{
   /* Still uses 'i', where did it go!? */
}

void baz(); // Fine, no longer need 'i'
------

Also, on a side note, function parameters can use types from private imports too. Be it function, data or import, it has to be left in even if private.