Jump to page: 1 26  
Page
Thread overview
DI Generation Needs your Help!
Dec 19, 2011
Adam Wilson
Dec 19, 2011
Jakob Ovrum
Dec 19, 2011
so
Dec 19, 2011
Adam Wilson
Dec 19, 2011
so
Dec 19, 2011
so
Dec 19, 2011
Jakob Ovrum
Dec 19, 2011
so
Dec 19, 2011
Adam Wilson
Dec 19, 2011
so
Dec 19, 2011
so
Dec 19, 2011
Jakob Ovrum
Dec 20, 2011
Adam Wilson
Dec 19, 2011
Rainer Schuetze
Dec 19, 2011
Martin Nowak
Dec 19, 2011
Adam Wilson
Dec 19, 2011
Adam Wilson
Dec 19, 2011
Andrej Mitrovic
Dec 19, 2011
Adam Wilson
Dec 19, 2011
Jonathan M Davis
Dec 20, 2011
Adam Wilson
Dec 20, 2011
Adam Wilson
Dec 20, 2011
Andrej Mitrovic
Dec 21, 2011
Adam Wilson
Dec 21, 2011
Adam Wilson
Dec 21, 2011
Adam Wilson
Dec 21, 2011
Jakob Ovrum
Dec 21, 2011
Benjamin Thaut
Dec 21, 2011
Timon Gehr
Dec 22, 2011
Adam Wilson
Dec 22, 2011
Jacob Carlborg
Dec 22, 2011
Adam Wilson
Dec 22, 2011
Jacob Carlborg
Dec 23, 2011
Adam Wilson
Dec 23, 2011
Jacob Carlborg
Dec 22, 2011
Adam Wilson
Mar 14, 2012
Andrej Mitrovic
Mar 14, 2012
Adam Wilson
Mar 14, 2012
Andrej Mitrovic
Mar 14, 2012
Adam Wilson
Mar 14, 2012
Andrej Mitrovic
Mar 14, 2012
Adam Wilson
Mar 14, 2012
Alvaro
Mar 14, 2012
Adam Wilson
Mar 14, 2012
Adam D. Ruppe
Mar 14, 2012
Adam Wilson
Mar 14, 2012
Alvaro
Mar 14, 2012
Adam Wilson
December 19, 2011
As you may all be aware, I've been trying to improve the automated generation of .di files and I now have something that I feel is testable. Currently the new code only makes the following changes.

1.	Function Implementations are removed
2.	Private Function Declarations are removed.
3.	Variable Initializers, except for const, are removed.

Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.

drey_ from IRC was kind enough to test build Derelict with the changes and has discovered a potential issue around private imports. Derelict uses private imports that contain types which are used in function alias declarations. As one would expect, this caused many compiler errors.  Currently, I feel that private imports should be stripped from the DI file as they are intended to be internal to the module. However, I want to put it to the community to decide, and I would especially appreciate Mr. Bright's opinion on private imports in DI files.

If anyone wishes to test my changes against their code, you can download them from my git account here: https://LightBender@github.com/LightBender/dmd.git
I only ask that you inform me of anything broken and provide a test case that reproduces the error.

The following is my current test case, it comes from a module in my project that has been modified to include the various tests I and others have been able to think of. I know for a fact that it is missing a great number of features available in D, and I would really appreciate any tests you might wish to add to this case.

module Horizon.Collections.Basic;

export void TestFunc(int a) {}
private void TestFunc2(int b) {}

const gss = "__gshared";

export interface IEnumerator
{
	export @property Object Current();

	export bool MoveNext();
	export void Reset();
}

export interface ICollection
{
	export @property uint Count();

	export int opApply(int delegate(ref Object) dg);
	export IEnumerator GetEnumerator();
}

export interface IList : ICollection
{
	export @property bool IsFixedSize();
	export @property bool IsReadOnly();

	export void Add(Object Value);
	export void Clear();
	export bool Contains(Object Value);
	export Object opIndex(ulong Index);
	export void opIndexAssign(Object Value, ulong Index);
	export int IndexOf(Object Value);
	export void Insert(ulong Index, Object Value);
	export void Remove(Object Value);
	export void RemoveAt(ulong Index);
}

export struct StructTesting
{
	private int i = 0;
	protected int j = 1;

	export void ExportedStructFunc() {}
	private void PrivateStructFunc() {}
}

export class List : IList
{
	export this(uint Capacity)
	{
		vCapacity = Capacity;
		internal = new Object[4];
	}

	private Object[] internal;

	private uint vCapacity = 0;
	export @property uint Capacity()
	{
		return vCapacity;
	}

	private uint vCount = 0;
	export @property uint Count()
	{
		return vCount;
	}

	export void Add(Object Value)
	{
		if(vCount == vCapacity) internal.length *= 2;

		internal[++vCount] = Value;
	}

	export int opApply(int delegate(ref Object) dg) { return 0; }
	export IEnumerator GetEnumerator() {return null;}

	export @property bool IsFixedSize() {return false;}
	export @property bool IsReadOnly() {return false;}

	export void Clear() { return; }
	export bool Contains(Object Value) {return false;}
	export Object opIndex(ulong Index) {return null;}
	export void opIndexAssign(Object Value, ulong Index) {}
	export int IndexOf(Object Value) {return 0;}
	export void Insert(ulong Index, Object Value) {}
	export void Remove(Object Value) {}
	export void RemoveAt(ulong Index) {}
	private int FooTest(int c) { return c*2; }
	protected int ProtectedTest(int d) { return d*3; }
}



This is the DI file as exported with the generation changes:

// D import file generated from 'Basic.d'
module Horizon.Collections.Basic;
export void TestFunc(int a);
const gss = "__gshared";
export interface IEnumerator
{
    export @property Object Current();

    export bool MoveNext();
    export void Reset();
}

export interface ICollection
{
    export @property uint Count();

    export int opApply(int delegate(ref Object) dg);
    export IEnumerator GetEnumerator();
}

export interface IList : ICollection
{
    export @property bool IsFixedSize();

    export @property bool IsReadOnly();

    export void Add(Object Value);
    export void Clear();
    export bool Contains(Object Value);
    export Object opIndex(ulong Index);
    export void opIndexAssign(Object Value, ulong Index);
    export int IndexOf(Object Value);
    export void Insert(ulong Index, Object Value);
    export void Remove(Object Value);
    export void RemoveAt(ulong Index);
}

export struct StructTesting
{
    private int i;

    protected int j;

    export void ExportedStructFunc();
}

export class List : IList
{
    export this(uint Capacity);
    private Object[] internal;

    private uint vCapacity;

    export @property uint Capacity();

    private uint vCount;

    export @property uint Count();

    export void Add(Object Value);
    export int opApply(int delegate(ref Object) dg);
    export IEnumerator GetEnumerator();
    export @property bool IsFixedSize();

    export @property bool IsReadOnly();

    export void Clear();
    export bool Contains(Object Value);
    export Object opIndex(ulong Index);
    export void opIndexAssign(Object Value, ulong Index);
    export int IndexOf(Object Value);
    export void Insert(ulong Index, Object Value);
    export void Remove(Object Value);
    export void RemoveAt(ulong Index);
    protected int ProtectedTest(int d);
}

Let me know what you think!

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
December 19, 2011
On Monday, 19 December 2011 at 08:11:26 UTC, Adam Wilson wrote:
> Let me know what you think!

Nice work! I think this is going to be more and more relevant now that the situation with shared libraries is getting more interesting in D.

I would argue that anything that can't be moved to a different library, like templates, manifest constants etc. must be kept in as-is. D interface generation isn't just useful for hiding source code; some people just want to put as much as possible in a shared library, speed up compilation or hide as much as possible with the restriction of keeping the code working as usual.

If people want to hide their source code at all cost, they can make this do a pass to easily identify what needs to be moved to a different module, then do the rest manually (which of course, only needs to be done once).

December 19, 2011
Good work Adam!

After class/struct/interface level "export" we shouldn't need it again.
It already implies that everything (if not stated otherwise via private) will be exported.

Removing them also would automagicaly solve the inconsistency below.

> export class List : IList
> {
>      export void RemoveAt(ulong Index);
>      protected int ProtectedTest(int d);
> }

On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:

> As you may all be aware, I've been trying to improve the automated generation of .di files and I now have something that I feel is testable. Currently the new code only makes the following changes.
>
> 1.	Function Implementations are removed
> 2.	Private Function Declarations are removed.
> 3.	Variable Initializers, except for const, are removed.
>
> Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.
>
> drey_ from IRC was kind enough to test build Derelict with the changes and has discovered a potential issue around private imports. Derelict uses private imports that contain types which are used in function alias declarations. As one would expect, this caused many compiler errors.  Currently, I feel that private imports should be stripped from the DI file as they are intended to be internal to the module. However, I want to put it to the community to decide, and I would especially appreciate Mr. Bright's opinion on private imports in DI files.
>
> If anyone wishes to test my changes against their code, you can download them from my git account here: https://LightBender@github.com/LightBender/dmd.git
> I only ask that you inform me of anything broken and provide a test case that reproduces the error.
>
> The following is my current test case, it comes from a module in my project that has been modified to include the various tests I and others have been able to think of. I know for a fact that it is missing a great number of features available in D, and I would really appreciate any tests you might wish to add to this case.
>
> module Horizon.Collections.Basic;
>
> export void TestFunc(int a) {}
> private void TestFunc2(int b) {}
>
> const gss = "__gshared";
>
> export interface IEnumerator
> {
> 	export @property Object Current();
>
> 	export bool MoveNext();
> 	export void Reset();
> }
>
> export interface ICollection
> {
> 	export @property uint Count();
>
> 	export int opApply(int delegate(ref Object) dg);
> 	export IEnumerator GetEnumerator();
> }
>
> export interface IList : ICollection
> {
> 	export @property bool IsFixedSize();
> 	export @property bool IsReadOnly();
>
> 	export void Add(Object Value);
> 	export void Clear();
> 	export bool Contains(Object Value);
> 	export Object opIndex(ulong Index);
> 	export void opIndexAssign(Object Value, ulong Index);
> 	export int IndexOf(Object Value);
> 	export void Insert(ulong Index, Object Value);
> 	export void Remove(Object Value);
> 	export void RemoveAt(ulong Index);
> }
>
> export struct StructTesting
> {
> 	private int i = 0;
> 	protected int j = 1;
>
> 	export void ExportedStructFunc() {}
> 	private void PrivateStructFunc() {}
> }
>
> export class List : IList
> {
> 	export this(uint Capacity)
> 	{
> 		vCapacity = Capacity;
> 		internal = new Object[4];
> 	}
>
> 	private Object[] internal;
>
> 	private uint vCapacity = 0;
> 	export @property uint Capacity()
> 	{
> 		return vCapacity;
> 	}
>
> 	private uint vCount = 0;
> 	export @property uint Count()
> 	{
> 		return vCount;
> 	}
>
> 	export void Add(Object Value)
> 	{
> 		if(vCount == vCapacity) internal.length *= 2;
>
> 		internal[++vCount] = Value;
> 	}
>
> 	export int opApply(int delegate(ref Object) dg) { return 0; }
> 	export IEnumerator GetEnumerator() {return null;}
>
> 	export @property bool IsFixedSize() {return false;}
> 	export @property bool IsReadOnly() {return false;}
>
> 	export void Clear() { return; }
> 	export bool Contains(Object Value) {return false;}
> 	export Object opIndex(ulong Index) {return null;}
> 	export void opIndexAssign(Object Value, ulong Index) {}
> 	export int IndexOf(Object Value) {return 0;}
> 	export void Insert(ulong Index, Object Value) {}
> 	export void Remove(Object Value) {}
> 	export void RemoveAt(ulong Index) {}
> 	private int FooTest(int c) { return c*2; }
> 	protected int ProtectedTest(int d) { return d*3; }
> }
>
>
>
> This is the DI file as exported with the generation changes:
>
> // D import file generated from 'Basic.d'
> module Horizon.Collections.Basic;
> export void TestFunc(int a);
> const gss = "__gshared";
> export interface IEnumerator
> {
>      export @property Object Current();
>
>      export bool MoveNext();
>      export void Reset();
> }
>
> export interface ICollection
> {
>      export @property uint Count();
>
>      export int opApply(int delegate(ref Object) dg);
>      export IEnumerator GetEnumerator();
> }
>
> export interface IList : ICollection
> {
>      export @property bool IsFixedSize();
>
>      export @property bool IsReadOnly();
>
>      export void Add(Object Value);
>      export void Clear();
>      export bool Contains(Object Value);
>      export Object opIndex(ulong Index);
>      export void opIndexAssign(Object Value, ulong Index);
>      export int IndexOf(Object Value);
>      export void Insert(ulong Index, Object Value);
>      export void Remove(Object Value);
>      export void RemoveAt(ulong Index);
> }
>
> export struct StructTesting
> {
>      private int i;
>
>      protected int j;
>
>      export void ExportedStructFunc();
> }
>
> export class List : IList
> {
>      export this(uint Capacity);
>      private Object[] internal;
>
>      private uint vCapacity;
>
>      export @property uint Capacity();
>
>      private uint vCount;
>
>      export @property uint Count();
>
>      export void Add(Object Value);
>      export int opApply(int delegate(ref Object) dg);
>      export IEnumerator GetEnumerator();
>      export @property bool IsFixedSize();
>
>      export @property bool IsReadOnly();
>
>      export void Clear();
>      export bool Contains(Object Value);
>      export Object opIndex(ulong Index);
>      export void opIndexAssign(Object Value, ulong Index);
>      export int IndexOf(Object Value);
>      export void Insert(ulong Index, Object Value);
>      export void Remove(Object Value);
>      export void RemoveAt(ulong Index);
>      protected int ProtectedTest(int d);
> }
>
> Let me know what you think!
December 19, 2011
On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:

> Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.

I am not exactly sure about your problem with templates and mixins but i'll give it a try regardless :)
Since with templates there is no distinction between definition and decleration,
exposing them IMO should be solely based on thier module access signatures.

private struct A() // hide
public struct B()  // expose

Now if B or some another exposed structure in ".di" should call A,
compiler will take care of it by outputting an error as usual.
December 19, 2011
On Mon, 19 Dec 2011 14:11:31 +0200, so <so@so.so> wrote:

> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:
>
>> Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.
>
> I am not exactly sure about your problem with templates and mixins but i'll give it a try regardless :)
> Since with templates there is no distinction between definition and decleration,
> exposing them IMO should be solely based on thier module access signatures.
>
> private struct A() // hide
> public struct B()  // expose
>
> Now if B or some another exposed structure in ".di" should call A,
> compiler will take care of it by outputting an error as usual.

One last thing!
Interfaces shouldn't have any business with "export", they should be treated just like templates.
December 19, 2011
On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:
> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:
>
>> Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.
>
> I am not exactly sure about your problem with templates and mixins but i'll give it a try regardless :)
> Since with templates there is no distinction between definition and decleration,
> exposing them IMO should be solely based on thier module access signatures.
>
> private struct A() // hide
> public struct B()  // expose
>
> Now if B or some another exposed structure in ".di" should call A,
> compiler will take care of it by outputting an error as usual.

And if the public template tries to access the private one?

Private module members must be treated like any other unless the compiler can prove it removed all references to the private member.
December 19, 2011
On Mon, 19 Dec 2011 14:43:07 +0200, Jakob Ovrum <jakobovrum@gmail.com> wrote:

> On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:
>> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:
>>
>>> Everything else is left alone. Templates and mixins are not addressed with this code and *should* not be modified. That's where I need your help, the test cases I have written cover some basic scenarios but I don't have the capability to test these changes with the diverse code base that the community has created.
>>
>> I am not exactly sure about your problem with templates and mixins but i'll give it a try regardless :)
>> Since with templates there is no distinction between definition and decleration,
>> exposing them IMO should be solely based on thier module access signatures.
>>
>> private struct A() // hide
>> public struct B()  // expose
>>
>> Now if B or some another exposed structure in ".di" should call A,
>> compiler will take care of it by outputting an error as usual.
>
> And if the public template tries to access the private one?
>
> Private module members must be treated like any other unless the compiler can prove it removed all references to the private member.

You are right it was obscure, i'll try to clarify.
By "B or some another exposed structure" i meant things resides "completely" in ".di" file.
Things like Templates, mixins and if we get them one day inlined functions.
In this scenario a "original.d" file should be interpreted as:

original.d
  public void A()()
  private void B()()

whatitwasmeant.di
  public void A()()

whatitwasmeant.d
  private void B()()

Here B can call A but not the opposite.
A simple doesn't see B here. Things that resides completely in ".di" file have no way to access the things in its ".d" file unless it wasn't "exported".
Imagine the final/generated ".di" file IS the user and the original ".d" file is the library designer. While i think this is the right way, it probably not that easy to implement.
December 19, 2011

On 19.12.2011 09:11, Adam Wilson wrote:
> Currently, I feel that private imports should be stripped from the DI
> file as they are intended to be internal to the module. However, I want
> to put it to the community to decide, and I would especially appreciate
> Mr. Bright's opinion on private imports in DI files.

I'd very much like the dependencies to be shrinked in the di file, but I think you can only remove private imports if you can guarantee that no symbols are referenced by the generated di file (e.g. function signatures, variable types, templates).

IIRC removing imports might also have an influence on generated static ctor execution order, because there are some optimizations that remove dependencies if no imports with static ctors are found in a referenced module.

Otherwise, having a version of the di file without any implementation sounds like a good idea.
December 19, 2011
On Mon, 19 Dec 2011 14:22:10 +0100, Rainer Schuetze <r.sagitario@gmx.de> wrote:

>
>
> On 19.12.2011 09:11, Adam Wilson wrote:
>> Currently, I feel that private imports should be stripped from the DI
>> file as they are intended to be internal to the module. However, I want
>> to put it to the community to decide, and I would especially appreciate
>> Mr. Bright's opinion on private imports in DI files.
>
> I'd very much like the dependencies to be shrinked in the di file, but I think you can only remove private imports if you can guarantee that no symbols are referenced by the generated di file (e.g. function signatures, variable types, templates).
>
> IIRC removing imports might also have an influence on generated static ctor execution order, because there are some optimizations that remove dependencies if no imports with static ctors are found in a referenced module.
>
No it should not, an import will not actually create the module info that
contains the import dependencies. So the initialization order will still rely
solely on the compiled module.
OTOH not declaring private static ctors can lead to unreferenced module infos
that would otherwise end up in the final executable.

> Otherwise, having a version of the di file without any implementation sounds like a good idea.
December 19, 2011
On 12/19/11 2:11 AM, Adam Wilson wrote:
> As you may all be aware, I've been trying to improve the automated
> generation of .di files and I now have something that I feel is
> testable. Currently the new code only makes the following changes.
>
> 1. Function Implementations are removed
> 2. Private Function Declarations are removed.
> 3. Variable Initializers, except for const, are removed.

Don't forget immutable.

> Everything else is left alone. Templates and mixins are not addressed
> with this code and *should* not be modified. That's where I need your
> help, the test cases I have written cover some basic scenarios but I
> don't have the capability to test these changes with the diverse code
> base that the community has created.
>
> drey_ from IRC was kind enough to test build Derelict with the changes
> and has discovered a potential issue around private imports. Derelict
> uses private imports that contain types which are used in function alias
> declarations. As one would expect, this caused many compiler errors.
> Currently, I feel that private imports should be stripped from the DI
> file as they are intended to be internal to the module. However, I want
> to put it to the community to decide, and I would especially appreciate
> Mr. Bright's opinion on private imports in DI files.

I suspect you'd still need the private imports because template code may use them.

This is great work. It's almost a textbook example of how one can make a great positive impact on D's development by finding an area of improvement and working on it. Congratulations!

You may want to generate DIs for Phobos and take a look at them. Phobos uses a vast array of D's capabilities so it's an effective unittest for DI generation.


Thanks,

Andrei
« First   ‹ Prev
1 2 3 4 5 6