View mode: basic / threaded / horizontal-split · Log in · Help
December 19, 2011
DI Generation Needs your Help!
As you may all be aware, I've been trying to improve the automated  
generation of .di files and I now have something that I feel is testable.  
Currently the new code only makes the following changes.

1.	Function Implementations are removed
2.	Private Function Declarations are removed.
3.	Variable Initializers, except for const, are removed.

Everything else is left alone. Templates and mixins are not addressed with  
this code and *should* not be modified. That's where I need your help, the  
test cases I have written cover some basic scenarios but I don't have the  
capability to test these changes with the diverse code base that the  
community has created.

drey_ from IRC was kind enough to test build Derelict with the changes and  
has discovered a potential issue around private imports. Derelict uses  
private imports that contain types which are used in function alias  
declarations. As one would expect, this caused many compiler errors.   
Currently, I feel that private imports should be stripped from the DI file  
as they are intended to be internal to the module. However, I want to put  
it to the community to decide, and I would especially appreciate Mr.  
Bright's opinion on private imports in DI files.

If anyone wishes to test my changes against their code, you can download  
them from my git account here:  
https://LightBender@github.com/LightBender/dmd.git
I only ask that you inform me of anything broken and provide a test case  
that reproduces the error.

The following is my current test case, it comes from a module in my  
project that has been modified to include the various tests I and others  
have been able to think of. I know for a fact that it is missing a great  
number of features available in D, and I would really appreciate any tests  
you might wish to add to this case.

module Horizon.Collections.Basic;

export void TestFunc(int a) {}
private void TestFunc2(int b) {}

const gss = "__gshared";

export interface IEnumerator
{
	export @property Object Current();

	export bool MoveNext();
	export void Reset();
}

export interface ICollection
{
	export @property uint Count();

	export int opApply(int delegate(ref Object) dg);
	export IEnumerator GetEnumerator();
}

export interface IList : ICollection
{
	export @property bool IsFixedSize();
	export @property bool IsReadOnly();

	export void Add(Object Value);
	export void Clear();
	export bool Contains(Object Value);
	export Object opIndex(ulong Index);
	export void opIndexAssign(Object Value, ulong Index);
	export int IndexOf(Object Value);
	export void Insert(ulong Index, Object Value);
	export void Remove(Object Value);
	export void RemoveAt(ulong Index);
}

export struct StructTesting
{
	private int i = 0;
	protected int j = 1;

	export void ExportedStructFunc() {}
	private void PrivateStructFunc() {}
}

export class List : IList
{
	export this(uint Capacity)
	{
		vCapacity = Capacity;
		internal = new Object[4];
	}

	private Object[] internal;

	private uint vCapacity = 0;
	export @property uint Capacity()
	{
		return vCapacity;
	}

	private uint vCount = 0;
	export @property uint Count()
	{
		return vCount;
	}

	export void Add(Object Value)
	{
		if(vCount == vCapacity) internal.length *= 2;

		internal[++vCount] = Value;
	}

	export int opApply(int delegate(ref Object) dg) { return 0; }
	export IEnumerator GetEnumerator() {return null;}

	export @property bool IsFixedSize() {return false;}
	export @property bool IsReadOnly() {return false;}

	export void Clear() { return; }
	export bool Contains(Object Value) {return false;}
	export Object opIndex(ulong Index) {return null;}
	export void opIndexAssign(Object Value, ulong Index) {}
	export int IndexOf(Object Value) {return 0;}
	export void Insert(ulong Index, Object Value) {}
	export void Remove(Object Value) {}
	export void RemoveAt(ulong Index) {}
	private int FooTest(int c) { return c*2; }
	protected int ProtectedTest(int d) { return d*3; }
}



This is the DI file as exported with the generation changes:

// D import file generated from 'Basic.d'
module Horizon.Collections.Basic;
export void TestFunc(int a);
const gss = "__gshared";
export interface IEnumerator
{
    export @property Object Current();

    export bool MoveNext();
    export void Reset();
}

export interface ICollection
{
    export @property uint Count();

    export int opApply(int delegate(ref Object) dg);
    export IEnumerator GetEnumerator();
}

export interface IList : ICollection
{
    export @property bool IsFixedSize();

    export @property bool IsReadOnly();

    export void Add(Object Value);
    export void Clear();
    export bool Contains(Object Value);
    export Object opIndex(ulong Index);
    export void opIndexAssign(Object Value, ulong Index);
    export int IndexOf(Object Value);
    export void Insert(ulong Index, Object Value);
    export void Remove(Object Value);
    export void RemoveAt(ulong Index);
}

export struct StructTesting
{
    private int i;

    protected int j;

    export void ExportedStructFunc();
}

export class List : IList
{
    export this(uint Capacity);
    private Object[] internal;

    private uint vCapacity;

    export @property uint Capacity();

    private uint vCount;

    export @property uint Count();

    export void Add(Object Value);
    export int opApply(int delegate(ref Object) dg);
    export IEnumerator GetEnumerator();
    export @property bool IsFixedSize();

    export @property bool IsReadOnly();

    export void Clear();
    export bool Contains(Object Value);
    export Object opIndex(ulong Index);
    export void opIndexAssign(Object Value, ulong Index);
    export int IndexOf(Object Value);
    export void Insert(ulong Index, Object Value);
    export void Remove(Object Value);
    export void RemoveAt(ulong Index);
    protected int ProtectedTest(int d);
}

Let me know what you think!

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
December 19, 2011
Re: DI Generation Needs your Help!
On Monday, 19 December 2011 at 08:11:26 UTC, Adam Wilson wrote:
> Let me know what you think!

Nice work! I think this is going to be more and more relevant now 
that the situation with shared libraries is getting more 
interesting in D.

I would argue that anything that can't be moved to a different 
library, like templates, manifest constants etc. must be kept in 
as-is. D interface generation isn't just useful for hiding source 
code; some people just want to put as much as possible in a 
shared library, speed up compilation or hide as much as possible 
with the restriction of keeping the code working as usual.

If people want to hide their source code at all cost, they can 
make this do a pass to easily identify what needs to be moved to 
a different module, then do the rest manually (which of course, 
only needs to be done once).
December 19, 2011
Re: DI Generation Needs your Help!
Good work Adam!

After class/struct/interface level "export" we shouldn't need it again.
It already implies that everything (if not stated otherwise via private)  
will be exported.

Removing them also would automagicaly solve the inconsistency below.

> export class List : IList
> {
>      export void RemoveAt(ulong Index);
>      protected int ProtectedTest(int d);
> }

On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:

> As you may all be aware, I've been trying to improve the automated  
> generation of .di files and I now have something that I feel is  
> testable. Currently the new code only makes the following changes.
>
> 1.	Function Implementations are removed
> 2.	Private Function Declarations are removed.
> 3.	Variable Initializers, except for const, are removed.
>
> Everything else is left alone. Templates and mixins are not addressed  
> with this code and *should* not be modified. That's where I need your  
> help, the test cases I have written cover some basic scenarios but I  
> don't have the capability to test these changes with the diverse code  
> base that the community has created.
>
> drey_ from IRC was kind enough to test build Derelict with the changes  
> and has discovered a potential issue around private imports. Derelict  
> uses private imports that contain types which are used in function alias  
> declarations. As one would expect, this caused many compiler errors.   
> Currently, I feel that private imports should be stripped from the DI  
> file as they are intended to be internal to the module. However, I want  
> to put it to the community to decide, and I would especially appreciate  
> Mr. Bright's opinion on private imports in DI files.
>
> If anyone wishes to test my changes against their code, you can download  
> them from my git account here:  
> https://LightBender@github.com/LightBender/dmd.git
> I only ask that you inform me of anything broken and provide a test case  
> that reproduces the error.
>
> The following is my current test case, it comes from a module in my  
> project that has been modified to include the various tests I and others  
> have been able to think of. I know for a fact that it is missing a great  
> number of features available in D, and I would really appreciate any  
> tests you might wish to add to this case.
>
> module Horizon.Collections.Basic;
>
> export void TestFunc(int a) {}
> private void TestFunc2(int b) {}
>
> const gss = "__gshared";
>
> export interface IEnumerator
> {
> 	export @property Object Current();
>
> 	export bool MoveNext();
> 	export void Reset();
> }
>
> export interface ICollection
> {
> 	export @property uint Count();
>
> 	export int opApply(int delegate(ref Object) dg);
> 	export IEnumerator GetEnumerator();
> }
>
> export interface IList : ICollection
> {
> 	export @property bool IsFixedSize();
> 	export @property bool IsReadOnly();
>
> 	export void Add(Object Value);
> 	export void Clear();
> 	export bool Contains(Object Value);
> 	export Object opIndex(ulong Index);
> 	export void opIndexAssign(Object Value, ulong Index);
> 	export int IndexOf(Object Value);
> 	export void Insert(ulong Index, Object Value);
> 	export void Remove(Object Value);
> 	export void RemoveAt(ulong Index);
> }
>
> export struct StructTesting
> {
> 	private int i = 0;
> 	protected int j = 1;
>
> 	export void ExportedStructFunc() {}
> 	private void PrivateStructFunc() {}
> }
>
> export class List : IList
> {
> 	export this(uint Capacity)
> 	{
> 		vCapacity = Capacity;
> 		internal = new Object[4];
> 	}
>
> 	private Object[] internal;
>
> 	private uint vCapacity = 0;
> 	export @property uint Capacity()
> 	{
> 		return vCapacity;
> 	}
>
> 	private uint vCount = 0;
> 	export @property uint Count()
> 	{
> 		return vCount;
> 	}
>
> 	export void Add(Object Value)
> 	{
> 		if(vCount == vCapacity) internal.length *= 2;
>
> 		internal[++vCount] = Value;
> 	}
>
> 	export int opApply(int delegate(ref Object) dg) { return 0; }
> 	export IEnumerator GetEnumerator() {return null;}
>
> 	export @property bool IsFixedSize() {return false;}
> 	export @property bool IsReadOnly() {return false;}
>
> 	export void Clear() { return; }
> 	export bool Contains(Object Value) {return false;}
> 	export Object opIndex(ulong Index) {return null;}
> 	export void opIndexAssign(Object Value, ulong Index) {}
> 	export int IndexOf(Object Value) {return 0;}
> 	export void Insert(ulong Index, Object Value) {}
> 	export void Remove(Object Value) {}
> 	export void RemoveAt(ulong Index) {}
> 	private int FooTest(int c) { return c*2; }
> 	protected int ProtectedTest(int d) { return d*3; }
> }
>
>
>
> This is the DI file as exported with the generation changes:
>
> // D import file generated from 'Basic.d'
> module Horizon.Collections.Basic;
> export void TestFunc(int a);
> const gss = "__gshared";
> export interface IEnumerator
> {
>      export @property Object Current();
>
>      export bool MoveNext();
>      export void Reset();
> }
>
> export interface ICollection
> {
>      export @property uint Count();
>
>      export int opApply(int delegate(ref Object) dg);
>      export IEnumerator GetEnumerator();
> }
>
> export interface IList : ICollection
> {
>      export @property bool IsFixedSize();
>
>      export @property bool IsReadOnly();
>
>      export void Add(Object Value);
>      export void Clear();
>      export bool Contains(Object Value);
>      export Object opIndex(ulong Index);
>      export void opIndexAssign(Object Value, ulong Index);
>      export int IndexOf(Object Value);
>      export void Insert(ulong Index, Object Value);
>      export void Remove(Object Value);
>      export void RemoveAt(ulong Index);
> }
>
> export struct StructTesting
> {
>      private int i;
>
>      protected int j;
>
>      export void ExportedStructFunc();
> }
>
> export class List : IList
> {
>      export this(uint Capacity);
>      private Object[] internal;
>
>      private uint vCapacity;
>
>      export @property uint Capacity();
>
>      private uint vCount;
>
>      export @property uint Count();
>
>      export void Add(Object Value);
>      export int opApply(int delegate(ref Object) dg);
>      export IEnumerator GetEnumerator();
>      export @property bool IsFixedSize();
>
>      export @property bool IsReadOnly();
>
>      export void Clear();
>      export bool Contains(Object Value);
>      export Object opIndex(ulong Index);
>      export void opIndexAssign(Object Value, ulong Index);
>      export int IndexOf(Object Value);
>      export void Insert(ulong Index, Object Value);
>      export void Remove(Object Value);
>      export void RemoveAt(ulong Index);
>      protected int ProtectedTest(int d);
> }
>
> Let me know what you think!
December 19, 2011
Re: DI Generation Needs your Help!
On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com> wrote:

> Everything else is left alone. Templates and mixins are not addressed  
> with this code and *should* not be modified. That's where I need your  
> help, the test cases I have written cover some basic scenarios but I  
> don't have the capability to test these changes with the diverse code  
> base that the community has created.

I am not exactly sure about your problem with templates and mixins but  
i'll give it a try regardless :)
Since with templates there is no distinction between definition and  
decleration,
exposing them IMO should be solely based on thier module access signatures.

private struct A() // hide
public struct B()  // expose

Now if B or some another exposed structure in ".di" should call A,
compiler will take care of it by outputting an error as usual.
December 19, 2011
Re: DI Generation Needs your Help!
On Mon, 19 Dec 2011 14:11:31 +0200, so <so@so.so> wrote:

> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com>  
> wrote:
>
>> Everything else is left alone. Templates and mixins are not addressed  
>> with this code and *should* not be modified. That's where I need your  
>> help, the test cases I have written cover some basic scenarios but I  
>> don't have the capability to test these changes with the diverse code  
>> base that the community has created.
>
> I am not exactly sure about your problem with templates and mixins but  
> i'll give it a try regardless :)
> Since with templates there is no distinction between definition and  
> decleration,
> exposing them IMO should be solely based on thier module access  
> signatures.
>
> private struct A() // hide
> public struct B()  // expose
>
> Now if B or some another exposed structure in ".di" should call A,
> compiler will take care of it by outputting an error as usual.

One last thing!
Interfaces shouldn't have any business with "export", they should be  
treated just like templates.
December 19, 2011
Re: DI Generation Needs your Help!
On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:
> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson 
> <flyboynw@gmail.com> wrote:
>
>> Everything else is left alone. Templates and mixins are not 
>> addressed with this code and *should* not be modified. That's 
>> where I need your help, the test cases I have written cover 
>> some basic scenarios but I don't have the capability to test 
>> these changes with the diverse code base that the community 
>> has created.
>
> I am not exactly sure about your problem with templates and 
> mixins but i'll give it a try regardless :)
> Since with templates there is no distinction between definition 
> and decleration,
> exposing them IMO should be solely based on thier module access 
> signatures.
>
> private struct A() // hide
> public struct B()  // expose
>
> Now if B or some another exposed structure in ".di" should call 
> A,
> compiler will take care of it by outputting an error as usual.

And if the public template tries to access the private one?

Private module members must be treated like any other unless the 
compiler can prove it removed all references to the private 
member.
December 19, 2011
Re: DI Generation Needs your Help!
On Mon, 19 Dec 2011 14:43:07 +0200, Jakob Ovrum <jakobovrum@gmail.com>  
wrote:

> On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:
>> On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw@gmail.com>  
>> wrote:
>>
>>> Everything else is left alone. Templates and mixins are not addressed  
>>> with this code and *should* not be modified. That's where I need your  
>>> help, the test cases I have written cover some basic scenarios but I  
>>> don't have the capability to test these changes with the diverse code  
>>> base that the community has created.
>>
>> I am not exactly sure about your problem with templates and mixins but  
>> i'll give it a try regardless :)
>> Since with templates there is no distinction between definition and  
>> decleration,
>> exposing them IMO should be solely based on thier module access  
>> signatures.
>>
>> private struct A() // hide
>> public struct B()  // expose
>>
>> Now if B or some another exposed structure in ".di" should call A,
>> compiler will take care of it by outputting an error as usual.
>
> And if the public template tries to access the private one?
>
> Private module members must be treated like any other unless the  
> compiler can prove it removed all references to the private member.

You are right it was obscure, i'll try to clarify.
By "B or some another exposed structure" i meant things resides  
"completely" in ".di" file.
Things like Templates, mixins and if we get them one day inlined functions.
In this scenario a "original.d" file should be interpreted as:

original.d
  public void A()()
  private void B()()

whatitwasmeant.di
  public void A()()

whatitwasmeant.d
  private void B()()

Here B can call A but not the opposite.
A simple doesn't see B here. Things that resides completely in ".di" file  
have no way to access the things in its ".d" file unless it wasn't  
"exported".
Imagine the final/generated ".di" file IS the user and the original ".d"  
file is the library designer. While i think this is the right way, it  
probably not that easy to implement.
December 19, 2011
Re: DI Generation Needs your Help!
On 19.12.2011 09:11, Adam Wilson wrote:
> Currently, I feel that private imports should be stripped from the DI
> file as they are intended to be internal to the module. However, I want
> to put it to the community to decide, and I would especially appreciate
> Mr. Bright's opinion on private imports in DI files.

I'd very much like the dependencies to be shrinked in the di file, but I 
think you can only remove private imports if you can guarantee that no 
symbols are referenced by the generated di file (e.g. function 
signatures, variable types, templates).

IIRC removing imports might also have an influence on generated static 
ctor execution order, because there are some optimizations that remove 
dependencies if no imports with static ctors are found in a referenced 
module.

Otherwise, having a version of the di file without any implementation 
sounds like a good idea.
December 19, 2011
Re: DI Generation Needs your Help!
On Mon, 19 Dec 2011 14:22:10 +0100, Rainer Schuetze <r.sagitario@gmx.de>  
wrote:

>
>
> On 19.12.2011 09:11, Adam Wilson wrote:
>> Currently, I feel that private imports should be stripped from the DI
>> file as they are intended to be internal to the module. However, I want
>> to put it to the community to decide, and I would especially appreciate
>> Mr. Bright's opinion on private imports in DI files.
>
> I'd very much like the dependencies to be shrinked in the di file, but I  
> think you can only remove private imports if you can guarantee that no  
> symbols are referenced by the generated di file (e.g. function  
> signatures, variable types, templates).
>
> IIRC removing imports might also have an influence on generated static  
> ctor execution order, because there are some optimizations that remove  
> dependencies if no imports with static ctors are found in a referenced  
> module.
>
No it should not, an import will not actually create the module info that
contains the import dependencies. So the initialization order will still  
rely
solely on the compiled module.
OTOH not declaring private static ctors can lead to unreferenced module  
infos
that would otherwise end up in the final executable.

> Otherwise, having a version of the di file without any implementation  
> sounds like a good idea.
December 19, 2011
Re: DI Generation Needs your Help!
On 12/19/11 2:11 AM, Adam Wilson wrote:
> As you may all be aware, I've been trying to improve the automated
> generation of .di files and I now have something that I feel is
> testable. Currently the new code only makes the following changes.
>
> 1. Function Implementations are removed
> 2. Private Function Declarations are removed.
> 3. Variable Initializers, except for const, are removed.

Don't forget immutable.

> Everything else is left alone. Templates and mixins are not addressed
> with this code and *should* not be modified. That's where I need your
> help, the test cases I have written cover some basic scenarios but I
> don't have the capability to test these changes with the diverse code
> base that the community has created.
>
> drey_ from IRC was kind enough to test build Derelict with the changes
> and has discovered a potential issue around private imports. Derelict
> uses private imports that contain types which are used in function alias
> declarations. As one would expect, this caused many compiler errors.
> Currently, I feel that private imports should be stripped from the DI
> file as they are intended to be internal to the module. However, I want
> to put it to the community to decide, and I would especially appreciate
> Mr. Bright's opinion on private imports in DI files.

I suspect you'd still need the private imports because template code may 
use them.

This is great work. It's almost a textbook example of how one can make a 
great positive impact on D's development by finding an area of 
improvement and working on it. Congratulations!

You may want to generate DIs for Phobos and take a look at them. Phobos 
uses a vast array of D's capabilities so it's an effective unittest for 
DI generation.


Thanks,

Andrei
« First   ‹ Prev
1 2 3 4 5
Top | Discussion index | About this forum | D home