On Wednesday, 14 March 2012 at 14:02:30 UTC, Steven Schveighoffer wrote:
> On Tue, 13 Mar 2012 22:39:25 -0400, Jakob Bornecrantz <wallbraker@gmail.com> wrote:
>
>> On Wednesday, 14 March 2012 at 00:52:32 UTC, H. S. Teoh wrote:
>>> Hi all,
>>>
>>> My AA implementation is slowly inching closer to being ready to replace aaA.d. So far I've been writing the implementation
>>> outside of object_.d for ease of testing & development; now I'm
>>> ready to start moving stuff into object_.d to start working on
>>> integration with druntime.
>>
>> Hi,
>>
>> If I'm understanding this correctly you are moving the entire
>> implementation of the AA into object.d and as such letting
>> programs be purview to its inner working? In sort meaning you
>> are making the entire AA implementation D ABI locked.
>>
>> This will make it impossible to either change the AA
>> implementation in any ABI breaking fashion or make it impossible
>> to pass AA's between libraries compiled against different
>> versions of druntime.
>>
>> Is this what we really want?
>
> This is unavoidable, whether it's a template or not.  What changes do you envision would be transparent using an opaque pImpl model (as was done in previous versions of phobos), but would break using templates?

struct AAver1(K, V)
{
   K[] tbl; V[] tlb2; uint size;
}

struct AAver2(K, V)
{
   K[] tbl; V[] tbl2; V[] optimizationTbl;
}

Would break if a AAver1 table was ever passed to code that
was compiled against a AAver2 table. In sort you could never
add anything to the AA struct. Without going in roundabout
ways of making sure you never access outside of any struct
version ever out there.

Or for that matter change how the internal tables are
populated by add and remove.

Cheers, Jakob.

On Thu, Mar 15, 2012 at 12:20:43AM +0100, Jakob Bornecrantz wrote: [...]
> struct AAver1(K, V)
> {
>    K[] tbl; V[] tlb2; uint size;
> }
> 
> struct AAver2(K, V)
> {
>    K[] tbl; V[] tbl2; V[] optimizationTbl;
> }
> 
> Would break if a AAver1 table was ever passed to code that was compiled against a AAver2 table. In sort you could never add anything to the AA struct. Without going in roundabout ways of making sure you never access outside of any struct version ever out there.
> 
> Or for that matter change how the internal tables are populated by add and remove.
[...]

How is this different from any other templates in druntime/phobos?

And FYI, the current AA implementation *already* suffers from this problem, because the Range interface already assumes a specific implementation behind the opaque pImpl pointer (see object_.d -- it *duplicates* the struct definitions from aaA.d and casts the void* into pointers to those structs). If aaA.d were to change its implementation today, the Range stuff in struct AssociativeArray would break horribly.

The motivation behind my rewriting AA's is to fix this schizophrenic mess. The internal aaA.d structs should *not* be duplicated in object_.d, but currently they are. So there are two options, either (1) we move everything back into aaA.d (and introduce a whole bunch more void* pImpl and C-linkage functions for the structs that Range support requires), or (2) we move everything out of aaA.d.

Personally, I feel the second option is better. If we want to improve or add to AA's API, we can just change it in object_.d. Key and value types are directly accessible, so we never have to play around with typeinfos and pointer arithmetic to do simple stuff.

If we go with the first option, every little change will require changes to aaA.d, and trying to add new functionality will constantly introduce new C-linkage functions in aaA.d, new void* pImpl's in object_.d, with the associated hacks using typeinfos (because Key/Value types are essentially opaque to aaA.d, so you have to rely on typeinfos and pointer arithmetic instead of letting the compiler figure it out for you). This is very hard to maintain, and much more bug-prone.


T

-- 
"I suspect the best way to deal with procrastination is to put off the procrastination itself until later. I've been meaning to try this, but haven't gotten around to it yet. " -- swr

On Wednesday, 14 March 2012 at 23:51:30 UTC, H. S. Teoh wrote:
> On Thu, Mar 15, 2012 at 12:20:43AM +0100, Jakob Bornecrantz wrote:
> [...]
>> struct AAver1(K, V)
>> {
>>    K[] tbl; V[] tlb2; uint size;
>> }
>> 
>> struct AAver2(K, V)
>> {
>>    K[] tbl; V[] tbl2; V[] optimizationTbl;
>> }
>> 
>> Would break if a AAver1 table was ever passed to code that
>> was compiled against a AAver2 table. In sort you could never
>> add anything to the AA struct. Without going in roundabout
>> ways of making sure you never access outside of any struct
>> version ever out there.
>> 
>> Or for that matter change how the internal tables are
>> populated by add and remove.
> [...]
>
> How is this different from any other templates in druntime/phobos?

Its not, I at least like to see it being possible to keep a
stable ABI in druntime, and as such making it possible to
share inbuilt language features like AA across libraries,
phobos is a lost cause.

>
> And FYI, the current AA implementation *already* suffers
> from this problem, because the Range interface already
> assumes a specific implementation behind the opaque pImpl
> pointer (see object_.d -- it *duplicates* the struct
> definitions from aaA.d and casts the void* into pointers to
> those structs). If aaA.d were to change its implementation
> today, the Range stuff in struct AssociativeArray would break horribly.
>
> The motivation behind my rewriting AA's is to fix this schizophrenic mess. The internal aaA.d structs should *not*
> be duplicated in object_.d, but currently they are. So there
> are two options, either (1) we move everything back into aaA.d
> (and introduce a whole bunch more void* pImpl and C-linkage
> functions for the structs that Range support requires), or
> (2) we move everything out of aaA.d.

Yes that is bad, I of course would like to see (2). I don't
feel that just because I wasn't around to hit people over
the head when this was introduced as reason for losing the
ability of D libraries, or making it even harder for people
to do them.

>
> Personally, I feel the second option is better. If we want
> to improve or add to AA's API, we can just change it in
> object_.d. Key and value types are directly accessible, so
> we never have to play around with typeinfos and pointer
> arithmetic to do simple stuff.
>
> If we go with the first option, every little change will
> require changes to aaA.d, and trying to add new functionality
> will constantly introduce new C-linkage functions in aaA.d,
> new void* pImpl's in object_.d, with the associated hacks
> using typeinfos (because Key/Value types are essentially
> opaque to aaA.d, so you have to rely on typeinfos and pointer
> arithmetic instead of letting the compiler figure it out for
> you). This is very hard to maintain, and much more bug-prone.

I'm assuming you mean "changes to aaA.d and _object.d".

I'm glad you listed cons of doing it as a opaque pointer,
I don't agree completely that it will be that much more
difficult to maintain or be that much more buggy.

Yes the current one of doing it in DMD, _object.d and in
druntime is bad and can be improved. Did you see my response
to Don? What do you think of that?

Yes there will be a cost to doing it the second way, but
we as a language provider are taking that cost so that
our users don't have to take it via being locked to the
exact same druntime/compiler version between libraries.

Yes doing a stable ABI isn't easy and free, so I think we
need to weigh the pros and cons of supporting language
level primitives easily (for users) across library
boundaries. I think we should just htfu[1] and provide
this feature.

I just wanted to make sure people understand what they are
loosing and what this means. And that they should't have
any illusions of things just magically working. I don't see
me changing my view on thism I want this feature, if I don't
get sure I'll be sad and D will be less attractive to me but
it isn't the end of the world. I want to thank you again for
taking the time to explain everything to me and taking the
time to respond.


What I really want to get to is: Do we really want to do this?


Cheers, Jakob.

[1] http://www.youtube.com/watch?v=unkIVvjZc9Y

On 15.03.2012 2:44, Jakob Bornecrantz wrote:
> On Wednesday, 14 March 2012 at 09:07:40 UTC, Dmitry Olshansky wrote:
>> On 14.03.2012 6:39, Jakob Bornecrantz wrote:
>>> On Wednesday, 14 March 2012 at 00:52:32 UTC, H. S. Teoh wrote:
>>>> Hi all,
>>>>
>>>> My AA implementation is slowly inching closer to being ready to
>>>> replace aaA.d. So far I've been writing the implementation
>>>> outside of object_.d for ease of testing & development; now I'm
>>>> ready to start moving stuff into object_.d to start working on
>>>> integration with druntime.
>>>
>>> Hi,
>>>
>>> If I'm understanding this correctly you are moving the entire
>>> implementation of the AA into object.d and as such letting
>>> programs be purview to its inner working? In sort meaning you
>>> are making the entire AA implementation D ABI locked.
>>>
>>> This will make it impossible to either change the AA
>>> implementation in any ABI breaking fashion or make it impossible
>>> to pass AA's between libraries compiled against different
>>> versions of druntime.
>>
>> I will just point out that the major point of ABI is to make sure
>> different D compiler produce compatible object code. Thus it makes AA
>> implementation locked already anyway, right?
>
> Not true, as Steven said a opaque pImpl implementation would
> work, most modern C library design follow this principle with
> only using opaque except for pointers. This is because the ABI
> will then only require you call a certain set of functions (that
> can be extended to) not what that pointers points to, the
> details can be completely hidden.
>

Trouble is when one dynamic lib uses one version of druntime and AA, and your app another one. Guess what happens with this opaque pointer?


-- 
Dmitry Olshansky

On 03/15/2012 09:40 AM, Dmitry Olshansky wrote:
> On 15.03.2012 2:44, Jakob Bornecrantz wrote:
>> Not true, as Steven said a opaque pImpl implementation would
>> work, most modern C library design follow this principle with
>> only using opaque except for pointers. This is because the ABI
>> will then only require you call a certain set of functions (that
>> can be extended to) not what that pointers points to, the
>> details can be completely hidden.
>>
>
> Trouble is when one dynamic lib uses one version of druntime and AA, and
> your app another one. Guess what happens with this opaque pointer?
>
>

AA should probably be a class with virtual methods.

On 15.03.2012 12:52, Timon Gehr wrote:
> On 03/15/2012 09:40 AM, Dmitry Olshansky wrote:
>> On 15.03.2012 2:44, Jakob Bornecrantz wrote:
>>> Not true, as Steven said a opaque pImpl implementation would
>>> work, most modern C library design follow this principle with
>>> only using opaque except for pointers. This is because the ABI
>>> will then only require you call a certain set of functions (that
>>> can be extended to) not what that pointers points to, the
>>> details can be completely hidden.
>>>
>>
>> Trouble is when one dynamic lib uses one version of druntime and AA, and
>> your app another one. Guess what happens with this opaque pointer?
>>
>>
>
> AA should probably be a class with virtual methods.

If its V-table layout is consistent and fixed in say ABI spec, it could work.

-- 
Dmitry Olshansky

On Thu, 15 Mar 2012 04:52:41 -0400, Timon Gehr <timon.gehr@gmx.ch> wrote:

> On 03/15/2012 09:40 AM, Dmitry Olshansky wrote:
>> On 15.03.2012 2:44, Jakob Bornecrantz wrote:
>>> Not true, as Steven said a opaque pImpl implementation would
>>> work, most modern C library design follow this principle with
>>> only using opaque except for pointers. This is because the ABI
>>> will then only require you call a certain set of functions (that
>>> can be extended to) not what that pointers points to, the
>>> details can be completely hidden.
>>>
>>
>> Trouble is when one dynamic lib uses one version of druntime and AA, and
>> your app another one. Guess what happens with this opaque pointer?
>>
>>
>
> AA should probably be a class with virtual methods.

First, that only works if the vtable layout doesn't change (I've had more trouble with binary compatibility because of this vs. binary compatibility from different function implementations in C++).

Second, that breaks a *LOT* of code which expects AA's to just be declared and used.

-Steve

On Wed, 14 Mar 2012 19:20:43 -0400, Jakob Bornecrantz <wallbraker@gmail.com> wrote:

> On Wednesday, 14 March 2012 at 14:02:30 UTC, Steven Schveighoffer wrote:
>>
>> This is unavoidable, whether it's a template or not.  What changes do you envision would be transparent using an opaque pImpl model (as was done in previous versions of phobos), but would break using templates?
>
> struct AAver1(K, V)
> {
>     K[] tbl; V[] tlb2; uint size;
> }
>
> struct AAver2(K, V)
> {
>     K[] tbl; V[] tbl2; V[] optimizationTbl;
> }
>
> Would break if a AAver1 table was ever passed to code that
> was compiled against a AAver2 table. In sort you could never
> add anything to the AA struct. Without going in roundabout
> ways of making sure you never access outside of any struct
> version ever out there.
>
> Or for that matter change how the internal tables are
> populated by add and remove.

So you are expecting druntime to be a .so/dll then.  When that happens, we can worry about this.  But right now, each dll gets it's own copy of druntime, so there still is no compatibility.

-Steve

On 03/15/2012 11:41 AM, Steven Schveighoffer wrote:
>
> Second, that breaks a *LOT* of code which expects AA's to just be
> declared and used.
>

Not necessarily. The compiler could still do the auto-initialization.

On Thu, 15 Mar 2012 13:28:00 -0400, Timon Gehr <timon.gehr@gmx.ch> wrote:

> On 03/15/2012 11:41 AM, Steven Schveighoffer wrote:
>>
>> Second, that breaks a *LOT* of code which expects AA's to just be
>> declared and used.
>>
>
> Not necessarily. The compiler could still do the auto-initialization.

Auto initialization is done by the type, not the compiler (even in previous implementation that did not use template wrapper).  AFAIK, classes can't initialize themselves.

You could make the pImpl a class.  And actually, this might be a good thing.  But the basic AA type should be a struct.

-Steve