June 28, 2005
"Regan Heath" <regan@netwin.co.nz> wrote in message news:opss2fh7pw23k2f5@nrage.netwin.co.nz...
> On Mon, 27 Jun 2005 21:31:18 -0700, Andrew Fedoniouk <news@terrainformatica.com> wrote:
>> And about yours q = new AutoPtr(p);
>> It does not call destructor of q on assignment
>
> True, In D the destructor is only called when the object is collected, or you call "delete".
>
>> so what is the point of this AutoPtr(p).
>> 'Auto' means that it manages pointer for
>> me.
>
> Well, it seems to me that one solution would be for an 'auto' class to be deleted the moment it is no longer referenced. (rather than waiting till the end of the scope). Not sure if that is possible, but it makes sense.
>
> So, that just leaves the problem of preventing a normal reference assignment, got any ideas?
>
>>>>> Notice you say "value" above. You do not assign values to reference,
>>>>> you
>>>>> assign references to references, thus opAssign has no place in
>>>>> classes.
>>>>> (I
>>>>> believe we've already agreed to this point elsewhere).
>>>>
>>>> Yes. This is why I am using struct in my example.
>>>> For classes '=' is copying references.
>>>> For sructures it is copying values.
>>>> The problem is that for structures '='
>>>> silently overrides values without any control
>>>> from your side.
>>>
>>> Structs also do not have ctor/dtor.
>>> Classes are the only option for AutoPtr IMO.
>>
>> Without control of assignment and ctor/dtor
>> it is impossible to implement the whole class of
>> smart pointers and such thing as COW string.
>
> Perhaps, but that does not mean we require opAssign. There may be another way.
>
>> We agreed that opAssign in classes makes no
>> sense. So only structs left.
>
> We could still use classes without using opAssign. This is where I have concentrated my efforts since reading the thread you found from back in 2002 where they decide on using 'auto'.
>
>> About real COW strings and motivation: http://www.wxwidgets.org/manuals/2.6.0/wx_wxstringoverview.html#wxstringrefcount
>>
>> It is impossible to reproduce in D such class
>> and any other class based on pointer management
>> paradigm.
>
> It's currently impossible to reproduce classes that work in 'exactly' the same way. The D solution does not have to be exactly the same, provided it still achieve the goals.
>
>> I mean to define string class which
>> will behave as a string type and not just a set
>> of C style functions dealing with
>> struct slice { char* ptr; size_t length; }.
>
> What would this string class do that char[] cannot?
>
> If your only motivation is to provide a way of passing 'const' string data then I think enforcing readonly for "const" and "in" is much more productive and would also work for other types in D.
>
>> Again, it depends on author motivation.
>>
>> As far as I can see there are three features
>> that C++ has and D doesn't have:
>> struct(opAssign/ctor/dtor) and const.
>>
>> If they will be there then D may be considered
>> as a successor of C++. With all possible
>> implications.
>
> You have to remember that D is different to C++ and will therefore solve the same problems in different ways. I'm looking for a D solution. You seem to be asking for D to become C++. I agree D cannot solve these problems, yet, but the best way to fix that is to think of a way it could solve them *without* changing some of the fundamental design decisions that make D what it is.
>

Sigh... Round came to the end. The cycle is closed.

I've started screaming about opAssign after I realized
that const arrays/pointers are persons non grata in D for
some ideological reasons. So idea was to have features in language
allowing to implement const types by myself. That is why
opAssign and co.

Again, currently there are no solutions in D
for strings and smart pointers. I mean real and natural solutions.

--------------------
And about string and char[]
char[] is a slice - fragment of character sequence and not a string.
String variable represent/manage storage of textual data.
In contrary char[] is just a poitner bundled with length.
It is not more than char* - it does not have a concept
of ownership - who and when can or cannot modify a string.

BTW: char* and char[] are different in principle:
in char* length of string is part of data (zero terminated).
and in char[] it is part of particular variable. This
is also why char[] is not a string.

char[] can be used as a string field in some structures only if it *always* points to some *unique* memory location - owns it. So you can *safely* manipulate owning text.

By itself char[] is convenient way to define and pass slices
to functions - temporary (short living) structures.
But here usability is limited by lack of const too. Sigh...

string class is a COW manager of character data. Using it you
don't need to invent COW algorithms each time when you are dealing
with text and to account what source this text came from - it significantly
reduces complexity of code, increases robustness of system in general
and uses memory in most optimal way.

Andrew.


June 28, 2005
<SNIP>
> 
> Sigh... Round came to the end. The cycle is closed.
> 
> I've started screaming about opAssign after I realized
> that const arrays/pointers are persons non grata in D for
> some ideological reasons. So idea was to have features in language
> allowing to implement const types by myself. That is why
> opAssign and co.
> 

<snip>

> Andrew. 
> 
> 

This topic, and many like them, have been kicked around for quite a while now.  I would like to hear what Walter has to say on all this. Walter - do you have particular reasons why opAssign, or "const" can't/won't/shouldn't be implemented in D?
To me it looks like Regan and Andrew (and many others) both want the same thing in the end - Regan is trying to do it within the confines of the current language (and lets face it, not having an easy time of it, which should tell us something), and Andrew has concluded that it is impossible without changes to D.

So, what do you say?

Are Andrew and Regan trying to do something that isn't required in D? Or is D lacking important functionality?

Cheers
Brad
June 28, 2005
On Tue, 28 Jun 2005 11:20:36 -0700, Brad Beveridge <brad@somewhere.net> wrote:
> <SNIP>
>>  Sigh... Round came to the end. The cycle is closed.
>>  I've started screaming about opAssign after I realized
>> that const arrays/pointers are persons non grata in D for
>> some ideological reasons. So idea was to have features in language
>> allowing to implement const types by myself. That is why
>> opAssign and co.
>>
>
> <snip>
>
>> Andrew.
>
> This topic, and many like them, have been kicked around for quite a while now.  I would like to hear what Walter has to say on all this. Walter - do you have particular reasons why opAssign, or "const" can't/won't/shouldn't be implemented in D?
> To me it looks like Regan and Andrew (and many others) both want the same thing in the end - Regan is trying to do it within the confines of the current language (and lets face it, not having an easy time of it, which should tell us something), and Andrew has concluded that it is impossible without changes to D.
>
> So, what do you say?
>
> Are Andrew and Regan trying to do something that isn't required in D? Or is D lacking important functionality?

Good post.

Regan
June 28, 2005
On Tue, 28 Jun 2005 11:08:49 -0700, Andrew Fedoniouk <news@terrainformatica.com> wrote:
>> You have to remember that D is different to C++ and will therefore solve
>> the same problems in different ways. I'm looking for a D solution. You
>> seem to be asking for D to become C++. I agree D cannot solve these
>> problems, yet, but the best way to fix that is to think of a way it could
>> solve them *without* changing some of the fundamental design decisions
>> that make D what it is.
>>
>
> Sigh... Round came to the end. The cycle is closed.
>
> I've started screaming about opAssign after I realized
> that const arrays/pointers are persons non grata in D for
> some ideological reasons. So idea was to have features in language
> allowing to implement const types by myself. That is why
> opAssign and co.
>
> Again, currently there are no solutions in D
> for strings and smart pointers. I mean real and natural solutions.

I agree.

> --------------------
> And about string and char[]
> char[] is a slice - fragment of character sequence and not a string.
> String variable represent/manage storage of textual data.

I still don't see the difference. A char[] represents textual data, a char manages storage of textual data (it has a length you can set, it handles concatenation, with const/in protection it would handle ownership)

> In contrary char[] is just a poitner bundled with length.
> It is not more than char* - it does not have a concept
> of ownership - who and when can or cannot modify a string.

It would if we had readonly protection for 'const' and 'in' thought, right?

> BTW: char* and char[] are different in principle:
> in char* length of string is part of data (zero terminated).

I know, I have a C background.

> and in char[] it is part of particular variable. This
> is also why char[] is not a string.

So char* is a string but char[] isn't? That statement seems odd to me.

> char[] can be used as a string field in some structures only
> if it *always* points to some *unique* memory location -
> owns it. So you can *safely* manipulate owning text.

You are supposed to use COW. I agree without some sort of const/in protection you're left guessing whether you own it. With const/in protection you *know* whether you own a string, if it's const or passed as 'in' you dont, otherwise you do, right?

> By itself char[] is convenient way to define and pass slices
> to functions - temporary (short living) structures.
> But here usability is limited by lack of const too. Sigh...

Sure, we need const/in protection. We both agree there.

> string class is a COW manager of character data.

Aha! So this is the definition you're using. I believe with const/in protection char[] can be used with COW safely and with surety (no guessing).

> Using it you
> don't need to invent COW algorithms each time when you are dealing
> with text and to account what source this text came from - it significantly
> reduces complexity of code, increases robustness of system in general
> and uses memory in most optimal way.

Agreed. We need something to help us use COW without guessing or adding complicated code to handle it.

In short, if char[] had const/in protection do you agree that it would be the 'string class' you require? If not, what other factors are missing? And would those factors be handled if we could design and use an AutoPtr to a char[]?

Regan.
June 29, 2005
Brad Beveridge wrote:

> So, what do you say?
> 
> Are Andrew and Regan trying to do something that isn't required in D?

Maybe isn't required if there was some other way to protect the strings.

> Or is D lacking important functionality?

Yes, as most have pointed out by now: D is sorely lacking a "readonly" !


This makes everyone look for workarounds like unsafe character arrays
(just like in old C) or string wrapper classes (just like in Java)...

Instead of having a "readonly char[]" parameter ?
(old name "const" changed, for political reasons)


My 2 öre,
--anders
June 29, 2005
Regan Heath wrote:

> I got little or no comment on this idea/code I posted to another (rather  large) thread here, I figured I'd post it again and see what people  thought.
> 
> After reading this thread (the one I mention above) and also this one  Andrew dug up:
>   http://www.digitalmars.com/d/archives/7988.html
> 
> It appears you need a type that:
> 
> 1 - is stack based.
> 2 - has deterministic destruction.
> 3 - has opAssign.
> 
> It seems (from recent posts) that Walter plans to put auto classes on the  stack, so they'd then fulfil requirements 1 and 2.
> 
> But what about 3? To me, opAssign isn't generally a good idea for classes  (assigning a value to a reference seems illogical) but would be a good for  structs (they're value types, assignment assigns a value).
> 
> Can you do reference counting without opAssign?
> 
<snip>
> Regan

I thought the stack based types were to be useful for speed gains, not managing memory.

Why do you need reference counting?  Why not just let the gc deallocator handle it?  Are you running into a problem running out of memory? Deallocation should not be such a central concept, otherwise we might as well ditch the gc.

In your threads example, you probably would use a manager to watch the threads, not rely on the runtime to tell you when to free one.

-DavidM

June 29, 2005
On Wed, 29 Jun 2005 09:41:01 -0400, David Medlock <noone@nowhere.com> wrote:
> Regan Heath wrote:
>
>> I got little or no comment on this idea/code I posted to another (rather  large) thread here, I figured I'd post it again and see what people  thought.
>>  After reading this thread (the one I mention above) and also this one  Andrew dug up:
>>   http://www.digitalmars.com/d/archives/7988.html
>>  It appears you need a type that:
>>  1 - is stack based.
>> 2 - has deterministic destruction.
>> 3 - has opAssign.
>>  It seems (from recent posts) that Walter plans to put auto classes on the  stack, so they'd then fulfil requirements 1 and 2.
>>  But what about 3? To me, opAssign isn't generally a good idea for classes  (assigning a value to a reference seems illogical) but would be a good for  structs (they're value types, assignment assigns a value).
>>  Can you do reference counting without opAssign?
>>
> <snip>
>> Regan
>
> I thought the stack based types were to be useful for speed gains, not managing memory.

Correct. We need/want the reference counter or auto pointer to be as fast as possible because they're typically instantiated and destroyed with abandon. Andrew posted a good example of this, where his graphics library needs to obtain ownership of a graphics context on every call to the 'paint' method, it needs to get it, use it and release it, this function is called very often.

> Why do you need reference counting?

I don't, yet. Andrew does, now.

> Why not just let the gc deallocator handle it?

Because the problem we're trying to solve is 'ownership' and 'efficiency of memory'. In other words knowing when you need to copy a char[] not just guessing. At the moment people are likely copying to be safe, meaning a bunch of memory allocations you dont actually need.

IMO we can solve these issues by enforcing readonly for 'const/readonly/whatever' and 'in parameters', eg.

//this class returns readonly access to a string it owns
class Foo {
  char[] ownedByFoo;
  readonly char[] getIt() {
    return ownedByFoo[0..5];
  }
}

void mutate(char[] arr) {
  arr[0] = 'b'; //err 'arr' is readonly
}

void mutate2(inout char[] arr) {
  arr[0] = 'b';
}

Foo f = new Foo();
char[] p = f.getIt(); //'readonly' not required**
char[] s = p.dup; //dup does not copy readonly flag
p[0] = 'a'; //error p is readonly
s[0] = 'a'; //ok
mutate(p);  //ok, function def is actually in error
mutate(s);  //ok
mutate2(p); //error p is readonly
mutate2(s); //ok

** on variable decls (it's _not_ a seperate type, it's the _same_ type with a readonly bit flagged).

All types could have a readonly bit. Ideally only during compile, i.e. the compiler does a pass where it verifies no variable it has previously flagged as readonly (does the flagging during it's normal existing passes) has been passed as a mutable variable, or modified with inbuilt operator etc.

> Are you running into a problem running out of memory?

No. It's all about ownership and not having to guess when COW should be actioned. A reference counted string knows it's the only reference thus does not need to dup. An autopointer knows whether it's the owner and if not does a dup. Currently char[] has no such knowledge.

> Deallocation should not be such a central concept, otherwise we might as well ditch the gc.

Of course not, however for certain types you need to action something as they die, refcnt, autoptr and classes with resources to release. 'auto' seems to handle these cases.

A remaining problem is that refcnt and autoptr need to fast, so we want them on the stack. Walter has mentioned 'auto' classes will eventually be on the stack, yay.

The last issue is that it need to be illegal to assign an RefCnt to another RefCnt without increasing the reference count, i.e.

RefCnt p = new RefCnt(object);
RefCnt s = p; //should either be illegal, or should increment refcnt.

This is currently impossible in D.

> In your threads example, you probably would use a manager to watch the threads, not rely on the runtime to tell you when to free one.

A manager? you mean a whole other thread just to watch the first 10 threads?

I've since decided it's just simpler to store the whole lot of them in an array, at least then I know the references cant be null (which was the problem with initial code - not handling null case).

You can add null handling, and assume null means terminated and the initial code functions correctly.


All in all, if you read the entire thread, you'll come to Brad's post which sums the situation up nicely. We need some direction from Walter, either:

 - You dont need those in D (and reasoning)
 - Heres how we do it in D  (example)
 - I see, I will think about soln
 - Ok you're right I will add x,y and z

Regan
June 30, 2005
"Regan Heath" <regan@netwin.co.nz> wrote in message news:opss5jwkvj23k2f5@nrage.netwin.co.nz...
> On Wed, 29 Jun 2005 09:41:01 -0400, David Medlock <noone@nowhere.com> wrote:
>> Regan Heath wrote:
>>
>>> I got little or no comment on this idea/code I posted to another
>>> (rather  large) thread here, I figured I'd post it again and see what
>>> people  thought.
>>>  After reading this thread (the one I mention above) and also this one
>>> Andrew dug up:
>>>   http://www.digitalmars.com/d/archives/7988.html
>>>  It appears you need a type that:
>>>  1 - is stack based.
>>> 2 - has deterministic destruction.
>>> 3 - has opAssign.
>>>  It seems (from recent posts) that Walter plans to put auto classes on
>>> the  stack, so they'd then fulfil requirements 1 and 2.
>>>  But what about 3? To me, opAssign isn't generally a good idea for
>>> classes  (assigning a value to a reference seems illogical) but would
>>> be a good for  structs (they're value types, assignment assigns a
>>> value).
>>>  Can you do reference counting without opAssign?
>>>
>> <snip>
>>> Regan
>>
>> I thought the stack based types were to be useful for speed gains, not managing memory.
>
> Correct. We need/want the reference counter or auto pointer to be as fast as possible because they're typically instantiated and destroyed with abandon. Andrew posted a good example of this, where his graphics library needs to obtain ownership of a graphics context on every call to the 'paint' method, it needs to get it, use it and release it, this function is called very often.
>
>> Why do you need reference counting?
>
> I don't, yet. Andrew does, now.

Reference counting is one of most famous design patterns.
RC as a memory management has its own limitations though ( e.g. cyclic
refs ).
Its power in those areas where GC is weak. And vice versa.
So be able to use both will benefit the whole system twice.

Good example of RC is in implementations of COW strings and shared_ptr in C++.

>
>> Why not just let the gc deallocator handle it?
>
> Because the problem we're trying to solve is 'ownership' and 'efficiency of memory'. In other words knowing when you need to copy a char[] not just guessing. At the moment people are likely copying to be safe, meaning a bunch of memory allocations you dont actually need.
>
> IMO we can solve these issues by enforcing readonly for 'const/readonly/whatever' and 'in parameters', eg.

Pretty much yes.

>
> //this class returns readonly access to a string it owns
> class Foo {
>   char[] ownedByFoo;
>   readonly char[] getIt() {
>     return ownedByFoo[0..5];
>   }
> }
>
> void mutate(char[] arr) {
>   arr[0] = 'b'; //err 'arr' is readonly
> }
>
> void mutate2(inout char[] arr) {
>   arr[0] = 'b';
> }
>
> Foo f = new Foo();
> char[] p = f.getIt(); //'readonly' not required**
> char[] s = p.dup; //dup does not copy readonly flag
> p[0] = 'a'; //error p is readonly
> s[0] = 'a'; //ok
> mutate(p);  //ok, function def is actually in error
> mutate(s);  //ok
> mutate2(p); //error p is readonly
> mutate2(s); //ok
>
> ** on variable decls (it's _not_ a seperate type, it's the _same_ type with a readonly bit flagged).
>
> All types could have a readonly bit. Ideally only during compile, i.e. the compiler does a pass where it verifies no variable it has previously flagged as readonly (does the flagging during it's normal existing passes) has been passed as a mutable variable, or modified with inbuilt operator etc.
>
>> Are you running into a problem running out of memory?
>
> No. It's all about ownership and not having to guess when COW should be actioned. A reference counted string knows it's the only reference thus does not need to dup. An autopointer knows whether it's the owner and if not does a dup. Currently char[] has no such knowledge.
>
>> Deallocation should not be such a central concept, otherwise we might as well ditch the gc.
>
> Of course not, however for certain types you need to action something as they die, refcnt, autoptr and classes with resources to release. 'auto' seems to handle these cases.
>
> A remaining problem is that refcnt and autoptr need to fast, so we want them on the stack. Walter has mentioned 'auto' classes will eventually be on the stack, yay.

Exactly!

>
> The last issue is that it need to be illegal to assign an RefCnt to another RefCnt without increasing the reference count, i.e.
>
> RefCnt p = new RefCnt(object);
> RefCnt s = p; //should either be illegal, or should increment refcnt.
>
> This is currently impossible in D.

Sigh....

>
>> In your threads example, you probably would use a manager to watch the threads, not rely on the runtime to tell you when to free one.
>
> A manager? you mean a whole other thread just to watch the first 10 threads?
>
> I've since decided it's just simpler to store the whole lot of them in an array, at least then I know the references cant be null (which was the problem with initial code - not handling null case).
>
> You can add null handling, and assume null means terminated and the initial code functions correctly.
>
>
> All in all, if you read the entire thread, you'll come to Brad's post which sums the situation up nicely. We need some direction from Walter, either:
>
>  - You dont need those in D (and reasoning)
>  - Heres how we do it in D  (example)
>  - I see, I will think about soln
>  - Ok you're right I will add x,y and z
>
> Regan


June 30, 2005
Regan Heath wrote:
> I got little or no comment on this idea/code I posted to another (rather  large) thread here, I figured I'd post it again and see what people  thought.
> 
> After reading this thread (the one I mention above) and also this one  Andrew dug up:
>   http://www.digitalmars.com/d/archives/7988.html
> 
> It appears you need a type that:
> 
> 1 - is stack based.
> 2 - has deterministic destruction.
> 3 - has opAssign.

You can give a struct or class a method called opAssign if you want. Whether

    qwert = yuiop;

will act as syntactic sugar for it is another matter.

> It seems (from recent posts) that Walter plans to put auto classes on the  stack, so they'd then fulfil requirements 1 and 2.
> 
> But what about 3? To me, opAssign isn't generally a good idea for classes  (assigning a value to a reference seems illogical) but would be a good for  structs (they're value types, assignment assigns a value).
> 
> Can you do reference counting without opAssign?
<snip>

Yes.

Of the claims by various people that we "need" overloading of the assignment operator, I'm yet to see one that's true.  AFAIC if you want something to the effect of custom assignment behaviour, you can create your own method with whatever name that does it.  Can anyone supply an example, let alone a real-world one, that doesn't work unless the aforementioned syntactic sugar is available?

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on the 'group where everyone may benefit.
June 30, 2005
On Thu, 30 Jun 2005 11:19:04 +0100, Stewart Gordon <smjg_1998@yahoo.com> wrote:
> Regan Heath wrote:
>> I got little or no comment on this idea/code I posted to another (rather  large) thread here, I figured I'd post it again and see what people  thought.
>>  After reading this thread (the one I mention above) and also this one  Andrew dug up:
>>   http://www.digitalmars.com/d/archives/7988.html
>>  It appears you need a type that:
>>  1 - is stack based.
>> 2 - has deterministic destruction.
>> 3 - has opAssign.
>
> You can give a struct or class a method called opAssign if you want. Whether
>
>      qwert = yuiop;
>
> will act as syntactic sugar for it is another matter.

True, but not the problem.. read on :)

>> It seems (from recent posts) that Walter plans to put auto classes on the  stack, so they'd then fulfil requirements 1 and 2.
>>  But what about 3? To me, opAssign isn't generally a good idea for classes  (assigning a value to a reference seems illogical) but would be a good for  structs (they're value types, assignment assigns a value).
>>  Can you do reference counting without opAssign?
> <snip>
>
> Yes.

Then please enlighten me. I've tried, it 'almost' works but it suffers the fatal flaw shown below.

> Of the claims by various people that we "need" overloading of the assignment operator, I'm yet to see one that's true. AFAIC if you want something to the effect of custom assignment behaviour, you can create your own method with whatever name that does it.  Can anyone supply an example, let alone a real-world one, that doesn't work unless the aforementioned syntactic sugar is available?

The problem, as I see it, is that it's not just syntactic sugar. You cannot prevent reference assignment in cases where the idea/technique requires it. As in these 2 cases. (reference counting, auto pointer)

I've tried to write a reference counting implementation and also an auto pointer implementation using the recommended practices found on the D website. Both work, both use a "copy constructor" instead of opAssign (as suggested by the D spec/website). The problem is this:

auto AutoPtr p = new AutoPtr(object); //create ptr passing object, p owns object. (or "pwns" for those of you..)
auto AutoPtr s;

s = new AutoPtr(p); //no problem, the copy constructor transfers ownership.
s = p; //problem, s now refers to p, both 'own' "object".

The same problem occurs with the reference counting implementation. s becomes a 2nd reference to object but the reference count does not increase.

We can 'ask' people not to use a straight reference assignment but we cannot enforce it.

Logically I agree, opAssign makes no sense for reference types, you assign references to reference types *not* values. opAssign would effectively change a reference assignment into a value assignment in a non obvious way. I agree with all this reasoning, however, the problem remains.

Regan