Currently, there is a problem in the runtime which can result in very odd behavior.  Let's say you declare a class like this:

class C
{
   int[1] x;
}

Now, let's say you do something like this:

auto c = new C;
auto x = c.x[];
x ~= 1;

What happens here?  Well, the memory for c and  c.x are on the heap, so the block allocated by c is considered for appending, and a "length" field is looked at, even though that length is possibly garbage.  The result is that it's extremely improbable, but possible, that the append could happen in place if that "length" happens to be correct (thereby overwriting other members of c).  I can't even begin to construct a case which shows this is possible, and it may not even be, but I think this needs attention.

In addition, if anyone allocates memory via GC.malloc, and then tries to append to such memory, a similar problem occurs, because the allocation did not go through the array allocation routine.  This is more provable, but less likely to happen (most people don't call GC.malloc).  However, a place where I am most concerned is closures, which I think call GC.malloc.

What I propose to fix this is to allocate one of the block attribute flags to designate that a block is appendable.  The odd part about it, is that the GC is not aware of appending, so it is somewhat of a "user-defined" flag.

My question is, how should we declare the flag?  I currently only defined it in lifetime.d, outside the gc, but I'm thinking it should be defined in the GC as well.  Also, should I allocate the next bit, or one of the higher order bits?

I'm going to check in my code that defines it in lifetime.d only, and depending on the responses to this, I'll add it to the GC definition too.

-Steve

I think the flag would be a must for SafeD, so we may as well add it now.

Andrei

On 07/13/2010 10:09 AM, Steve Schveighoffer wrote:
> Currently, there is a problem in the runtime which can result in very odd behavior.  Let's say you declare a class like this:
>
> class C
> {
>     int[1] x;
> }
>
> Now, let's say you do something like this:
>
> auto c = new C;
> auto x = c.x[];
> x ~= 1;
>
> What happens here?  Well, the memory for c and  c.x are on the heap, so the block allocated by c is considered for appending, and a "length" field is looked at, even though that length is possibly garbage.  The result is that it's extremely improbable, but possible, that the append could happen in place if that "length" happens to be correct (thereby overwriting other members of c).  I can't even begin to construct a case which shows this is possible, and it may not even be, but I think this needs attention.
>
> In addition, if anyone allocates memory via GC.malloc, and then tries to append to such memory, a similar problem occurs, because the allocation did not go through the array allocation routine.  This is more provable, but less likely to happen (most people don't call GC.malloc).  However, a place where I am most concerned is closures, which I think call GC.malloc.
>
> What I propose to fix this is to allocate one of the block attribute flags to designate that a block is appendable.  The odd part about it, is that the GC is not aware of appending, so it is somewhat of a "user-defined" flag.
>
> My question is, how should we declare the flag?  I currently only defined it in lifetime.d, outside the gc, but I'm thinking it should be defined in the GC as well.  Also, should I allocate the next bit, or one of the higher order bits?
>
> I'm going to check in my code that defines it in lifetime.d only, and depending on the responses to this, I'll add it to the GC definition too.
>
> -Steve
>
>
>
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Steve Schveighoffer wrote:
> Currently, there is a problem in the runtime which can result in very odd behavior.  Let's say you declare a class like this:
>
> class C
> {
>    int[1] x;
> }
>
> Now, let's say you do something like this:
>
> auto c = new C;
> auto x = c.x[];
> x ~= 1;
>
> What happens here?  Well, the memory for c and  c.x are on the heap, so the block allocated by c is considered for appending, and a "length" field is looked at, even though that length is possibly garbage.  The result is that it's extremely improbable, but possible, that the append could happen in place if that "length" happens to be correct (thereby overwriting other members of c).  I can't even begin to construct a case which shows this is possible, and it may not even be, but I think this needs attention.
>
> 

It will never happen, as c.x[1] is not at the beginning of an allocated block.

----- Original Message ----
> From: Walter Bright <walter at digitalmars.com>
> 
> 
> 
> Steve Schveighoffer wrote:
> > Currently, there is a problem in the  runtime which can result in very odd
>behavior.  Let's say you declare a  class like this:
> > 
> > class C
> > {
> >    int[1]  x;
> > }
> > 
> > Now, let's say you do something like this:
> > 
> > auto c = new C;
> > auto x = c.x[];
> > x ~= 1;
> > 
> > What happens here?  Well, the memory for c and  c.x are on  the heap, so the
>block allocated by c is considered for appending, and a  "length" field is looked at, even though that length is possibly garbage.   The result is that it's extremely improbable, but possible, that the append  could happen in place if that "length" happens to be correct (thereby  overwriting other members of c).  I can't even begin to construct a case  which shows this is possible, and it may not even be, but I think this needs  attention.
> > 
> > 
> 
> It will never happen, as c.x[1] is  not at the beginning of an allocated
>block.

That is no longer a requirement with the new array append functionality.  All that is required is that the end of an array is at the end of the "allocated" data (that is, data that has been requested from the array append routines).

-Steve

The GC is going to need a bit of an overhaul to support finalization of structs anyway. Maybe this could be another use for the stored typeinfo.

Sent from my iPhone

On Jul 13, 2010, at 8:09 AM, Steve Schveighoffer <schveiguy at yahoo.com> wrote:

> Currently, there is a problem in the runtime which can result in very odd behavior.  Let's say you declare a class like this:
> 
> class C
> {
>   int[1] x;
> }
> 
> Now, let's say you do something like this:
> 
> auto c = new C;
> auto x = c.x[];
> x ~= 1;
> 
> What happens here?  Well, the memory for c and  c.x are on the heap, so the block allocated by c is considered for appending, and a "length" field is looked at, even though that length is possibly garbage.  The result is that it's extremely improbable, but possible, that the append could happen in place if that "length" happens to be correct (thereby overwriting other members of c).  I can't even begin to construct a case which shows this is possible, and it may not even be, but I think this needs attention.
> 
> In addition, if anyone allocates memory via GC.malloc, and then tries to append to such memory, a similar problem occurs, because the allocation did not go through the array allocation routine.  This is more provable, but less likely to happen (most people don't call GC.malloc).  However, a place where I am most concerned is closures, which I think call GC.malloc.
> 
> What I propose to fix this is to allocate one of the block attribute flags to designate that a block is appendable.  The odd part about it, is that the GC is not aware of appending, so it is somewhat of a "user-defined" flag.
> 
> My question is, how should we declare the flag?  I currently only defined it in lifetime.d, outside the gc, but I'm thinking it should be defined in the GC as well.  Also, should I allocate the next bit, or one of the higher order bits?
> 
> I'm going to check in my code that defines it in lifetime.d only, and depending on the responses to this, I'll add it to the GC definition too.
> 
> -Steve
> 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Idea: you can easily accommodate allocation of structs by wrapping them in a class:

S * allocateStruct(S)()
{
     class Classify { S payload_; }
     auto result = new Classify;
     return &result.payload_;
}

That works with the existing GC which knows how to collect classes.

Another, cheaper variant which assumes arrays are GCd properly:

S * allocateStruct(S)()
{
     return (new S[1]).ptr;
}


Andrei

On 07/13/2010 07:21 PM, Sean Kelly wrote:
> The GC is going to need a bit of an overhaul to support finalization of structs anyway. Maybe this could be another use for the stored typeinfo.
>
> Sent from my iPhone
>
> On Jul 13, 2010, at 8:09 AM, Steve Schveighoffer<schveiguy at yahoo.com>  wrote:
>
>> Currently, there is a problem in the runtime which can result in very odd behavior.  Let's say you declare a class like this:
>>
>> class C
>> {
>>    int[1] x;
>> }
>>
>> Now, let's say you do something like this:
>>
>> auto c = new C;
>> auto x = c.x[];
>> x ~= 1;
>>
>> What happens here?  Well, the memory for c and  c.x are on the heap, so the block allocated by c is considered for appending, and a "length" field is looked at, even though that length is possibly garbage.  The result is that it's extremely improbable, but possible, that the append could happen in place if that "length" happens to be correct (thereby overwriting other members of c).  I can't even begin to construct a case which shows this is possible, and it may not even be, but I think this needs attention.
>>
>> In addition, if anyone allocates memory via GC.malloc, and then tries to append to such memory, a similar problem occurs, because the allocation did not go through the array allocation routine.  This is more provable, but less likely to happen (most people don't call GC.malloc).  However, a place where I am most concerned is closures, which I think call GC.malloc.
>>
>> What I propose to fix this is to allocate one of the block attribute flags to designate that a block is appendable.  The odd part about it, is that the GC is not aware of appending, so it is somewhat of a "user-defined" flag.
>>
>> My question is, how should we declare the flag?  I currently only defined it in lifetime.d, outside the gc, but I'm thinking it should be defined in the GC as well.  Also, should I allocate the next bit, or one of the higher order bits?
>>
>> I'm going to check in my code that defines it in lifetime.d only, and depending on the responses to this, I'll add it to the GC definition too.
>>
>> -Steve
>>
>>
>>
>>
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

On Jul 13, 2010, at 8:09 AM, Steve Schveighoffer wrote:
> 
> What I propose to fix this is to allocate one of the block attribute flags to designate that a block is appendable.  The odd part about it, is that the GC is not aware of appending, so it is somewhat of a "user-defined" flag.
> 
> My question is, how should we declare the flag?  I currently only defined it in lifetime.d, outside the gc, but I'm thinking it should be defined in the GC as well.  Also, should I allocate the next bit, or one of the higher order bits?
> 
> I'm going to check in my code that defines it in lifetime.d only, and depending on the responses to this, I'll add it to the GC definition too.

Your checkin doesn't compile.  I added the requisite comma for now to fix this.  That aside, assuming this design were to be kept I think the meaning of the bit has to be flipped.  The bit defaults to unset, so that should be the default meaning.  You really don't want all blocks to be appendable by default, do you?  More generally, if we start holding TypeInfo references in the GC then we should be able to determine appendabiility based on whether the TypeInfo is an array type.  This may not make the next release, but I'd like to address it once I've polished std.concurrency a bit more.

No, because typeinfo has nothing to do with whether you allocated a block of memory using an array allocation routine or GC.malloc.

I need a bit.  I've used the next one in line, hopefully that's OK (the code is already checked in).

-Steve



----- Original Message ----
> From: Sean Kelly <sean at invisibleduck.org>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Cc: Phobos <phobos at puremagic.com>
> Sent: Tue, July 13, 2010 8:21:45 PM
> Subject: Re: [phobos] custom BlkAttr flags
> 
> The GC is going to need a bit of an overhaul to support finalization of structs
>anyway. Maybe this could be another use for the stored typeinfo.
>
> 
> Sent  from my iPhone
> 
> On Jul 13, 2010, at 8:09 AM, Steve Schveighoffer <schveiguy at yahoo.com> wrote:
> 
> >  Currently, there is a problem in the runtime which can result in very odd
> > behavior.  Let's say you declare a class like this:
> > 
> > class C
> > {
> >   int[1] x;
> > }
> > 
> >  Now, let's say you do something like this:
> > 
> > auto c = new  C;
> > auto x = c.x[];
> > x ~= 1;
> > 
> > What happens  here?  Well, the memory for c and  c.x are on the heap, so the

> > block allocated by c is considered for appending, and a "length" field  is
>looked
>
> > at, even though that length is possibly garbage.  The  result is that it's extremely improbable, but possible, that the append  could happen in place if
>
> > that "length" happens to be correct (thereby  overwriting other members of
>c).  I
>
> > can't even begin to construct  a case which shows this is possible, and it
>may
>
> > not even be, but I  think this needs attention.
> > 
> > In addition, if anyone allocates  memory via GC.malloc, and then tries to
>append
>
> > to such memory, a  similar problem occurs, because the allocation did not go

> > through the  array allocation routine.  This is more provable, but less
>likely to
>
> > happen (most people don't call GC.malloc).  However, a place where  I am most
>
> > concerned is closures, which I think call GC.malloc.
> > 
> > What I propose to fix this is to allocate one of the block attribute  flags
>to
>
> > designate that a block is appendable.  The odd part about  it, is that the GC
>is
>
> > not aware of appending, so it is somewhat of a  "user-defined" flag.
> > 
> > My question is, how should we declare the  flag?  I currently only defined it
>in
>
> > lifetime.d, outside the gc,  but I'm thinking it should be defined in the GC
>as
>
> > well.  Also,  should I allocate the next bit, or one of the higher order
>bits?
> > 
> > I'm going to check in my code that defines it in lifetime.d only, and
>depending
>
> > on the responses to this, I'll add it to the GC definition  too.
> > 
> > -Steve
> > 
> > 
> > 
> > 
> >  _______________________________________________
> > phobos mailing  list
> > phobos at puremagic.com
> >  http://lists.puremagic.com/mailman/listinfo/phobos
> _______________________________________________
> phobos  mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
> 

I added the fix for the comma, sorry for that.  All the other places had an ALL_BITS enum member, so the NO_MOVE line that I copied and changed had a comma.  I thought I compiled before I checked in, but I guess I did not.

Regarding the bit being flipped, yes I want the default meaning to be "no append" because anyone can allocate memory via GC.malloc for any purpose, and GC.malloc does not initialize the "allocated length" field inside the block, so it is not appendable.  It does not mean you cannot append to the block, it just means the first append will reallocate into an appendable block.

Regarding using the typeinfo, that doesn't work.  Arrays can easily be created out of blocks that weren't allocated via the array creation routines.  Think of void[], and GC.malloc above.

-Steve




----- Original Message ----
> From: Sean Kelly <sean at invisibleduck.org>
> To: Discuss the phobos library for D <phobos at puremagic.com>
> Sent: Wed, July 14, 2010 12:25:09 AM
> Subject: Re: [phobos] custom BlkAttr flags
> 
> On Jul 13, 2010, at 8:09 AM, Steve Schveighoffer wrote:
> > 
> > What I  propose to fix this is to allocate one of the block attribute flags
>to
>
> >  designate that a block is appendable.  The odd part about it, is that the
>GC is
>
> > not aware of appending, so it is somewhat of a "user-defined"  flag.
> > 
> > My question is, how should we declare the flag?  I  currently only defined it
>in
>
> > lifetime.d, outside the gc, but I'm  thinking it should be defined in the GC
>as
>
> > well.  Also, should I  allocate the next bit, or one of the higher order
>bits?
> > 
> > I'm  going to check in my code that defines it in lifetime.d only, and
>depending
>
> > on the responses to this, I'll add it to the GC definition  too.
> 
> Your checkin doesn't compile.  I added the requisite comma for  now to fix
>this.  That aside, assuming this design were to be kept I think  the meaning of the bit has to be flipped.  The bit defaults to unset, so  that should be the default meaning.  You really don't want all blocks to be  appendable by default, do you?  More generally, if we start holding  TypeInfo references in the GC then we should be able to determine appendabiility  based on whether the TypeInfo is an array type.  This may not make the next  release, but I'd like to address it once I've polished std.concurrency a bit  more.
> _______________________________________________
> phobos mailing  list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
> 

But in both cases, TypeInfo could be supplied (either at the point of allocation or set later).  A bit would be more efficient however.  Regarding how this would all work, I was thinking of redefining calloc as:

void* calloc(TypeInfo ti, size_t count);

So the TypeInfo would replace the 'size' argument from C.  This makes calloc a lot more useful than just "malloc with bzero" as it is now.

On Jul 14, 2010, at 3:26 AM, Steve Schveighoffer wrote:

> I added the fix for the comma, sorry for that.  All the other places had an ALL_BITS enum member, so the NO_MOVE line that I copied and changed had a comma.  I thought I compiled before I checked in, but I guess I did not.
> 
> Regarding the bit being flipped, yes I want the default meaning to be "no append" because anyone can allocate memory via GC.malloc for any purpose, and GC.malloc does not initialize the "allocated length" field inside the block, so it is not appendable.  It does not mean you cannot append to the block, it just means the first append will reallocate into an appendable block.
> 
> Regarding using the typeinfo, that doesn't work.  Arrays can easily be created out of blocks that weren't allocated via the array creation routines.  Think of void[], and GC.malloc above.
> 
> -Steve
> 
> 
> 
> 
> ----- Original Message ----
>> From: Sean Kelly <sean at invisibleduck.org>
>> To: Discuss the phobos library for D <phobos at puremagic.com>
>> Sent: Wed, July 14, 2010 12:25:09 AM
>> Subject: Re: [phobos] custom BlkAttr flags
>> 
>> On Jul 13, 2010, at 8:09 AM, Steve Schveighoffer wrote:
>>> 
>>> What I  propose to fix this is to allocate one of the block attribute flags
>> to
>> 
>>> designate that a block is appendable.  The odd part about it, is that the
>> GC is
>> 
>>> not aware of appending, so it is somewhat of a "user-defined"  flag.
>>> 
>>> My question is, how should we declare the flag?  I  currently only defined it
>> in
>> 
>>> lifetime.d, outside the gc, but I'm  thinking it should be defined in the GC
>> as
>> 
>>> well.  Also, should I  allocate the next bit, or one of the higher order
>> bits?
>>> 
>>> I'm  going to check in my code that defines it in lifetime.d only, and
>> depending
>> 
>>> on the responses to this, I'll add it to the GC definition  too.
>> 
>> Your checkin doesn't compile.  I added the requisite comma for  now to fix
>> this.  That aside, assuming this design were to be kept I think  the meaning of
>> the bit has to be flipped.  The bit defaults to unset, so  that should be the
>> default meaning.  You really don't want all blocks to be  appendable by default,
>> do you?  More generally, if we start holding  TypeInfo references in the GC then
>> we should be able to determine appendabiility  based on whether the TypeInfo is
>> an array type.  This may not make the next  release, but I'd like to address it
>> once I've polished std.concurrency a bit  more.
>> _______________________________________________
>> phobos mailing  list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>> 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos