phobos commit, revision 1930


user: schveiguy

msg:
Upon discussion in the appender access bug, it was determined we could make the Appender able to use the entire memory block, not just the data passed in.

With this fix, Appender and builtin appending should both be callable on the same data without corruption, of course using the builtin append on data held
by an Appender will reallocate.

http://www.dsource.org/projects/phobos/changeset/1930

Nice work.  One small question/possible feature request, though:  Will calling assumeSafeAppend() on an array before constructing an Appender using it prevent it from being reallocated on the first append?  If not, can Appender have some kind of assumeSafeAppend workaround in it?

On Thu, Aug 26, 2010 at 1:04 PM, dsource.org <noreply at dsource.org> wrote:

> phobos commit, revision 1930
>
>
> user: schveiguy
>
> msg:
> Upon discussion in the appender access bug, it was determined we could make
> the Appender able to use the entire memory block, not just the data passed
> in.
>
> With this fix, Appender and builtin appending should both be callable on
> the same data without corruption, of course using the builtin append on data
> held
> by an Appender will reallocate.
>
> http://www.dsource.org/projects/phobos/changeset/1930
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20100826/9b9110e7/attachment-0001.html>

The constructor uses arr.capacity to determine how much it can append.

assumeSafeAppend will ensure that arr.capacity is as large as possible, so yes, calling before creating the Appender will maximize the capacity, regardless of whether it will stomp or not.

One thing that Appender will do, which is bad, but I'm not sure we need to care, it can be given a string literal, and then try to write over the string literal like this:

auto app = appender("hello");
// app.put('c'); // this would be ok
app.clear();
app.put('c'); // segfault on Linux

I'm not sure how to get around this.  I thought of checking the heap to see if it owns the pointer, but then you disallow useful things like using a stack-allocated buffer.  I think this case may be uncommon enough to not care.

-Steve

>
>From: David Simcha <dsimcha at gmail.com>
>To: Discuss the phobos library for D <phobos at puremagic.com>
>Sent: Thu, August 26, 2010 1:10:31 PM
>Subject: Re: [phobos] phobos commit, revision 1930
>
>Nice work.  One small question/possible feature request, though:  Will calling assumeSafeAppend() on an array before constructing an Appender using it prevent

>it from being reallocated on the first append?  If not, can Appender have some kind of assumeSafeAppend workaround in it?

Walter, Sean, could we arrange such that capacity of static arrays yields zero?

Andrei

On 8/26/10 10:16 PDT, Steve Schveighoffer wrote:
> The constructor uses arr.capacity to determine how much it can append.
>
> assumeSafeAppend will ensure that arr.capacity is as large as possible, so yes, calling before creating the Appender will maximize the capacity, regardless of whether it will stomp or not.
>
> One thing that Appender will do, which is bad, but I'm not sure we need to care, it can be given a string literal, and then try to write over the string literal like this:
>
> auto app = appender("hello");
> // app.put('c'); // this would be ok
> app.clear();
> app.put('c'); // segfault on Linux
>
> I'm not sure how to get around this.  I thought of checking the heap to see if it owns the pointer, but then you disallow useful things like using a stack-allocated buffer.  I think this case may be uncommon enough to not care.
>
> -Steve
>
>>
>> From: David Simcha<dsimcha at gmail.com>
>> To: Discuss the phobos library for D<phobos at puremagic.com>
>> Sent: Thu, August 26, 2010 1:10:31 PM
>> Subject: Re: [phobos] phobos commit, revision 1930
>>
>> Nice work.  One small question/possible feature request, though:  Will calling assumeSafeAppend() on an array before constructing an Appender using it prevent
>
>> it from being reallocated on the first append?  If not, can Appender have some kind of assumeSafeAppend workaround in it?
>
>
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

GAH! I did it again!

Andrei, stop replying to all :)

-Steve



----- Forwarded Message ----
> From: Steve Schveighoffer <schveiguy at yahoo.com>
> To: Andrei Alexandrescu <andrei at erdani.com>
> Sent: Thu, August 26, 2010 1:24:43 PM
> Subject: Re: [phobos] phobos commit, revision 1930
> 
> No, I want to be able to use static arrays as a buffer.  This allows nifty things like using the stack if possible, and reallocating if you go beyond  the
>
> stack buffer size.
> 
> BTW, capacity already returns 0 for  these.
> 
> It's immutable *literals* that I want to disallow.  Can we  get a function to determine if a string is a literal vs a stack or  heap-allocated value?
> 
> On that topic a bit, calling clear on an  Appender!string allows you to
>overwrite
>
> immutable data.  Do we care too  much?  I think we can just mark the clear function as @system and call  it a day...
> 
> -Steve
> 
> 
> 
> ----- Original Message ----
> >  From: Andrei Alexandrescu <andrei at erdani.com>
> > To: Discuss  the phobos library for D <phobos at puremagic.com>
> > Cc:  Steve Schveighoffer <schveiguy at yahoo.com>
> > Sent: Thu,  August 26, 2010 1:18:25 PM
> > Subject: Re: [phobos] phobos commit, revision  1930
> > 
> > Walter, Sean, could we arrange such that capacity of  static arrays yields  zero?
> > 
> > Andrei
> > 
> > On 8/26/10 10:16 PDT, Steve Schveighoffer  wrote:
> > > The  constructor uses arr.capacity to determine how much it can  append.
> >  >
> > > assumeSafeAppend will ensure that arr.capacity is as   large as possible,
>so
>
> >yes,
> > > calling before creating the  Appender will  maximize the capacity,
>regardless
>
> >of
> > >  whether it will stomp or  not.
> > >
> > > One thing that  Appender will do, which is bad, but I'm not  sure we need
>to
>
> >care,
> > > it can be given a string literal, and then try  to  write over the string
> >literal
> > > like this:
> >  >
> > > auto app =  appender("hello");
> > > //  app.put('c'); // this would be ok
> > >  app.clear();
> > >  app.put('c'); // segfault on Linux
> > >
> > > I'm not  sure  how to get around this.  I thought of checking the heap to
>see
>
> >if
> > > it owns the pointer, but then you disallow useful things  like using  a stack-allocated buffer.  I think this case  may be uncommon enough  to not

> >care.
> > >
> > >  -Steve
> > >
> > >>
> > >> From: David   Simcha<dsimcha at gmail.com>
> > >>  To:  Discuss the phobos library for D<phobos at puremagic.com>
> >  >>  Sent: Thu, August 26, 2010 1:10:31 PM
> > >> Subject:  Re: [phobos] phobos  commit, revision 1930
> > >>
> >  >> Nice work.  One small  question/possible feature request,  though:  Will

> >calling
> > >>  assumeSafeAppend() on  an array before constructing an Appender using it
>
> >prevent
> >  >
> > >> it from being reallocated on the first append?   If  not, can Appender
>have
>
> >some
> > >> kind of assumeSafeAppend  workaround  in it?
> > >
> > >
> > >
> >  >  _______________________________________________
> > > phobos  mailing  list
> > > phobos at puremagic.com
> > >   http://lists.puremagic.com/mailman/listinfo/phobos
> > 
> 
> 
> 
> 

Damn, sorry. Will do.

Andrei

On 8/26/10 10:27 PDT, Steve Schveighoffer wrote:
> GAH! I did it again!
>
> Andrei, stop replying to all :)
>
> -Steve
>
>
>
> ----- Forwarded Message ----
>> From: Steve Schveighoffer<schveiguy at yahoo.com>
>> To: Andrei Alexandrescu<andrei at erdani.com>
>> Sent: Thu, August 26, 2010 1:24:43 PM
>> Subject: Re: [phobos] phobos commit, revision 1930
>>
>> No, I want to be able to use static arrays as a buffer.  This allows nifty things like using the stack if possible, and reallocating if you go beyond  the
>>
>> stack buffer size.
>>
>> BTW, capacity already returns 0 for  these.
>>
>> It's immutable *literals* that I want to disallow.  Can we  get a function to determine if a string is a literal vs a stack or  heap-allocated value?
>>
>> On that topic a bit, calling clear on an  Appender!string allows you to overwrite
>>
>> immutable data.  Do we care too  much?  I think we can just mark the clear function as @system and call  it a day...
>>
>> -Steve
>>
>>
>>
>> ----- Original Message ----
>>>   From: Andrei Alexandrescu<andrei at erdani.com>
>>> To: Discuss  the phobos library for D<phobos at puremagic.com>
>>> Cc:  Steve Schveighoffer<schveiguy at yahoo.com>
>>> Sent: Thu,  August 26, 2010 1:18:25 PM
>>> Subject: Re: [phobos] phobos commit, revision  1930
>>>
>>> Walter, Sean, could we arrange such that capacity of  static arrays yields  zero?
>>>
>>> Andrei
>>>
>>> On 8/26/10 10:16 PDT, Steve Schveighoffer  wrote:
>>>> The  constructor uses arr.capacity to determine how much it can  append.
>>>   >
>>>> assumeSafeAppend will ensure that arr.capacity is as   large as possible,
>> so
>>
>>> yes,
>>>> calling before creating the  Appender will  maximize the capacity,
>> regardless
>>
>>> of
>>>>   whether it will stomp or  not.
>>>>
>>>> One thing that  Appender will do, which is bad, but I'm not  sure we need
>> to
>>
>>> care,
>>>> it can be given a string literal, and then try  to  write over the string
>>> literal
>>>> like this:
>>>   >
>>>> auto app =  appender("hello");
>>>> //  app.put('c'); // this would be ok
>>>>   app.clear();
>>>>   app.put('c'); // segfault on Linux
>>>>
>>>> I'm not  sure  how to get around this.  I thought of checking the heap to
>> see
>>
>>> if
>>>> it owns the pointer, but then you disallow useful things  like using  a stack-allocated buffer.  I think this case  may be uncommon enough  to not
>
>>> care.
>>>>
>>>>   -Steve
>>>>
>>>>>
>>>>> From: David   Simcha<dsimcha at gmail.com>
>>>>>   To:  Discuss the phobos library for D<phobos at puremagic.com>
>>>   >>   Sent: Thu, August 26, 2010 1:10:31 PM
>>>>> Subject:  Re: [phobos] phobos  commit, revision 1930
>>>>>
>>>   >>  Nice work.  One small  question/possible feature request,  though:  Will
>
>>> calling
>>>>>   assumeSafeAppend() on  an array before constructing an Appender using it
>>
>>> prevent
>>>   >
>>>>> it from being reallocated on the first append?   If  not, can Appender
>> have
>>
>>> some
>>>>> kind of assumeSafeAppend  workaround  in it?
>>>>
>>>>
>>>>
>>>   >   _______________________________________________
>>>> phobos  mailing  list
>>>> phobos at puremagic.com
>>>>    http://lists.puremagic.com/mailman/listinfo/phobos
>>>
>>
>>
>>
>>
>
>
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Sure, we can do that.

-Steve



----- Original Message ----
> From: Andrei Alexandrescu <andrei at erdani.com>
> To: Steve Schveighoffer <schveiguy at yahoo.com>
> Sent: Thu, August 26, 2010 1:26:58 PM
> Subject: Re: [phobos] phobos commit, revision 1930
> 
> On 8/26/10 10:24 PDT, Steve Schveighoffer wrote:
> > On that topic a bit,  calling clear on an Appender!string allows you to
>overwrite
> > immutable  data.  Do we care too much?  I think we can just mark the  clear function as @system and call it a day...
> 
> I think it's  important to preserve immutable. Let's guard clear with static if
>(!is(T ==  const) && !is(T == immutable)). Thanks!
> 
> Andrei
> 

incidentally, this also fixes the problem of being able to overwrite string literals ;)

-Steve



----- Original Message ----
> From: Steve Schveighoffer <schveiguy at yahoo.com>
> To: Phobos <phobos at puremagic.com>
> Sent: Thu, August 26, 2010 1:30:19 PM
> Subject: Re: [phobos] phobos commit, revision 1930
> 
> Sure, we can do that.
> 
> -Steve
> 
> 
> 
> ----- Original Message  ----
> > From: Andrei Alexandrescu <andrei at erdani.com>
> > To: Steve  Schveighoffer <schveiguy at yahoo.com>
> > Sent: Thu,  August 26, 2010 1:26:58 PM
> > Subject: Re: [phobos] phobos commit, revision  1930
> > 
> > On 8/26/10 10:24 PDT, Steve Schveighoffer wrote:
> >  > On that topic a bit,  calling clear on an Appender!string allows you  to
> >overwrite
> > > immutable  data.  Do we care too  much?  I think we can just mark the
>clear
> > > function as  @system and call it a day...
> > 
> > I think it's  important to  preserve immutable. Let's guard clear with static
>if
>
> >(!is(T ==   const) && !is(T == immutable)). Thanks!
> > 
> > Andrei
> > 
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
> 

It already does.  I've just realized that the current length implementation can have some weird side-effects at times though:

    writeln(GC.sizeOf((new byte[2046]).ptr));
    writeln(GC.sizeOf((new byte[2047]).ptr));

prints:

    2048
    0

Because the length of large arrays is located at the beginning of the memory block, and this is placed by the runtime (I believe), the GC assumes that large arrays are all interior slices.

Also:

    auto x = new byte[100];
    writeln(x.capacity);
    x = x[1 .. $];
    writeln(x.capacity);

prints:

    127
    126

Shouldn't the second capacity be 0, since it's a slice?

On Aug 26, 2010, at 10:18 AM, Andrei Alexandrescu wrote:

> Walter, Sean, could we arrange such that capacity of static arrays yields zero?
> 
> Andrei
> 
> On 8/26/10 10:16 PDT, Steve Schveighoffer wrote:
>> The constructor uses arr.capacity to determine how much it can append.
>> 
>> assumeSafeAppend will ensure that arr.capacity is as large as possible, so yes, calling before creating the Appender will maximize the capacity, regardless of whether it will stomp or not.
>> 
>> One thing that Appender will do, which is bad, but I'm not sure we need to care, it can be given a string literal, and then try to write over the string literal like this:
>> 
>> auto app = appender("hello");
>> // app.put('c'); // this would be ok
>> app.clear();
>> app.put('c'); // segfault on Linux
>> 
>> I'm not sure how to get around this.  I thought of checking the heap to see if it owns the pointer, but then you disallow useful things like using a stack-allocated buffer.  I think this case may be uncommon enough to not care.
>> 
>> -Steve
>> 
>>> 
>>> From: David Simcha<dsimcha at gmail.com>
>>> To: Discuss the phobos library for D<phobos at puremagic.com>
>>> Sent: Thu, August 26, 2010 1:10:31 PM
>>> Subject: Re: [phobos] phobos commit, revision 1930
>>> 
>>> Nice work.  One small question/possible feature request, though:  Will calling assumeSafeAppend() on an array before constructing an Appender using it prevent
>> 
>>> it from being reallocated on the first append?  If not, can Appender have some kind of assumeSafeAppend workaround in it?
>> 
>> 
>> 
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

----- Original Message ----
> From: Sean Kelly <sean at invisibleduck.org>
> 
> On Aug  26, 2010, at 10:18 AM, Andrei Alexandrescu wrote:
> 
> > Walter, Sean,  could we arrange such that capacity of static arrays yields
>zero?
> > 
> > Andrei
> 
> It already does.  I've just realized that the current length implementation
>can have some weird side-effects at times though:
> 
>      writeln(GC.sizeOf((new byte[2046]).ptr));
>      writeln(GC.sizeOf((new byte[2047]).ptr));
> 
> prints:
> 
>      2048
>     0
> 
> Because the length of large arrays is located at  the beginning of the memory
>block, and this is placed by the runtime (I  believe), the GC assumes that large arrays are all interior  slices.

That is something I want to fix.  My reasons for putting the length at the front were purely for shared arrays.  The issue is, once you call gc_query on a block, it could change if another thread came along and extended it.  This can be a problem if the length is stored at the end of the block, because the end of the block can move!  Unshared arrays don't have this problem because they cannot be changed from another thread.

There's a few ways we could fix this so the length is not stored at the beginning.  One is, of course, store the length outside the block :)  Another is, we could put a global lock on all shared appends to avoid the block changing underneath us.  Another possibility is to store a lock bit for the block, thereby preventing anything else from changing the size.  This would only happen for shared arrays of course, unshared arrays wouldn't need any locking.

> 
> Also:
> 
>     auto x = new byte[100];
>      writeln(x.capacity);
>     x = x[1 .. $];
>      writeln(x.capacity);
> 
> prints:
> 
>     127
>      126
> 
> Shouldn't the second capacity be 0, since it's a slice?

Slices can be appended to.  The key is not where they start, but where they end.  If you think about it, it makes a lot of sense.

-Steve