I'm sure we had already this conversation but I don't find the thread.

T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.

Thoughts?

Namespace:

> T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.
>
> Thoughts?

In Python if you append items to an array, it sovra-allocates, but if you create an array with the [x] * n syntax, then it creates a list (array) exactly of size n.

Bye,
bearophile

On Thursday, 31 October 2013 at 09:27:11 UTC, bearophile wrote:
> Namespace:
>
>> T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.
>>
>> Thoughts?
>
> In Python if you append items to an array, it sovra-allocates,
Never heard of 'sovra'. What does it mean?

> but if you create an array with the [x] * n syntax, then it creates a list (array) exactly of size n.
>
> Bye,
> bearophile

So you agree with me.

On Thursday, 31 October 2013 at 09:15:53 UTC, Namespace wrote:
> I'm sure we had already this conversation but I don't find the thread.
>
> T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.
>
> Thoughts?

To me it looks like it is derived directly from the way the GC allocates chunks:
Next power of two if less than 4096 otherwise some multiple of 4096.

Unless you modify the GC, this behaviour is present whether you can see it or not (http://dpaste.dzfl.pl/5481ffc2 .)

On Thursday, October 31, 2013 10:15:51 Namespace wrote:
> I'm sure we had already this conversation but I don't find the thread.
> 
> T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.
> 
> Thoughts?

You're making the assumption that it would be normal to not want to then append to something you allocated with new T[N], and I don't think that that's a valid assumption. Also, from what I understand of how allocators work, they normally grab memory in sizes which are a multiple of 2, so I would expect it to be highly abnormal to ever get exactly the amount of memory that you requested. You're pretty much always going to get extra.

- Jonathan M Davis

On Thursday, 31 October 2013 at 09:48:23 UTC, safety0ff wrote:
> On Thursday, 31 October 2013 at 09:15:53 UTC, Namespace wrote:
>> I'm sure we had already this conversation but I don't find the thread.
>>
>> T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.
>>
>> Thoughts?
>
> To me it looks like it is derived directly from the way the GC allocates chunks:
> Next power of two if less than 4096 otherwise some multiple of 4096.
>
> Unless you modify the GC, this behaviour is present whether you can see it or not (http://dpaste.dzfl.pl/5481ffc2 .)

Maybe (and hopefully) I'm wrong, but it seems that the static array is on the heap?

On Thursday, October 31, 2013 10:59:48 Namespace wrote:
> On Thursday, 31 October 2013 at 09:48:23 UTC, safety0ff wrote:
> > On Thursday, 31 October 2013 at 09:15:53 UTC, Namespace wrote:
> >> I'm sure we had already this conversation but I don't find the thread.
> >> 
> >> T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.
> >> 
> >> Thoughts?
> > 
> > To me it looks like it is derived directly from the way the GC
> > allocates chunks:
> > Next power of two if less than 4096 otherwise some multiple of
> > 4096.
> > 
> > Unless you modify the GC, this behaviour is present whether you can see it or not (http://dpaste.dzfl.pl/5481ffc2 .)
> 
> Maybe (and hopefully) I'm wrong, but it seems that the static
> array is on the heap?

T[] buffer = new T[N];

is a dynamic array. Now, in the code in your dpaste link, you have a static array which is in a struct which is on the heap, so that's different, but now you're dealing with the amount of memory that gets allocated on the heap for the struct, and as the GC is going to allocate in multiples of 2, more than the size of the struct is going to be allocated. It might be that some of the memory beyond the end of the struct might actually be used for another object rather than the struct using the memory block by itself (I'm not sure what the GC does in that regard), but it's definitely going to allocate in multiples of 2, so if the GC has to go up another multiple of 2 to make the struct fit, it'll go up another multiple of 2.

- Jonathan M Davis

On Thursday, 31 October 2013 at 10:12:10 UTC, Jonathan M Davis wrote:
> On Thursday, October 31, 2013 10:59:48 Namespace wrote:
>> On Thursday, 31 October 2013 at 09:48:23 UTC, safety0ff wrote:
>> > On Thursday, 31 October 2013 at 09:15:53 UTC, Namespace wrote:
>> >> I'm sure we had already this conversation but I don't find the
>> >> thread.
>> >> 
>> >> T[] buffer = new T[N]; assumes more space than stated (in
>> >> average 2010 elements more. See:
>> >> http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like
>> >> reserve and that is IMO wrong. If I reserve memory with
>> >> buffer.reserve(N), I want to have at least N elements. That
>> >> behaviour is correct. But if I use new T[N] I mostly want
>> >> exactly N elements and no extra space.
>> >> 
>> >> Thoughts?
>> > 
>> > To me it looks like it is derived directly from the way the GC
>> > allocates chunks:
>> > Next power of two if less than 4096 otherwise some multiple of
>> > 4096.
>> > 
>> > Unless you modify the GC, this behaviour is present whether you
>> > can see it or not (http://dpaste.dzfl.pl/5481ffc2 .)
>> 
>> Maybe (and hopefully) I'm wrong, but it seems that the static
>> array is on the heap?
>
> T[] buffer = new T[N];
>
> is a dynamic array. Now, in the code in your dpaste link, you have a static
> array which is in a struct which is on the heap, so that's different, but now
> you're dealing with the amount of memory that gets allocated on the heap for
> the struct, and as the GC is going to allocate in multiples of 2, more than
> the size of the struct is going to be allocated. It might be that some of the
> memory beyond the end of the struct might actually be used for another object
> rather than the struct using the memory block by itself (I'm not sure what the
> GC does in that regard), but it's definitely going to allocate in multiples of
> 2, so if the GC has to go up another multiple of 2 to make the struct fit,
> it'll go up another multiple of 2.
>
> - Jonathan M Davis

Hm, seems not very performant for a *system* language. But fine, thanks for your explanation. :)

On Thursday, October 31, 2013 11:16:35 Namespace wrote:
> Hm, seems not very performant for a *system* language. But fine, thanks for your explanation. :)

I believe that C's malloc does the same thing.

- Jonathan M Davis

Am Thu, 31 Oct 2013 02:52:49 -0700
schrieb Jonathan M Davis <jmdavisProg@gmx.com>:

> On Thursday, October 31, 2013 10:15:51 Namespace wrote:
> > I'm sure we had already this conversation but I don't find the thread.
> > 
> > T[] buffer = new T[N]; assumes more space than stated (in average 2010 elements more. See: http://dpaste.dzfl.pl/af92ad22c). It behaves exactly like reserve and that is IMO wrong. If I reserve memory with buffer.reserve(N), I want to have at least N elements. That behaviour is correct. But if I use new T[N] I mostly want exactly N elements and no extra space.
> > 
> > Thoughts?

Maybe run a comparison with another popular GC enabled language and check the memory use to find pathological cases. I'd assume that D's GC doesn't waste more memory than any other for dynamic arrays.

> You're making the assumption that it would be normal to not want to then append to something you allocated with new T[N], and I don't think that that's a valid assumption.

That doesn't happen much in my programming practice. I'm with the OP on this.

> Also, from what I understand of how allocators work, they normally grab memory in sizes which are a multiple of 2, so I would expect it to be highly abnormal to ever get exactly the amount of memory that you requested. You're pretty much always going to get extra.
> 
> - Jonathan M Davis

That's true. Although now I wonder if you could gain anything from having memory pages split into chunks of T.sizeof from which you can allocate single objects or whole arrays of T with a compact layout...

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|T|T|T[6]       |T|T|           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

-- 
Marco