Is object.reserve() only useful for large arrays, or should I always use it if I intend to append to an array?

Let's say I want to read some data, and I expect there to be ~100 bytes of data.  Is there any point in using reserve() first, or will there always be that much memory available to an array?

  byte[] buffer;
  buffer.reserve(100);
  foreach(byte b; dataSource) buffer ~= b;

-Lars

On Wed, 14 Jul 2010 06:01:10 -0400, Lars T. Kyllingstad <public@kyllingen.nospamnet> wrote:

> Is object.reserve() only useful for large arrays, or should I always use
> it if I intend to append to an array?
>
> Let's say I want to read some data, and I expect there to be ~100 bytes
> of data.  Is there any point in using reserve() first, or will there
> always be that much memory available to an array?
>
>   byte[] buffer;
>   buffer.reserve(100);
>   foreach(byte b; dataSource) buffer ~= b;

Yes, you should always reserve if you know how much data is coming.  If you do not here is what happens:

Upon adding the first byte, a block of 16 bytes is allocated
Upon adding the 16th byte (there is one byte for padding), a *new* block of 32 bytes is allocated, and the 15 previous bytes are copied to the 32-byte block.  The original 16 byte block is left allocated because there could be other aliases to the data.
Upon adding the 32nd byte, a block of 64 bytes is allocated, rinse and repeat.
Upon adding the 64th byte, a block of 128 bytes is allocated, same deal.

Then the block will hold 100 bytes.

If you reserve 100 bytes, then a block of 128 bytes is allocated, and your data all goes in there.

OT, I assume you realize that reading data in this way is horribly inefficient :)  You should read a block at once if possible.

-Steve

On Wed, 14 Jul 2010 10:35:19 -0400, Steven Schveighoffer wrote:

> On Wed, 14 Jul 2010 06:01:10 -0400, Lars T. Kyllingstad <public@kyllingen.nospamnet> wrote:
> 
>> Is object.reserve() only useful for large arrays, or should I always use it if I intend to append to an array?
>>
>> Let's say I want to read some data, and I expect there to be ~100 bytes of data.  Is there any point in using reserve() first, or will there always be that much memory available to an array?
>>
>>   byte[] buffer;
>>   buffer.reserve(100);
>>   foreach(byte b; dataSource) buffer ~= b;
> 
> Yes, you should always reserve if you know how much data is coming.  If you do not here is what happens:
> 
> Upon adding the first byte, a block of 16 bytes is allocated Upon adding
> the 16th byte (there is one byte for padding), a *new* block of 32 bytes
> is allocated, and the 15 previous bytes are copied to the 32-byte block.
>  The original 16 byte block is left allocated because there could be
> other aliases to the data.
> Upon adding the 32nd byte, a block of 64 bytes is allocated, rinse and
> repeat.
> Upon adding the 64th byte, a block of 128 bytes is allocated, same deal.
> 
> Then the block will hold 100 bytes.
> 
> If you reserve 100 bytes, then a block of 128 bytes is allocated, and your data all goes in there.
> 
> OT, I assume you realize that reading data in this way is horribly inefficient :)  You should read a block at once if possible.


Yeah, that was just an artificial example. :)  The actual use case was building a string from substrings.

Thanks!

-Lars