On 14/12/2020 6:01 AM, Jackson22 wrote:
>>>
>>> `array = array.ptr[0..newLength];`
>>
>> You're setting yourself up for failure with that. What are you trying to "work around"? The allocation, or the initialization?
> 
> How is avoiding an expensive potentially memory leaking operation "setting yourself up for failure"?

No bounds checking. That slice can extend into memory that isn't of that type or even allocated to the process.

By avoiding that "expensive" memory operation, you instead create a silent memory corruption in its place which is far worse.

On Sunday, 13 December 2020 at 17:26:45 UTC, rikki cattermole wrote:
> On 14/12/2020 6:01 AM, Jackson22 wrote:
>>>>
>>>> `array = array.ptr[0..newLength];`
>>>
>>> You're setting yourself up for failure with that. What are you trying to "work around"? The allocation, or the initialization?
>> 
>> How is avoiding an expensive potentially memory leaking operation "setting yourself up for failure"?
>
> No bounds checking. That slice can extend into memory that isn't of that type or even allocated to the process.

No *automatic* bounds checking != no bounds checking.

There's a reason .ptr exist, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure when there are more successful programming languages that have zero automatic bounds checking.

> By avoiding that "expensive" memory operation, you instead create a silent memory corruption in its place which is far worse.

Why did you quote expensive? Are you implying it isn't expensive? Are you saying re-allocating 4 GB of memory every 6 ms isn't expensive?

On 14/12/2020 9:03 AM, Jackson22 wrote:
> On Sunday, 13 December 2020 at 17:26:45 UTC, rikki cattermole wrote:
>> On 14/12/2020 6:01 AM, Jackson22 wrote:
>>>>>
>>>>> `array = array.ptr[0..newLength];`
>>>>
>>>> You're setting yourself up for failure with that. What are you trying to "work around"? The allocation, or the initialization?
>>>
>>> How is avoiding an expensive potentially memory leaking operation "setting yourself up for failure"?
>>
>> No bounds checking. That slice can extend into memory that isn't of that type or even allocated to the process.
> 
> No *automatic* bounds checking != no bounds checking.
> 
> There's a reason .ptr exist, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure 

I have used it in the past where appropriate with 0 issues resulting from it. I do not believe that this is the case here.

> when there are more successful programming languages that have zero automatic bounds checking.

int[] a = [1, 2, 3];
assert(a.ptr is a.ptr[0 .. 4].ptr);

Out of bounds, runs successfully. Doesn't mean that the GC is aware that it now has a length of 4.

int[] b = a;
b.length = 4;
assert(a.ptr !is b.ptr);

This is a case where .length is clearly doing the right thing.

int[] c = a;
c.length = 1;
assert(a.ptr is c.ptr);

>> By avoiding that "expensive" memory operation, you instead create a silent memory corruption in its place which is far worse.
> 
> Why did you quote expensive? Are you implying it isn't expensive? Are you saying re-allocating 4 GB of memory every 6 ms isn't expensive?

Allocating memory is always more expensive than using a buffer where life times are known and predictable. You are right about that.

In this case, that isn't what is being described. If length is allocating, then that code was not designed to be used with a buffer.

The most expensive thing in this scenario is not allocating memory, it is silent memory corruption. Once corrupted not only can the process die at any point, but you can't trust its output any longer.

On Sunday, 13 December 2020 at 20:03:46 UTC, Jackson22 wrote:
>> No bounds checking. That slice can extend into memory that isn't of that type or even allocated to the process.
>
> No *automatic* bounds checking != no bounds checking.
>
> There's a reason .ptr exist, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure when there are more successful programming languages that have zero automatic bounds checking.

Yes it's possible to without automatic bounds checks. Sometimes one has to -when using those older langages or doing very low-level system programming. And other times it may not be necessary, but still worth it to gain that last bit of performance when optimizing. These are the reasons why `.ptr` exists.

We really don't know whether either of those cases apply to OP:s case, but if the length extension with implicit duplications were even close to the desired performance, it seems unlikely.

>
> Why did you quote expensive? Are you implying it isn't expensive? Are you saying re-allocating 4 GB of memory every 6 ms isn't expensive?

I think he was comparing to extending the array in-place, but in a bounds-checked way.

On Sunday, 13 December 2020 at 21:01:18 UTC, Dukc wrote:
> Yes it's possible to without automatic bounds checks.

Meant: Yes it's possible to live without automatic bounds checks.

On Sunday, 13 December 2020 at 20:03:46 UTC, Jackson22 wrote:
> On Sunday, 13 December 2020 at 17:26:45 UTC, rikki cattermole wrote:
>> On 14/12/2020 6:01 AM, Jackson22 wrote:
>>>>>
>>>>> `array = array.ptr[0..newLength];`
>>>>
>>>> You're setting yourself up for failure with that. What are you trying to "work around"? The allocation, or the initialization?
>>> 
>>> How is avoiding an expensive potentially memory leaking operation "setting yourself up for failure"?

"avoiding an expensive potentially memory leaking operation" is not the issue, it's how the OP is going about it.

Based on the OP's question and the example, the impression I get is that it's an attempt to arbitrarily increase the length of a slice with no regard to the capacity of its memory store. If `newLength` is greater than the remaining capacity in the memory store, then the new length will go beyond whatever has been allocated. That is what I meant by "setting yourself up for failure", and that is why the lack of bounds checking is an issue here. |

Steven's post lays out other potential issues with taking this approach in D.

>>
>> No bounds checking. That slice can extend into memory that isn't of that type or even allocated to the process.
>
> No *automatic* bounds checking != no bounds checking.
>

But even with manual bounds checking, there has to be enough memory allocated somewhere to hold the new array elements. For a dynamically resizable array, there is no escaping the need to allocate memory. The cost can be mitigated by allocating enough up front, or with a tailored reallocation strategy, but it can't be eliminated.

On Sunday, 13 December 2020 at 20:03:46 UTC, Jackson22 wrote:
>
> There's a reason .ptr exist, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure when there are more successful programming languages that have zero automatic bounds checking.
>

There's no pretending here. What the OP is doing *is* dangerous.

On Monday, 14 December 2020 at 01:36:02 UTC, Mike Parker wrote:
> On Sunday, 13 December 2020 at 20:03:46 UTC, Jackson22 wrote:
>>
>> There's a reason .ptr exist, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure when there are more successful programming languages that have zero automatic bounds checking.
>>
>
> There's no pretending here. What the OP is doing *is* dangerous.

If someone writes a wrapper around .ptr which checks. It'd be literally no different than the implementation in druntime.

Like I said, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure. Maybe those people just aren't knowledgeable enough to understand, I don't know.

On Monday, 14 December 2020 at 20:53:39 UTC, Jackson22 wrote:
> On Monday, 14 December 2020 at 01:36:02 UTC, Mike Parker wrote:
>> On Sunday, 13 December 2020 at 20:03:46 UTC, Jackson22 wrote:
>>>
>>> There's a reason .ptr exist, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure when there are more successful programming languages that have zero automatic bounds checking.
>>>
>>
>> There's no pretending here. What the OP is doing *is* dangerous.
>
> If someone writes a wrapper around .ptr which checks. It'd be literally no different than the implementation in druntime.
>
> Like I said, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure. Maybe those people just aren't knowledgeable enough to understand, I don't know.

Good practice is good practice. If you know what you're doing you probably shouldn't need to ask.

What Mike is saying is important to know, because even if you use exactly the same concept as what druntime does in your code, you're still repeating a pattern which will lead to bugs if you get it wrong. Good code is all about compartmentalizing bad code, especially with memory where (thankfully we have sanitizers now) things can often go badly wrong without actually exhibiting any side-effects (i.e. we all know why C code has so many security problems)

On 12/14/20 3:53 PM, Jackson22 wrote:
> On Monday, 14 December 2020 at 01:36:02 UTC, Mike Parker wrote:
>> On Sunday, 13 December 2020 at 20:03:46 UTC, Jackson22 wrote:
>>>
>>> There's a reason .ptr exist, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure when there are more successful programming languages that have zero automatic bounds checking.
>>>
>>
>> There's no pretending here. What the OP is doing *is* dangerous.
> 
> If someone writes a wrapper around .ptr which checks. It'd be literally no different than the implementation in druntime.
> 
> Like I said, I wish people would stop pretending that using it where it is appropriate is somehow going to lead to failure. Maybe those people just aren't knowledgeable enough to understand, I don't know.

It's possible you have misinterpreted what the OP is asking for.

Maybe the OP misstated what he is looking to do. Without a clarifying response from him, it's hard to tell how to respond, which means we have to respond with the most pessimistic interpretation of the post possible.

Yes, you can use .ptr to avoid bounds checks, and it's safe if you do it correctly. No you shouldn't use .ptr to create array slices that refer to memory outside the range that exists (and using .ptr slicing as posted in the original can do this). It's as basic as that.

-Steve