1) The D Language Reference says:

"There are four kinds of arrays..." with the first example being
"type*	Pointers to data"  and "int* p;  etc.

At the risk of sounding overly nitpicky, isn't a pointer to an integer simply a pointer to an integer?  How does that pertain to an array?


2) "The total size of a static array cannot exceed 16Mb" What limits this? And with modern systems of 16GB and 32GB, isn't 16Mb excessively small?   (an aside: shouldn't that be 16MB in the reference instead of 16Mb? that is, Doesn't b = bits and B = bytes)


3) Lastly, In the following code snippet, is arrayA and arrayB both allocated on the stack? And how does their scopes and/or lifetimes differ?

==== module1 =====
int[100] arrayA;
void main()
{
    int[100] arrayB;
    // ...
}
==== module1 =====

On Monday, 20 July 2020 at 22:05:35 UTC, WhatMeWorry wrote:
> 1) The D Language Reference says:
>
> "There are four kinds of arrays..." with the first example being
> "type*	Pointers to data"  and "int* p;  etc.
>
> At the risk of sounding overly nitpicky, isn't a pointer to an integer simply a pointer to an integer?  How does that pertain to an array?
>
>
> 2) "The total size of a static array cannot exceed 16Mb" What limits this? And with modern systems of 16GB and 32GB, isn't 16Mb excessively small?   (an aside: shouldn't that be 16MB in the reference instead of 16Mb? that is, Doesn't b = bits and B = bytes)
>
>
> 3) Lastly, In the following code snippet, is arrayA and arrayB both allocated on the stack? And how does their scopes and/or lifetimes differ?
>
> ==== module1 =====
> int[100] arrayA;
> void main()
> {
>     int[100] arrayB;
>     // ...
> }
> ==== module1 =====

1) Pointers can be used as arrays with the [] operator, int* p = arrayA.ptr; assert(*(p + 99) == p[99]); should access the same element.
http://ddili.org/ders/d.en/pointers.html ("Using pointers with the array indexing operator []")
2) I've encountered this problem too, it's arbitrary AFAIK but it can be circumvented with dynamic arrays.

On 7/20/20 8:16 PM, a@a.com wrote:

>> 3) Lastly, In the following code snippet, is arrayA and arrayB both
>> allocated on the stack?

arrayA is allocated on thread-local storage and lives as long as the program is active. I guess a final interaction with it can be in a 'static ~this()' or a 'shared static ~this()' block.

Note that this is different from e.g. C++: In that language, arrayA would be a "global" variable and there would be a single instance of it. In D, there will be as many arrayA variables as there are active threads. (One thread's modification to its own arrayA is not seen by other threads.)

arrayB is allocated on the stack and lives as long as the scope that it is defined inside. That scope is main's body in your code.

> And how does their scopes and/or lifetimes
>> differ?
>>
>> ==== module1 =====
>> int[100] arrayA;
>> void main()
>> {
>>     int[100] arrayB;
>>     // ...
>> }
>> ==== module1 =====

Ali

On Monday, 20 July 2020 at 22:05:35 UTC, WhatMeWorry wrote:
>
> 2) "The total size of a static array cannot exceed 16Mb" What limits this? And with modern systems of 16GB and 32GB, isn't 16Mb excessively small?   (an aside: shouldn't that be 16MB in the reference instead of 16Mb? that is, Doesn't b = bits and B = bytes)
>

I didn't know this but it makes sense and I guess this is a constraint of the D language itself. In practice 16MB should be well enough for most cases. I'm not sure where 16MB is taken from, if there is any OS out there that has this limitation or if it was just taken as an adequate limit.

Let's say you have a program with 4 threads, then suddenly the TLS area is 4 * 16 MB = 64MB. This size rapidly increases with number of threads and TLS area size. Let's say TLS area of 128MB and 8 threads, which gives you a memory consumption of 1GB. That's how quickly it starts to consume memory if you don't limit the TLS variables.

If you want global variables like in good old C/C++, then use __gshared. Of course you have to take care if any multiple accesses from several threads.

On Monday, 20 July 2020 at 22:05:35 UTC, WhatMeWorry wrote:
> 2) "The total size of a static array cannot exceed 16Mb" What limits this? And with modern systems of 16GB and 32GB, isn't 16Mb excessively small?   (an aside: shouldn't that be 16MB in the reference instead of 16Mb? that is, Doesn't b = bits and B = bytes)

Static arrays are passed by value.

(Also I think you're right about Mb vs MB except it should be MiB. 1MB = 1000^2 (decimal) and 1MiB = 1024^2 (binary).
Note that MB is defined 1024^2 in JEDEC 100B.01 but, IMO, ISO standard is superior because it's unambiguous and JEDEC only defines units up to GB (inclusive))

On 7/21/20 7:10 AM, IGotD- wrote:
> On Monday, 20 July 2020 at 22:05:35 UTC, WhatMeWorry wrote:
>>
>> 2) "The total size of a static array cannot exceed 16Mb" What limits this? And with modern systems of 16GB and 32GB, isn't 16Mb excessively small?   (an aside: shouldn't that be 16MB in the reference instead of 16Mb? that is, Doesn't b = bits and B = bytes)
>>
> 
> I didn't know this but it makes sense and I guess this is a constraint of the D language itself. In practice 16MB should be well enough for most cases. I'm not sure where 16MB is taken from, if there is any OS out there that has this limitation or if it was just taken as an adequate limit.

I believe it stems from a limitation in the way the stacks are allocated? Or maybe a limitation in DMC, the basis for DMD.

Also, you CAN actually have larger arrays, they just cannot be put on the stack (which most static arrays are):

struct S
{
    ubyte[17_000_000] big;
}

void main()
{
    auto s = new S; // ok
    S s; // crash (signal 11 on run.dlang.io)
}

This may not work if `big` had a static initializer, I'm not sure.

-Steve

On Monday, 20 July 2020 at 22:05:35 UTC, WhatMeWorry wrote:
> How does that pertain to an array?

C arrays work as pointers to the first element and D can use that style too.

> 2) "The total size of a static array cannot exceed 16Mb" What limits this?

The others aren't wrong about stack size limits playing some role, but the primary reason is that it is a weird hack for @safe, believe it or not.

The idea is:

---
class A {
    ubyte[4_000_000_000] whole_system;
}

@safe void lol() {
    A a;
    a.whole_system[any_address] = whatever;
}
---


With the null `a`, the offset to the static array is just 0 + whatever and the @safe mechanism can't trace that.

So the arbitrary limit was put in place to make it more likely that such a situation will hit a protected page and segfault instead of carrying on. (most low addresses are not actually allocated by the OS... though there's no reason why they couldn't, it just usually doesn't, so that 16 MB limit makes the odds of something like this actually happening a lot lower)

I don't recall exactly when this was discussed but it came up in the earlier days of @safe, I'm pretty sure it worked before then.

On Tuesday, 21 July 2020 at 12:34:14 UTC, Adam D. Ruppe wrote:
>
> With the null `a`, the offset to the static array is just 0 + whatever and the @safe mechanism can't trace that.
>
> So the arbitrary limit was put in place to make it more likely that such a situation will hit a protected page and segfault instead of carrying on. (most low addresses are not actually allocated by the OS... though there's no reason why they couldn't, it just usually doesn't, so that 16 MB limit makes the odds of something like this actually happening a lot lower)
>
> I don't recall exactly when this was discussed but it came up in the earlier days of @safe, I'm pretty sure it worked before then.

If that's the case I would consider this 16MB limit unnecessary. Most operating systems put a guard page at the very bottom of the stack (which is usually 1MB - 4MB, usually 1MB on Linux). Either the array will hit that page during initialization or something else during the execution.

Let's say someone puts a 15MB array on the stack, then we will have a page fault instead for sure and this artificial limit there for nothing. With 64-bits or more and some future crazy operating system, it might support large stack sizes like 256MB. This is a little like a 640kB limit.

On Tuesday, 21 July 2020 at 13:16:44 UTC, IGotD- wrote:
> Either the array will hit that page during initialization or something else during the execution.

But the array isn't initialized in the justification scenario. It is accessed through a null pointer and the type system thinks it is fine because it is still inside the static limit.

At run time, the cpu just sees access to memory address 0 + x, and if x is sufficient large, it can bypass those guard pages.

On 7/21/20 8:34 AM, Adam D. Ruppe wrote:

> The others aren't wrong about stack size limits playing some role, but the primary reason is that it is a weird hack for @safe, believe it or not.
...
> I don't recall exactly when this was discussed but it came up in the earlier days of @safe, I'm pretty sure it worked before then.

I think this was discussed, but was not the reason for the limitation. The limitation exists even in D1, which is before @safe: https://digitalmars.com/d/1.0/arrays.html#static-arrays

I have stressed before that any access of a pointer to a large object in @safe code should also check that the base of the object is not within the null page (this is not currently done). This is the only way to ensure safety.

-Steve