On Wednesday, 24 August 2016 at 09:38:43 UTC, Walter Bright wrote:
>> 1) Why? I know everything in compile-time, why force me to (A) allocate it
>> separately in `shared static this()`
>
> I'm not sure how that hurts anything. It's just a call to malloc().

Static this()es require a topological order on modules, or it blows up when running the module ctors. We've had many issues with static-this()es that required us to split modules into "top and bottom parts", which only serve breaking cycles for the ctors (i.e., not a logical compilation unit, but a technical one).

100% agree this limit should be removed. We do have a workaround with dynamic arrays, but that doesn't justify not fixing the problem.

On 8/24/2016 3:35 AM, Lodovico Giaretta wrote:
> On Wednesday, 24 August 2016 at 09:38:43 UTC, Walter Bright wrote:
>> I don't know why pointers cannot be used.
>
> In my code I have a line like this:
>
> static immutable MyStruct[] data = [ MyTemplate!MyArgs ];
>
> This would not work if `MyStruct.sizeof * MyTemplate!MyArgs.length` is bigger
> than 16MB, right? And for this case there's no workaround, I think...

The compiler will run out of memory long before you've got 16 million array initializers.

On 8/24/2016 3:55 AM, Tomer Filiba wrote:
> On Wednesday, 24 August 2016 at 09:38:43 UTC, Walter Bright wrote:
>>> 1) Why? I know everything in compile-time, why force me to (A) allocate it
>>> separately in `shared static this()`
>>
>> I'm not sure how that hurts anything. It's just a call to malloc().
>
> Static this()es require a topological order on modules, or it blows up when
> running the module ctors. We've had many issues with static-this()es that
> required us to split modules into "top and bottom parts", which only serve
> breaking cycles for the ctors (i.e., not a logical compilation unit, but a
> technical one).

Splitting them, as you say, works, as well as simply creating a function that does it and calling it when main() starts.

On 8/24/2016 3:35 AM, Tomer Filiba wrote:
> On Wednesday, 24 August 2016 at 09:38:43 UTC, Walter Bright wrote:
>>> 2) Many times I need memory-contiguity, e.g., several big arrays inside a
>>> struct, which is dumped to disk/sent over network. I can't use pointers there.
>>
>> I don't know why pointers cannot be used. Can you show the struct definition
>> you're using?
>
> Our configuration is a struct of several static hash tables (allocated in-place,
> not via GC). So the entire configuration is contiguous is memory, which allows
> us to dump/load/send it easily.
>
> When we increase the capacity of these tables we run into these pain-in-the-ass
> 16MB limits. So although the struct itself is over 16MB, no single table can
> cross several thousand entries, as the static arrays it uses internally would
> overflow that boundary.
>
> The configuration itself may very well be dynamically allocated (e.g., not a
> global variable) but that won't solve anything as the restriction is on the
> *type* of the array.


If I understand you correctly, removing the size limitation on the type will resolve the issue for you? Even though allocating static data of such a size will still not be allowed?

BTW, given globals in C++:

   int a[100];
   float b[200];
   long c;

there actually is no guarantee that they are allocated contiguously, and indeed I've run into bugs in my code because I had relied on that. They'd have to be put into a struct to get a guarantee.

Maybe you can merge this: https://github.com/dlang/dmd/pull/6081

On Wednesday, 24 August 2016 at 18:16:01 UTC, Walter Bright wrote:
> On 8/24/2016 3:35 AM, Tomer Filiba wrote:
...
>> Our configuration is a struct of several static hash tables (allocated in-place, not via GC). So the entire configuration is contiguous in memory
...
> If I understand you correctly, removing the size limitation on the type will resolve the issue for you? Even though allocating static data of such a size will still not be allowed?

(1) Why won't allocating such static data be allowed? Gold has no such limitations
(2) Why do linker limitations have to do with the inner working on types?
(3) I may very well allocate the entire config using malloc, or on the stack (given a huge stack). The linker has nothing to do with it

> BTW, given globals in C++:
>
>    int a[100];
>    float b[200];
>    long c;
>
> there actually is no guarantee that they are allocated contiguously, ... They'd have to be put into a struct to get a guarantee.

As I said (quoted above), the configuration is a struct. It looks like

struct Config {
    Table!(K1, V1, 10000) myTable;
    Table!(K2, V2, 10000) yourTable;
    Table!(K3, V3, 10000) hisTable;
    Table!(K4, V4, 10000) herTable;
}

So it is surely contiguous.

Table is a static hash table built on top of a static array, since it has a known capacity. When `V1.sizeof * 10_000 > 16MB`, it fails to compile

Btw, I set declare everything with `field = void` to prevent the struct from having a huge init symbol. I take care of initialization in runtime.

-tomer