Thread overview
OutOfMemoryError in D DLL appending to module-level array
May 02, 2021
cc
May 02, 2021
cc
May 02, 2021
Adam D. Ruppe
May 02, 2021
cc
May 02, 2021

Ordinarily, it seems legal to append to an array that has been declared at module level (or as a static class member) that hasn't been otherwise initialized, for example:

class Foo {}
private Foo[] cache;
void main() {
	auto foo = new Foo();
	cache ~= foo;
}

However, when building code like this as a DLL, such as:

class Foo {}
private Foo[] cache;
extern(C) export Foo createFoo() {
	auto foo = new Foo();
	cache ~= foo;
	return foo;
}

and then calling it from another application (in this case, C#), I get core.exception.OutOfMemoryError@src\core\exception.d(647): Memory allocation failed at the cache ~= foo; line.

I was able to get around this by adding:

static this() {
	cache.length = 0;
}

which seems to fix it, but I'm not entirely sure what's going on, if this is expected behavior, if that's the correct way to handle it, and so on. Does it have something to do with the D runtime being initialized differently in a DLL versus a statically linked program? I am calling Runtime.initialize() as expected when the DLL is attached.

May 02, 2021

On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:

>

[...]

Just to add, only appending to the array seems to give OutOfMemoryErrors. I can idup strings, call stdc malloc, etc just fine.

May 02, 2021

On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:

>

which seems to fix it, but I'm not entirely sure what's going on, if this is expected behavior, if that's the correct way to handle it, and so on.

Oh I've been working on this the last couple weeks and having a hard time reproducing outside the work application. In the work app, the GC wasn't scanning the dll's TLS variables and freeing them prematurely.

In a sample test program, I used a thing kinda like yours, if a dll creates a thread and calls back into the exe you get a separate problem of partially initialize data.

D dlls on Windows work in simple cases right now but break down in more advanced cases. The good news is there's major fixes coming soon - my druntime hack might be coming, gdc is getting full dll support very soon from mingw, there's a good chance ldc is going to in a release or two as well outside mingw.

But the bad news is none of that is actually out right now, so dll + tls variables (which includes the top-level things on modules) are potentially buggy among other things like duplicated symbols.

You might find some improvement making your variable __gshared there.

But if you can do any reduced test case I'd really appreciate it. More tests that we can do in public is better!

May 02, 2021

On Sunday, 2 May 2021 at 02:42:46 UTC, Adam D. Ruppe wrote:

>

On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:

>

which seems to fix it, but I'm not entirely sure what's going on, if this is expected behavior, if that's the correct way to handle it, and so on.

Oh I've been working on this the last couple weeks and having a hard time reproducing outside the work application. In the work app, the GC wasn't scanning the dll's TLS variables and freeing them prematurely.

In a sample test program, I used a thing kinda like yours, if a dll creates a thread and calls back into the exe you get a separate problem of partially initialize data.

D dlls on Windows work in simple cases right now but break down in more advanced cases. The good news is there's major fixes coming soon - my druntime hack might be coming, gdc is getting full dll support very soon from mingw, there's a good chance ldc is going to in a release or two as well outside mingw.

But the bad news is none of that is actually out right now, so dll + tls variables (which includes the top-level things on modules) are potentially buggy among other things like duplicated symbols.

You might find some improvement making your variable __gshared there.

But if you can do any reduced test case I'd really appreciate it. More tests that we can do in public is better!

Cool, thanks for the update. Setting it as __gshared does seem to work. I put together some test cases here:

https://gitlab.com/-/snippets/2114152

It's got the DLL written in D, and test programs for loading it in D, C#, and C++. I haven't done much .NET interop stuff but it seems to work. I'd welcome any recommendations on how to improve the interfaces if there are any, I made a little mixin to create C wrappers for the member functions since that seems to be the suggested solution for calling class methods.