On Thursday, 25 October 2012 at 17:17:01 UTC, Rob T wrote:
> Yes I can build my own D shared libs, both as static PIC (.a) and dynamically loadable (.so). however I cannot statically link my shared libs to druntime + phobos as-is. The only way I can do that, is to also compile druntime + phobos into PIC, which can be done as a static PIC lib.

Sorry, I keep forgetting that this is needed on non-Windows systems.

> So what you are saying is that I can statically link PIC compiled druntime to my own shared lib, but I cannot build druntime as a dynamically loadable shared lib? I can see why thatmay work, if each shared lib has it's own private compy of the GC. Correct?

Yes, this is possible.

Sending references to GC memory between the D modules then has the same rules as when sending it to non-D code, unless the host (the loader module) uses druntime to load the other modules, in which case it can in principle share the same GC with them.

> I recall that druntime may have some ASM code that will not work when compiled to PIC. I think gdc removed the offending ASM code, but it may still be present in the dmd version, but I don't know for sure.

I think it was relatively recently that DMD could also compile the runtime as PIC, but I might be remembering wrong.

> Another question is if I can link a dynamic loadable D lib to C/C++ code or not? Yes I can do it, and it seems to work, but I was told that the GC will not necessarily work. Am I misunderstanding this part?

The GC will work the same as usual inside the D code, but you have to manually keep track of references you send outside the scope of the GC, such as references to GC memory put on the C heap.

This can be done with the GC.addRoot/removeRoot and GC.addRange/removeRange functions found in core.memory, or by retaining the references in global, TLS or GC memory.

It's good practice to do this for all GC references sent to external code, as you don't know where the reference may end up.

Of course, you have other options. You don't have to send references to GC memory to external code, you can always copy the data over to a different buffer, such as one on the C heap (i.e. malloc()).

If the caller (in the case of a return value) or the callee (in the case of a function argument) expects to be able to call free() on the memory referenced, then you must do it this way regardless.

On Thursday, 25 October 2012 at 17:17:01 UTC, Rob T wrote:
> Yes I can build my own D shared libs, both as static PIC (.a) and dynamically loadable (.so). however I cannot statically link my shared libs to druntime + phobos as-is. The only way I can do that, is to also compile druntime + phobos into PIC, which can be done as a static PIC lib.

Sorry, I keep forgetting that this is needed on non-Windows systems.

> So what you are saying is that I can statically link PIC compiled druntime to my own shared lib, but I cannot build druntime as a dynamically loadable shared lib? I can see why thatmay work, if each shared lib has it's own private compy of the GC. Correct?

Yes, this is possible.

Sending references to GC memory between the D modules then has the same rules as when sending it to non-D code, unless the host (the loader module) uses druntime to load the other modules, in which case it can in principle share the same GC with them.

> I recall that druntime may have some ASM code that will not work when compiled to PIC. I think gdc removed the offending ASM code, but it may still be present in the dmd version, but I don't know for sure.

I think it was relatively recently that DMD could also compile the runtime as PIC, but I might be remembering wrong.

> Another question is if I can link a dynamic loadable D lib to C/C++ code or not? Yes I can do it, and it seems to work, but I was told that the GC will not necessarily work. Am I misunderstanding this part?

The GC will work the same as usual inside the D code, but you have to manually keep track of references you send outside the scope of the GC, such as references to GC memory put on the C heap.

This can be done with the GC.addRoot/removeRoot and GC.addRange/removeRange functions found in core.memory, or by retaining the references in global, TLS or GC memory.

It's good practice to do this for all GC references sent to external code, as you don't know where the reference may end up.

Of course, you have other options. You don't have to send references to GC memory to external code, you can always copy the data over to a different buffer, such as one on the C heap (i.e. malloc()).

If the caller (in the case of a return value) or the callee (in the case of a function argument) expects to be able to call free() on the memory referenced, then you must do it this way regardless.

On Thursday, 25 October 2012 at 17:20:40 UTC, Rob T wrote:
> On Thursday, 25 October 2012 at 02:15:41 UTC, Jakob Ovrum wrote:
>> You can even share the same GC between dynamic libraries and the host application  (if both are D and use GC, of course) using the GC proxy system.
>
> What is the GC proxy system, and how do I make use of it?
>
> --rt

There's a function Runtime.loadLibrary in core.runtime that is supposed to load a shared library and get the symbol named `gc_setProxy` using the platform's dynamic library loading routines, then use that to share the host GC with the loaded library.

I say "is supposed to" because I checked the code and it's currently a throwing stub on POSIX systems, it's only implemented for Windows (the source of the function can be found in rt_loadLibrary in rt/dmain2.d of druntime).

When it comes to gc_setProxy - GDC exports this symbol by default on Windows, while DMD doesn't. I don't know why this is the case. I haven't built shared libraries on other OS' before so I don't know how GDC and DMD behave there.

I use this GC thread to show a little GC-related benchmark.

A little Reddit thread about using memory more compactly in Java:

http://www.reddit.com/r/programming/comments/120xvf/compact_offheap_structurestuples_in_java/

The relative blog post:
http://mechanical-sympathy.blogspot.it/2012/10/compact-off-heap-structurestuples-in.html

So I have written a D version, in my test I have reduced the amount of memory allocated (NUM_RECORDS = 10_000_000):
http://codepad.org/IhHjqUua

With this lower memory usage the D version it's more than twice faster than the compact Java version that uses the same NUM_RECORDS (0.5 seconds against 1.2 seconds each loop after the first two ones).

In D I have improved the loops, I have used an align() and a minimallyInitializedArray, this is not too much bad.

But in the main() I have also had to use a deprecated "delete", because otherwise the GC doesn't deallocate the arrays and the program burns all the memory (setting the array to null and using GC.collect() isn't enough). This is not good.

Bye,
bearophile

On Friday, 26 October 2012 at 14:21:51 UTC, bearophile wrote:
> But in the main() I have also had to use a deprecated "delete", because otherwise the GC doesn't deallocate the arrays and the program burns all the memory (setting the array to null and using GC.collect() isn't enough). This is not good.
>

Is this happening with dmd 2.060 as released?

Rob T:

> Is this happening with dmd 2.060 as released?

I'm using 2.061alpha git head, but I guess the situation is the same with dmd 2.060. The code is linked in my post, so trying it is easy, it's one small module.

Bye,
bearophile

On Friday, 26 October 2012 at 23:10:48 UTC, bearophile wrote:
> Rob T:
>
>> Is this happening with dmd 2.060 as released?
>
> I'm using 2.061alpha git head, but I guess the situation is the same with dmd 2.060. The code is linked in my post, so trying it is easy, it's one small module.
>
> Bye,
> bearophile

I tried it with dmd 2.60 (released), and gdc 4.7 branch. I tried to check if memory was being freed by creating a struc destructor for JavaMemoryTrade, but that did not work as expected, leading me down the confusing and inconsistent path of figuring out why destructors do not get called when memory is freed.

Long story short, I could not force a struct to execute its destructor if it was allocated on the heap unless I used delete. I tried destroy and clear, as well as GC.collect and GC.free(), nothing else worked.

Memory heap management as well as struct destructors appear to be seriously broken.

--rt

On Saturday, 27 October 2012 at 01:03:57 UTC, Rob T wrote:
> On Friday, 26 October 2012 at 23:10:48 UTC, bearophile wrote:
>> Rob T:
>>
>>> Is this happening with dmd 2.060 as released?
>>
>> I'm using 2.061alpha git head, but I guess the situation is the same with dmd 2.060. The code is linked in my post, so trying it is easy, it's one small module.
>>
>> Bye,
>> bearophile
>
> I tried it with dmd 2.60 (released), and gdc 4.7 branch. I tried to check if memory was being freed by creating a struc destructor for JavaMemoryTrade, but that did not work as expected, leading me down the confusing and inconsistent path of figuring out why destructors do not get called when memory is freed.
>
> Long story short, I could not force a struct to execute its destructor if it was allocated on the heap unless I used delete. I tried destroy and clear, as well as GC.collect and GC.free(), nothing else worked.
>
> Memory heap management as well as struct destructors appear to be seriously broken.
>
> --rt

OK my bad, partially.

Heap allocated struct destructors will not get called using clear or destroy unless the struct reference is manually dereferenced. I got confused that class references behave differently than heap allocated struct references. I cannot be the first person to do this, and it must happen all the time. The auto dereferencing of a struc pointer when accessing members may be nice, but it makes struct pointers look exactly like class references, which will lead to mistakes.

I do get the concept between classes and structs, but why does clear and destroy using a struct pointer not give me a compiler error or at least a warning? Is there any valid purpose to clear or destroy a pointer that is not dereferenced? Seems like a bug to me.

--rt

On Saturday, 27 October 2012 at 01:03:57 UTC, Rob T wrote:
> On Friday, 26 October 2012 at 23:10:48 UTC, bearophile wrote:
>> Rob T:
>>
>>> Is this happening with dmd 2.060 as released?
>>
>> I'm using 2.061alpha git head, but I guess the situation is the same with dmd 2.060. The code is linked in my post, so trying it is easy, it's one small module.
>>
>> Bye,
>> bearophile
>
> I tried it with dmd 2.60 (released), and gdc 4.7 branch. I tried to check if memory was being freed by creating a struc destructor for JavaMemoryTrade, but that did not work as expected, leading me down the confusing and inconsistent path of figuring out why destructors do not get called when memory is freed.
>
> Long story short, I could not force a struct to execute its destructor if it was allocated on the heap unless I used delete. I tried destroy and clear, as well as GC.collect and GC.free(), nothing else worked.
>
> Memory heap management as well as struct destructors appear to be seriously broken.
>
> --rt

I made a mistake. The clear and destroy operations require that a pointer to a struc be manually dereferenced. What I don't understand is why the compiler allows you to pass a -not- dereferenced pointer to clear and destroy, this looks like a bug to me. It should either work just like a class reference does, or it should refuse to compile.

I'm sure you've heard this many times before, but I have to say that it's very confusing when struct pointers behave exactly like class references, but not always.

--rt

> But in the main() I have also had to use a deprecated "delete",

And setting trades.length to zero and then using GC.free() on its ptr gives the same good result.

Bye,
bearophile