How can I link my dmd-compiled program with a specific version of the druntime?

I need this when experimenting with a new GC.

On Sunday, 30 September 2018 at 19:03:17 UTC, Per Nordlöw wrote:
> How can I link my dmd-compiled program with a specific version of the druntime?

druntime is within libphobos. So you must change this.

>
> I need this when experimenting with a new GC.

Did you try what i proposed earlier ? Until the handlers are plugged there can be a fallback to the manual allocs.

For example you start with the manual implementation and add handlers + fallback for every functions, like here for malloc

```
    __gshared void* function(size_t, uint, const TypeInfo) nothrow mallocHandler;
    void* malloc(size_t size, uint bits, const TypeInfo ti) nothrow
    {
        if (mallocHandler) // experimental stuff
        {
            return mallocHandler(size, bits, ti);
        }
        else // fallback until handler is assigned
        {
            void* p = cstdlib.malloc(size);
            if (size && p is null)
                onOutOfMemoryError();
            return p;
        }
    }
```

this way you can very easily change-compile-test, without recompiling the whole runtime and phobos each time.

On Sunday, 30 September 2018 at 19:53:02 UTC, Basile B. wrote:
> this way you can very easily change-compile-test, without recompiling the whole runtime and phobos each time.

Ok, thanks.

Is it possible to register an extra GC in a separate program by overriding the logic in `gc.proxy` in druntime?

That would be the most effective way.

On Monday, 1 October 2018 at 07:17:59 UTC, Per Nordlöw wrote:
> On Sunday, 30 September 2018 at 19:53:02 UTC, Basile B. wrote:
>> this way you can very easily change-compile-test, without recompiling the whole runtime and phobos each time.
>
> Ok, thanks.
>
> Is it possible to register an extra GC in a separate program by overriding the logic in `gc.proxy` in druntime?
>
> That would be the most effective way.

I think so. Apparently it's registered with a string, e.g "manual" and you pass a special druntime option with your program to select.
Actually i would be interested to make the interface with assignable handlers since you don't seem to be very hot. Maybe tomorrow i can try. I'm almost sure that this could work but cant be 100% sure. Maybe there'll be issues with privacy and events to assign.

On Monday, 1 October 2018 at 08:27:54 UTC, Basile B. wrote:
> I think so. Apparently it's registered with a string, e.g "manual" and you pass a special druntime option with your program to select.
> Actually i would be interested to make the interface with assignable handlers since you don't seem to be very hot. Maybe tomorrow i can try. I'm almost sure that this could work but cant be 100% sure. Maybe there'll be issues with privacy and events to assign.

Be my guest :)

Thanks!

On Monday, 1 October 2018 at 11:10:07 UTC, Per Nordlöw wrote:
> On Monday, 1 October 2018 at 08:27:54 UTC, Basile B. wrote:
>> I think so. Apparently it's registered with a string, e.g "manual" and you pass a special druntime option with your program to select.
>> Actually i would be interested to make the interface with assignable handlers since you don't seem to be very hot. Maybe tomorrow i can try. I'm almost sure that this could work but cant be 100% sure. Maybe there'll be issues with privacy and events to assign.
>
> Be my guest :)
>
> Thanks!

I see other related topics, did you already start something ?

On Tuesday, 2 October 2018 at 13:07:04 UTC, Basile B. wrote:
> On Monday, 1 October 2018 at 11:10:07 UTC, Per Nordlöw wrote:
>> On Monday, 1 October 2018 at 08:27:54 UTC, Basile B. wrote:
>>> I think so. Apparently it's registered with a string, e.g "manual" and you pass a special druntime option with your program to select.
>>> Actually i would be interested to make the interface with assignable handlers since you don't seem to be very hot. Maybe tomorrow i can try. I'm almost sure that this could work but cant be 100% sure. Maybe there'll be issues with privacy and events to assign.
>>
>> Be my guest :)
>>
>> Thanks!
>
> I see other related topics, did you already start something ?

This works https://github.com/BBasile/druntime/pull/1. Not sure if it will be useful.

On Tuesday, 2 October 2018 at 16:20:52 UTC, Basile B. wrote:
> This works https://github.com/BBasile/druntime/pull/1. Not sure if it will be useful.

Ahh, thanks!

I've just found my own way of iterating via a script at

    https://github.com/nordlow/scripts/blob/master/dmd-own

that (re)compiles druntime and phobos and then compiles my GC test application using toolchain in around 7 secs on my 2 year old laptop. I'll stick to that for now.

This assumes the home-directory structure

~/Work/dmd ~master
       druntime ~dmitry-gc
       phobos ~master

. The playground for my GC experiments will be

https://github.com/nordlow/druntime/blob/dmitry-gc/src/gc/impl/dmitry/gc.d

So far I've only described my plan and added a set of debug prints.

You're very welcome to comment and destroy my plan. I'm uncertain of what kinds of locking (if any) that is needed for a thread-local GC allocation (non-mutex I suppose). Please elaborate on the subject if you have any experience with thread-local GCs.

Would you be interested in making this a druntime PR so you can make comments?

Contents of `dmd-own` follows:

#!/usr/bin/env bash

function dmd-own_fn()
{
    local DLANG_SRC_ROOT=${HOME}/Work
    local DMD_ROOT=${DLANG_SRC_ROOT}/dmd
    local DRUNTIME_ROOT=${DLANG_SRC_ROOT}/druntime
    local PHOBOS_ROOT=${DLANG_SRC_ROOT}/phobos
    local BUILD="debug"

    if type clang++ &> /dev/null; then
        local HOST_CXX=clang++
    else
        local HOST_CXX=g++-8
    fi
    HOST_CXX=g++-8 # use g++ for now because building DMD with clang++ seems to generate a dmd binary that segfauls

    # rebuild dmd, druntime, phobos on Linux
    command make -f posix.mak BUILD=${BUILD} -C ${DMD_ROOT} HOST_CXX=${HOST_CXX} > /dev/null 2> /dev/null
    # command make -f posix.mak BUILD=${BUILD} -C ${DRUNTIME_ROOT} > /dev/null
    command make -f posix.mak BUILD=${BUILD} -C ${PHOBOS_ROOT} > /dev/null

    if [ $# = 0 ]; then
        echo -e "Usage: $FUNCNAME D_SOURCE_FILE ARG
    Example: dmd-own gctester.d --DRT-gcopt=gc:dmitry"
    else
        local FILE="$1"
        local out=$(mktemp)
        local ARG="$2"
        local NEW_DMD=${DMD_ROOT}/generated/linux/${BUILD}/64/dmd
        ${NEW_DMD} -debug -unittest -wi -vcolumns ${PWD}/${FILE} -of$out
        $out ${ARG}
    fi
}

dmd-own_fn "$@"

On Tuesday, 2 October 2018 at 13:07:04 UTC, Basile B. wrote:
> On Monday, 1 October 2018 at 11:10:07 UTC, Per Nordlöw wrote:
>> On Monday, 1 October 2018 at 08:27:54 UTC, Basile B. wrote:
>>> I think so. Apparently it's registered with a string, e.g "manual" and you pass a special druntime option with your program to select.
>>> Actually i would be interested to make the interface with assignable handlers since you don't seem to be very hot. Maybe tomorrow i can try. I'm almost sure that this could work but cant be 100% sure. Maybe there'll be issues with privacy and events to assign.
>>
>> Be my guest :)
>>
>> Thanks!
>
> I see other related topics, did you already start something ?

Yes!

I'm making interesting progress here:
https://github.com/nordlow/druntime/blob/fastalloc-gc/src/gc/impl/fastalloc/gc.d

I'm currently writing the spec in a comment and experimenting with faster (both global and thread-local) allocations

using

https://github.com/nordlow/phobos-next/blob/master/snippets/gctester.d

as a benchmark, compiled with

https://github.com/nordlow/scripts/blob/master/dmd-own

which gives

[per:~/Work/knet/phobos-next/snippets] 12s $ dmd-own gctester.d --DRT-gcopt=gc:fastalloc
 size new-C new-S GC.malloc gc_tlmalloc_N GC.calloc malloc calloc FreeList!(GCAllocator)
    8  46.8  58.3    26.0        7.5        23.8     31.2   31.3   29.1
   16  33.6  23.1    12.7        4.7        13.0     16.5   14.9   16.0
   32  15.5  13.5     6.7        2.9         7.0      8.2   10.2    9.6
   64  11.5   9.7     4.1        1.8         3.9      6.4    5.2    4.8
  128   9.3   6.9     2.6        1.6         2.5      4.2    3.8    3.0
  256   8.8   5.4     1.9        1.4         1.9      3.2    2.9    2.2
  512   7.6   4.3     1.6        1.3         1.5      2.6    2.8    1.7
 1024   7.3   4.1     1.5        1.3         1.4      2.2    2.5    1.5
  ns/w: nanoseconds per word

vs

[per:~/Work/knet/phobos-next/snippets] 4s $ dmd-own gctester.d --DRT-gcopt=gc:conservative
 size new-C new-S GC.malloc gc_tlmalloc_N GC.calloc malloc calloc FreeList!(GCAllocator)
    8  75.4  54.7    39.1        8.9        42.1     28.8   32.1   39.6
   16  32.0  27.5    18.6        4.6        20.9     15.2   17.0   20.1
   32  15.7  16.0    10.5        2.9        12.3      9.0   10.7   11.0
   64  10.2   9.0     7.7        1.9         6.2      5.3    6.3    6.6
  128   7.9   6.0     5.0        1.5         4.2      4.5    3.8    4.8
  256   6.1   4.7     2.9        1.3         3.4      3.2    2.9    3.6
  512   5.6   3.5     3.0        1.3         3.0      2.6    2.8    2.8
 1024   5.0   2.9     2.5        1.2         2.6      2.3    2.5    2.5
  ns/w: nanoseconds per word

Note that gc_tlmalloc_N uses the new allocator in both cases. I haven't bothered branching the benchmark on type of GC config.

I'm planning on making it sweep-free as described in

https://github.com/golang/proposal/blob/master/design/12800-sweep-free-alloc.md