[GSoC] 'Independency of D from the C Standard Library' progress and update thread

I'm moving forward with the D implementations of the C parts that the D Runtime
uses. A not-so-small description of the project. I hope it will
be descriptive enough to clear things up for people, as I probably did
not do a good job in the previous public discussions about this project.

    The goal of this project is to remove the dependency of the D Runtime
    from the C Standard Library. Currently, the D Runtime uses a small part
    of the C Standard Library. That is:
        a) The string.h family of functions: memcpy(), memmove(), etc.
        b) The standard allocator functions: malloc(), free(), etc.

    Those don't justify the dependency on the C Standard Library, as a very
    small part of it is utilized. However, there are problems coming with it:
        1) C’s implementations are not type-safe and memory-safe.
        2) C’s implementations have accumulated a lot of cruft over the years.
        3) Cross-compiling is more difficult as now one should have available and
        configured a C runtime and toolchain apart from the D runtime. This
        makes it difficult for D to create freestanding software.

    So, this project will provide alternative implementations of this functions,
    dependent only in the D Runtime. We hope that in the process, we will
    leverage D features that C doesn't have:
        1) Type-safety and memory safety (bounds-checking etc.)
        2) Templates to branch to an optimal implementation at compile-time.
        3) Inlining, as the branching in C happens at runtime.
        4) Compile-Time Function Execution (CTFE) and introspection (type info).

    Important clarifications:
        1) It will not use the C Standard Library.
        2) C Standard Library will be still available.
        3) We target the D Runtime and not the user (although, of course the
        users will be able to use it).
        4) We will provide a different interface from the C implementations,
        with the prospect to be more idiomatic D.
        5) Same or better performance with libc is not a hard constraint. We
        might succeed to reach it, but we might not. The hard constraint is that
        it will at least be close.

This month
==========

Implementation of string.h family:
    -- Week 1-2: Handling mis-alignment in memcpy().
    -- Week 3: memmove()
    -- Week 4: memset()

As a starting point, I reached the same performance with libc memcpy for (big) aligned data (hopefully, the implementation will be part of the memcpyD in some time).
Now, along with Mike Franklin's previous work, memcpyD() is faster than libc memcpy for small data (less than 32768) and as fast for big data.

Next month
==========
Mike Franklin initially proposed the idea that it we may be able to do
something better than implementing malloc() and free() again, just in D.
So, we decided that the best option is to integrate the std.experimental.allocator
to D Runtime. That involves creating an allocator of its building blocks
and removing the dependency in Phobos.

Week 1: Create a basic allocator using std.experimental.allocator interface
Week 2: Replace malloc(), free(), realloc() in D runtime
Week 3: Re-iterate until we have good benchmarks.
Week 4: Remove Phobos dependencies from the allocator.

Blockers
========
Not any major one.

On Friday, 31 May 2019 at 21:01:01 UTC, Stefanos Baziotis wrote: > I'm moving forward with the D implementations of the C parts that the D Runtime > uses. Hi Stefanos, good project :) Here's something to consider if you're replacing malloc() et al: it's popular (especially with large server deployments) to tune application memory allocation performance by replacing libc malloc() with alternatives such as tcmalloc and jemalloc. That works because they use the same libc malloc() API but with a different implementation, injected at link or load time (using LD_PRELOAD or something). It would be great if D code can still take advantage of alternative allocators developed by third-parties who may or may not be writing for D.

On Friday, 31 May 2019 at 21:01:01 UTC, Stefanos Baziotis wrote: > The goal of this project is to remove the dependency of the D Runtime > from the C Standard Library. Cool project! Is it possible to follow the project somewhere on github?

On Saturday, 1 June 2019 at 02:40:10 UTC, sarn wrote: > On Friday, 31 May 2019 at 21:01:01 UTC, Stefanos Baziotis wrote: >> I'm moving forward with the D implementations of the C parts that the D Runtime >> uses. > > Hi Stefanos, good project :) > Thank you! > Here's something to consider if you're replacing malloc() et al: it's popular (especially with large server deployments) to tune application memory allocation performance by replacing libc malloc() with alternatives such as tcmalloc and jemalloc. That works because they use the same libc malloc() API but with a different implementation, injected at link or load time (using LD_PRELOAD or something). > > It would be great if D code can still take advantage of alternative allocators developed by third-parties who may or may not be writing for D. As you can see in the "Next Month" above, we're planning to replace malloc() et al but with a different interface. The reason is that we believe that it is idiomatic D this way (I personally also believe that malloc(), free() etc. have a bad interface for allocation). We even hope that in the end (probably after GSoC) the allocator will be typed. But the allocators you proposed might be an inspiration for the allocator I will build using the std.experimental.allocator interface. Moreover, let me stress that malloc(), free().. will be available as well.

On Saturday, 1 June 2019 at 05:59:05 UTC, Walter Bright wrote: > Note that DMC++'s standard library is now Boost licensed, and so can be used. > > https://github.com/DigitalMars/dmc/tree/master/src Thanks, I'll take a look. Do we have any benchmarks?

On Saturday, 1 June 2019 at 09:29:55 UTC, Thomas Mader wrote: > On Friday, 31 May 2019 at 21:01:01 UTC, Stefanos Baziotis wrote: >> The goal of this project is to remove the dependency of the D Runtime >> from the C Standard Library. > > Cool project! Thanks! > Is it possible to follow the project somewhere on github? For now, you can follow this repo: https://github.com/JinShil/memcpyD related to memcpy. It's my mistake that I haven't forked that and do the changes with PRs, so you can see a commit "... from Stefanos". I'll post an update about where my experimentation will be visible.

On Saturday, 1 June 2019 at 05:59:05 UTC, Walter Bright wrote: > Note that DMC++'s standard library is now Boost licensed, and so can be used. > > https://github.com/DigitalMars/dmc/tree/master/src Do you think it is planned to remove all C++ dependencies from D as well?

On Saturday, 1 June 2019 at 19:50:08 UTC, Aurélien Plazzotta wrote: > On Saturday, 1 June 2019 at 05:59:05 UTC, Walter Bright wrote: >> Note that DMC++'s standard library is now Boost licensed, and so can be used. >> >> https://github.com/DigitalMars/dmc/tree/master/src > > Do you think it is planned to remove all C++ dependencies from D as well? If I understand you correctly, this has already happened. DMD can be built today without the need for any C++ as it's written in 100% D.

Forums