On Thursday, 4 April 2019 at 13:56:08 UTC, Stefanos Baziotis wrote:
> [GSoC 2019] Interested in Baremetal D Runtime and project ideas
>
> Hello,
>
> I'm Stefanos Baziotis and I'm both new to D and GSoC.
> I'm an undergraduate computer science student in Greece, and
> I would like to contribute to the D programming language.
>
> Unfortunately, probably I came late on the GSoC as I learned about it just
> 2 days before, so any info about it and/or D's selection process would be greatly appreciated.
>
> I heard about D probably a month ago from a friend, and I thought I would
> give it a try as I am not 100% satisfied with C, C++. I have to say, that was
> a really fun month. :)
> After about 3000 lines (of mostly writing a not-yet-finished compiler front-end), it was very exciting!
>
> I'm mainly interested on the Baremetal D runtime project, as I am mostly comfortable
> with low-level C programming. I have developed some allocators [1]
> and have some SIMD experience as I have written a small image convolution MPI
> program that uses AVX extensions [2].
>
> I'm interested in other projects as well, specifically compiler-related and the persistent data structures.
>
> I'm very enthusiastic to learn a lot about any of these projects, but I would like to
> ask: What is the expected / required experience / knowledge / skills?
> Also, is this the best place to discuss about this subject?
>
> Best regards,
> Stefanos Baziotis
>
>
> [1] https://github.com/baziotis/Allocators
> [2] https://github.com/baziotis/2D-Image-Convolution-MPI-SIMD

Hi Stefanos,

Very interesting, it would be nice to have SIMD convolution algorithms in D.
SIMD extensions actively used in mir-glas (experimental D BLAS implementation), a D's FFT library should use SIMD as well.

The same for low-level BetterC projects. Many Mir libraries were created to be used from C/C++/Python.

We need a few D language extensions:
 1. multiple auto ref values
 2. more clever alias syntax for templates (it is required for data science libraries in D)
 3. opApply and foreach API design rework (requires 1.)

Each of them requires a DIP (D Improvement Proposal) and Pull Request to DMD with required changes.

As a small bonus, I will pay an additional $400 for each D language extension merged to DMD.

https://github.com/libmir

Let me know if you would like to work during GSoC on one of the related directions and especially DIPs.

ilyayaroshenko at google mai

Best regards,
Ilya

On Thursday, 4 April 2019 at 13:56:08 UTC, Stefanos Baziotis wrote:
> [GSoC 2019] Interested in Baremetal D Runtime and project ideas
>
> Hello,
>
> I'm Stefanos Baziotis and I'm both new to D and GSoC.
> I'm an undergraduate computer science student in Greece, and
> I would like to contribute to the D programming language.
>
> Unfortunately, probably I came late on the GSoC as I learned about it just
> 2 days before, so any info about it and/or D's selection process would be greatly appreciated.
>
> I heard about D probably a month ago from a friend, and I thought I would
> give it a try as I am not 100% satisfied with C, C++. I have to say, that was
> a really fun month. :)
> After about 3000 lines (of mostly writing a not-yet-finished compiler front-end), it was very exciting!
>
> I'm mainly interested on the Baremetal D runtime project, as I am mostly comfortable
> with low-level C programming. I have developed some allocators [1]
> and have some SIMD experience as I have written a small image convolution MPI
> program that uses AVX extensions [2].
>
> I'm interested in other projects as well, specifically compiler-related and the persistent data structures.
>
> I'm very enthusiastic to learn a lot about any of these projects, but I would like to
> ask: What is the expected / required experience / knowledge / skills?
> Also, is this the best place to discuss about this subject?
>
> Best regards,
> Stefanos Baziotis
>
>
> [1] https://github.com/baziotis/Allocators
> [2] https://github.com/baziotis/2D-Image-Convolution-MPI-SIMD


Hi Stefanos,

Very interesting, it would be nice to have SIMD convolution algorithms in D.
SIMD extensions actively used in mir-glas (experimental D BLAS implementation), a D's FFT library should use SIMD as well.

The same for low-level BetterC projects. Many Mir libraries were created to be used from C/C++/Python.

We need a few D language extensions:
 1. multiple auto ref values
 2. more clever alias syntax for templates (it is required for data science libraries in D)
 3. opApply and foreach API design rework (requires 1.)

Each of them requires a DIP (D Improvement Proposal) and Pull Request to DMD with required changes.

As a small bonus, I will pay an additional $400 for each D language extension merged to DMD.

https://github.com/libmir

Let me know if you would like to work during GSoC on one of the related directions and especially DIPs.

ilyayaroshenko at google mai

Best regards,
Ilya

On Sunday, 7 April 2019 at 04:07:43 UTC, Mike Franklin wrote:
> I don't see that Andrei objects much to the C/C++ dependency.  Afterall, he recently approved adding the C++ standard library to the druntime: https://github.com/dlang/druntime/pull/2286

Yes, I don't mean Andrei specifically, I mean the ones who
proposed the idea. And even them, not for generally but
for embedded systems.

> And that's not to mention turtles-all-the-way-down safety and purity guarantees provided by the compiler.
>
> I'm not sure who posted the original GSoC idea, so they'd have to speak for themselves, but it appears to be a synthesis of things I have been advocating for for a long time.  It's not really about creating a bare metal runtime, it's mostly just about being able to use D for bare-metal programming in a pay-as-you-go fashion.

Mike, you give informative answers. To be honest, the more
I discuss this project idea, the more unclear what the goals
are becomes.

> Probably the best thing you can do is focus on your proposal and make it great.  I hope those reviewing the proposals will be able to see the relationship and natural progression with Dan's proposal and yours and both will be selected.

Thanks, but frankly I don't see any interest from the
community. That's ok as I (and all of us I guess)
will be better off if I work on a project
that the community is enthusiastic about.

>
> For the past 5 years this has been my experience working on D:  https://www.youtube.com/watch?v=AbSehcT19u0

Hahahahahahahahaha.. Honestly, that resonated and made me laugh
more than it should. If I had a penny for every time I got into
a project to do something other than memory management but
the memory management was so bad, that now instead of doing
the thing I was supposed to do, I had to redo most of the
mem management -.-

>
> If you're anxious to get your hands dirty, you can start by choosing a not-yet-converted runtime hook from https://wiki.dlang.org/Runtime_Hooks and have at it.  See https://github.com/dlang/druntime/pull/2508 for another recent example of another contributor getting busy.

That will probably be my first start.

>
> https://github.com/dlang/dmd/pull/9068 is a low barrier to entry task that would help us out a lot.
>
> You can also start stripping out utilities from Phobos (e.g. std.traits, std.meta, std.conv, etc.) to create a no-dependency utility library.  Basically my "Suggestion 2" here: https://forum.dlang.org/post/qooasdrdlrqdhghjwmus@forum.dlang.org
>  It could potentially serve as a starting point for std.v2 and the opt-in continuum.  See https://forum.dlang.org/post/mailman.7799.1553797857.29801.digitalmars-d@puremagic.com for the thread where that was first proposed.

I will have to ask again about this library, but surely, after I finish with the
GSoC proposals, I will give it a try.

- Stefanos

On Sunday, 7 April 2019 at 07:27:01 UTC, 9il wrote:

> Very interesting, it would be nice to have SIMD convolution algorithms in D.
I'm not an expert in image convolution, but any proposals
are welcome.

> We need a few D language extensions:
>  1. multiple auto ref values
>  2. more clever alias syntax for templates (it is required for data science libraries in D)
>  3. opApply and foreach API design rework (requires 1.)

Can you please explain more 1. and 2. ?

>
> As a small bonus, I will pay an additional $400 for each D language extension merged to DMD.

That's kind :). For me, certainly not needed though.

>
> Let me know if you would like to work during GSoC on one of the related directions and especially DIPs.

Yes.

> ilyayaroshenko at google mai

I will reach you by mail.

On Sunday, 7 April 2019 at 10:56:51 UTC, Stefanos Baziotis wrote:

> Mike, you give informative answers. To be honest, the more
> I discuss this project idea, the more unclear what the goals
> are becomes.

These are the goals as I see them:
1. Allow for the opt-in continuum described here:  https://forum.dlang.org/post/q7j4sl$17pe$1@digitalmars.com

2. Remove artificial dependencies.  IMO the current dependency D has on C is completely artificial.  D does not need C (https://forum.dlang.org/post/kemnbymcuwycglevtbox@forum.dlang.org).  It's simply utilizing C as a matter of convenience and expediency.  Many of C's implementations are not type and memory safe, and that is a disadvantage for D.  C's implementations can't be evaluated at compile-time, that is a disadvantage for D.

3. Detangle dependencies.  e.g. many of the runtime hooks don't need runtime TypeInfo; that information is available at compile-time.  This also applies to modules in Phobos.  e.g. std.traits, std.meta, std.conv, shouldn't require runtime support.

4. Embrace design by introspection (https://www.youtube.com/watch?v=tcyb1lpEHm0) all the way down the call stack.

5. Embrace D's compiler guarantees (type safety, memory safety, purity, etc.) all the way down the call stack

It is the opt-in continuum that enables bare-metal programming in D. (2)~(5) are just things that need to be done to get there.  We don't need -betterC; users can simply opt-in to the features they want by importing the modules they need, and no more.

Currently, the compiler assumes an OS is present.  It assumes the *entire* druntime library is complete and available at both compile-time and link time.  Well, at least it used to; v2.079 introduced some improvements (https://dlang.org/changelog/2.079.0.html#minimal_runtime).  We need to continue that work until (1) becomes a reality.

Converting C's software building-blocks to D is not absolutely necessary for bare-metal programming, but it will make for a more polished experience and will truly embrace the zen of D if the conversion were done.  I'm under the impression, also, that once converted to D, some D programs will out-perform their C counterparts.

I think the GSoC idea was ported from a number of different ideas proposed at the Symmetry Autumn of Code here:  https://wiki.dlang.org/SAOC_2018_ideas#Re-implement_Software_Building_Blocks_.28e.g._memcpy.2C_memset.2C_memcmp.2C_malloc.2C_free.2C_etc..29_in_D

Don't worry about what's posted at GSoC ideas page.  Understand why converting these software building blocks to D would be advantageous and make the case for it in your proposal, and why you should be the one to do it.

If it's difficult to see why converting them to D would be advantageous, well, that's probably my fault.  I don't know how else to say it.  As I mentioned previously, once the runtime hooks are converted to templates, it should become much more apparent to all without me having to advocate for it.

Mike

On Sunday, 7 April 2019 at 11:10:02 UTC, Stefanos Baziotis wrote:
> On Sunday, 7 April 2019 at 07:27:01 UTC, 9il wrote:
>
>> Very interesting, it would be nice to have SIMD convolution algorithms in D.
> I'm not an expert in image convolution, but any proposals
> are welcome.
>
>> We need a few D language extensions:
>>  1. multiple auto ref values
>>  2. more clever alias syntax for templates (it is required for data science libraries in D)
>>  3. opApply and foreach API design rework (requires 1.)
>
> Can you please explain more 1. and 2. ?
>
>>
>> As a small bonus, I will pay an additional $400 for each D language extension merged to DMD.
>
> That's kind :). For me, certainly not needed though.
>
>>
>> Let me know if you would like to work during GSoC on one of the related directions and especially DIPs.
>
> Yes.
>
>> ilyayaroshenko at google mail
>
> I will reach you by mail.

Copy-pasted the answer here, maybe someone has good ideas for syntax.

1. Multiple auto ref values.

D allows returning a single value from a function. To return multiple values few workarounds can be used:

a. Add additional arguments as ref/out parameters.
b. Use std.type.Tuple

Both of them don't allow to return multiple values by reference: for example, references on array/container elements.

To return multiple values by reference we can use

c. Add additional arguments as ref/out pointer parameters
d. Use mir.functional.RefTuple

Both of them don't well fit to modern D, Mir, and future generic containers. For example, Mir's Series consist as pair of two arrays (keys and values), keys are sorted. It would be awesome to use D Range syntax (front, popFront, empty) to iterate Series by reference. Also, it would be very good for performance, for D GC-free reference-counted libraries.

The possible (but may not be the best) syntax is:

auto ref fun(int[] a)
{
    return (a[0], a[1] * 3.4); // returns ref int and double
}

void handle(ref int a, double b)
{
    a = cast(int) (b * b);
}

int twice(int a) { a * 2; }

void main()
{
    int[] ar = [1, 2];
    handle(fun(ar));
    auto (i : &$0, f, e : $0 + $1, f : $1.twice) = fun(ar);

    // i is a pointer to ar[0], type of int*
    // d stores a values of a[1] * 3.4, type of double
    // e stores value ar[0] + a[1] * 3.4, type of double
    // f stores value ar[0] * 2, type of int
}

2. This DIP for clever alias syntax is the fix for the following issues:

  https://issues.dlang.org/show_bug.cgi?id=16486
  https://issues.dlang.org/show_bug.cgi?id=16465
The brief DIP idea is that the code like below should work:

alias PackedUpperTriangularMatrix(T) = Slice!(StairsIterator!(T*, "-"));

// fails, issue 16486
auto foo(T)(PackedUpperTriangularMatrix!T m)
{
}

// Current workaround: it is too crazy for users to
// know what is StairsIterator!(T*, "-")).
auto foo(T)(Slice!(StairsIterator!(T*, "-")) m)
{
}

Currently used Slice types in Lubeck / Production code

Slice!(double*) - D slice analog
Slice!(double*, 1, Universal) - BLAS vector, used in mir-lapack and mir-blas.
Slice!(double*, 2) - Contiguous matrix, that has an efficient loop for iteration over elements, see mir.algorithm.iteration sources.
Slice!(double*, 2, Canonical) - BLAS/LAPACK matrix representation, used in mir-lapack and mir.blas
Slice!(double*, N, Universal) - zero copy view to work with ndarray in numpy, see also low level API bindgins, and high level bindings
Slice!(StairsIterator!(double*, "+")) and ...
Slice!(StairsIterator!(double*, "-")) - packed storage for triangular matrixes, for BLAS/LAPACK
Slice!(ChopIterator!(size_t*, uint*)); - Memory efficient graph representation without labels.
Possible future Slice types (2019?):
Slice!(double*, 1, Contiguous, string*) - like Pandas Series
Slice!(double*, 2, Contiguous, LabelT1*, LabelT2*) - like Pandas DataFrame
Slice!(double*, 2, Contiguous, LabelT1*, LabelT2*, LabelT3*) - like Pandas Panel
Slice!(ChopIterator!(size_t*, uint*), 1, Contiguous, string*); - Memory efficient graph representation with labels.
Slice!(ChopIterator!(size_t*, Slice!(double*, 1, Contiguous, uint*))) - Sparse Matrix representation that can be used to interact with existing C/C++/Fortran libraries

On Monday, 8 April 2019 at 02:13:30 UTC, Mike Franklin wrote:

>
> If it's difficult to see why converting them to D would be advantageous, well, that's probably my fault.  I don't know how else to say it.  As I mentioned previously, once the runtime hooks are converted to templates, it should become much more apparent to all without me having to advocate for it.
>

Here's a little data I gathered very quickly to illustrate why there might be performance benefits.  Below is a table of all the calls to `memcpy` that DMD makes when compiling Phobos and druntime.

What's disappointing to me is the extremely larger number of calls to `memcpy` to copy 8-bytes of data on a 64-bit machine (not to mention the few to for 0 bytes!).

Now imagine if that were a template like this (https://github.com/JinShil/memcpyD/blob/56dfe65cbae4407fa584e53dac54c105d433e958/memcpyd.d#L17-L22) that could be inlined.  I think there would be some performance benefits.

It's actually quite alarming how many calls there are to memcpy for such small amounts of data.

Phobos:
  count fn     size_in_bytes
  48420 memcpy 8
  26298 memcpy 2
  26298 memcpy 1
   6484 memcpy 33
   4766 memcpy 136
   4250 memcpy 25
   2308 memcpy 32
   1994 memcpy 35
   1444 memcpy 31
   1272 memcpy 34
   1000 memcpy 37
    928 memcpy 28
    816 memcpy 41
    614 memcpy 42
    560 memcpy 44
    458 memcpy 43
    388 memcpy 40
    388 memcpy 30
    372 memcpy 36
    324 memcpy 39
    304 memcpy 45
    272 memcpy 29
    268 memcpy 38
    254 memcpy 20
    248 memcpy 46
    192 memcpy 49
    192 memcpy 47
    176 memcpy 48
    104 memcpy 27
     84 memcpy 51
     72 memcpy 50
     72 memcpy 26
     64 memcpy 63
     64 memcpy 52
     48 memcpy 0
     28 memcpy 53
     26 memcpy 16
     22 memcpy 24
     20 memcpy 58
     20 memcpy 57
     20 memcpy 54
     16 memcpy 73
     16 memcpy 55
     12 memcpy 82
     12 memcpy 70
     12 memcpy 56
     12 memcpy 22
      8 memcpy 92
      8 memcpy 83
      8 memcpy 81
      8 memcpy 76
      8 memcpy 72
      8 memcpy 68
      8 memcpy 65
      8 memcpy 64
      8 memcpy 61
      8 memcpy 60
      8 memcpy 19
      4 memcpy 98
      4 memcpy 90
      4 memcpy 89
      4 memcpy 85
      4 memcpy 79
      4 memcpy 78
      4 memcpy 74
      4 memcpy 71
      4 memcpy 69
      4 memcpy 67
      4 memcpy 66
      4 memcpy 168
      4 memcpy 104
      2 memcpy 8192
      2 memcpy 57344
      2 memcpy 109

druntime:
   3285 memcpy 8
    447 memcpy 136
    130 memcpy 16
     10 memcpy 8192
     10 memcpy 32
      2 memcpy 24
      2 memcpy 0

On Monday, 8 April 2019 at 02:13:30 UTC, Mike Franklin wrote:
> Don't worry about what's posted at GSoC ideas page.  Understand why converting these software building blocks to D would be advantageous and make the case for it in your proposal, and why you should be the one to do it.
>
> If it's difficult to see why converting them to D would be advantageous, well, that's probably my fault.  I don't know how else to say it.  As I mentioned previously, once the runtime hooks are converted to templates, it should become much more apparent to all without me having to advocate for it.
>
> Mike

Thanks for the stats Mike. Actually the points were pretty self-explanatory. I had a rough
idea about 2 and thanks for the rest. I was headed in the wrong direction because this project idea needs a proposal that answers "why" questions and not "how/when" questions. I think you know your insight was key in this discussion. It's an interesting topic and given that I will have time, I will tackle it GSoC or not so wait for further contact :). I think this thread is a good place to report progress.

On Sunday, 7 April 2019 at 07:27:01 UTC, 9il wrote:
> 
>
> We need a few D language extensions:
>  1. multiple auto ref values
>  2. more clever alias syntax for templates (it is required for data science libraries in D)
>  3. opApply and foreach API design rework (requires 1.)
>
> Each of them requires a DIP (D Improvement Proposal) and Pull Request to DMD with required changes.
>
> As a small bonus, I will pay an additional $400 for each D language extension merged to DMD.
>

I'm in for an extra $100 at least for #2, could be persuaded on the others.

On Saturday, 6 April 2019 at 18:56:42 UTC, Stefanos Baziotis wrote:
> In any case, this is the proposal: https://docs.google.com/document/d/1B3YbjY70iVgGT7hq_16tBriLuCAUtjFR0gf0xkxmQ5c/edit?usp=sharing
>
> Any comments are welcome!

Here's a new proposal I made. I think this clarifies what the project will try to solve better.

https://docs.google.com/document/d/1kLV9PA1QN3KTqGQDrEvnSELbp2eHtL7GImCQS9rqNLU/edit?usp=sharing

And of course, any comments are welcome!