Hello all,

I will greatly appreciate if you could help me with my first step in the D land.

*YOU CAN SKIP DIRECTLY TO THE QUESTION BELLOW*:

Please allow me to introduce myself and give you my feelings about the D language then I'll ask my question. I'm a mathematician that works mainly on computer vision and machine learning for a research institute. All my CV code is written in C++ (runs on android and iOs) and no need to say that I'm desperately looking to replace it by a more modern language. My main complains about C++ are safety, implicit conversion between numeric types, tooling (no unified package manager, benchmarking or testing framework), verbose generics, poor standard library (although boost is great, but then compile time goes crazy), not so great at writing "high level" code when needed ... it just old.

I looked into some contenders: D, ocaml, Go, rust, even Haskell! GC languages are definitely not an option for computer vision and low level machine learning (writing ML libs), so GO, ocaml and haskell are out. (Although I use Go for production when I have to deal with concurrency and networks (getting energy data from sensors or web crawlers). The language is meh but the finishing (tooling, stability, documentation and libraries) is great).

So the only left possible alternatives are D and rust. I looked into D few years later but didn't look far as it was advertised as a GC language and the community (and support) seemed too small, and the tooling/libs not so great. I spent allot of time following rust and learning it, and put allot of hope in it. However, today rust seems to have 3 main deal breaking weaknesses for my particular use :

1-Bad C++ interop, and no plans to make it better. For me as for many developers in my field, I cannot work without my tools (like openCV, Eigen, Armadillo ...). No matter how good the language is, if it does not allow me to use well known and tested libraries that I need in my day job (or offers equivalent quality libs), I just cannot use it, its impossible.

2- Limited support for generics and little interest at enhancing them. Generic programming is essential to build highly efficient numeric libraries, like Eigen and Armadillo that relays heavily on expression templates to avoid temporaries. Rust still does not support integer template parameters and its trait based generics may be more secure but are very hard to use and impractical for non trivial generic code.

3- Memory management. The borrow checker is a real advantage when there is a need for highly efficient code. But for the less demanding applications in term of efficiency, it is a burden and I think a GC is more adapted for such applications.

Although rust is a wonderful language with a great community, The points 1 and 2 are deal breaking for me, and are what made me return to D and have a better look (expect allot of peoples like me I think when the hype start to diminish around rust and people that have tasted a modern programming language find them self unable to return to C++ but too limited by rust constraints).

By contrast, the 3 main weaknesses of rust (for me) are what attract me to the D language:

1- Good foundations for a full C++ interopt. Although we still cannot use C++ libs without any bindings like in objective-C. The work made so far on the D compiler + Calypso ( https://github.com/Syniurge/Calypso ) give me hope that one day, I'll just be able to put and "import opencv2.core;" into my code and use D in my work. By the way, make Calypso an official project to allow full interop with C++, stabilize one compiler on all major platforms and I'll throw C++ from the window and never look back.

2- Great at generics (hello Andrei !)

3- Optional Gc with I hope full support for non-GC memory management soon.

In brief, D is great, if you give it full c++ interop and stabilize the compiler(s) and the tooling, get backed by some big name in the industry, I think it is condemned to succeed.

****THE QUESTION****

Expression templates are heavily used in C++ to build highly efficient linear algebra libraries with lazy evaluation. I order to learn D, I'm trying to implement such system in D, so I can maybe interface it with ndslices and translate a linear algebra lib like Eigen in D.

For a first step, I tried to translate the example given here https://en.wikipedia.org/wiki/Expression_templates , and here is my attempt:

***************** start of code ******************
import std.stdio;
import std.traits;
import std.conv;

struct VecExpression(alias mixins) {
  mixin (mixins);
  VecSum!(typeof(this), VecExpression!(RHS)) opBinary(string op, alias RHS)(ref VecExpression!(RHS) rhs)
{
  static if (op == "+") return VecSum!(typeof(this), VecExpression!(RHS))(&this, &rhs);
}
}


mixin template Vec_impl(int n) {
  double[n] elems;

  @disable this();

  this(double[n] e){
    elems=e;
  }

  double opIndex(int i) {
    return elems[i];
  }
}

alias Vec(alias n) = VecExpression!("mixin Vec_impl!("~to!string(n)~");"); //Vec(int n) does not work

mixin template VecSum_impl(E1, E2) {
  E1 * e1;
  E2 * e2;

  @disable this();

  this(E1 * ee1, E2 * ee2){
    e1=ee1;
    e2=ee2;
  }

  double opIndex(int i) {
    return (*e1)[i]+(*e2)[i];
  }
}

alias VecSum(E1, E2) = VecExpression!("mixin VecSum_impl!("~fullyQualifiedName!E1~","~fullyQualifiedName!E2~");");

void main()
{
  Vec!(3) v1 = Vec!(3)([5., 2., 3.]), v2 = Vec!(3)([1., 4., 3.]), v3 = Vec!(3)([3., 2., 1.]);
  auto res = v1+v2;
  for(int i=0; i < 3; ++i){
    writefln("%f + %f = %f", v1[i], v2[i],  res[i]);
  }
  auto res2 = res+v3;
  for(int i=0; i < 3; ++i){
    writefln("%f + %f = %f", res[i], v3[i],  res2[i]);
  }
  writeln(res);
  writeln(res2);

  // VecExpression!(Vec_impl!(3)) ve; // Error: template instance Vec_impl!3 mixin templates are not regular templates
}

***************** end of code ******************

My questions:

1- Is there a cleaner way to do it ? I had to use struct because I want every thing to happen at compile time and on the stack (without gc). And I had to use string mixins because template mixin does not work the way I tried to use it ( see the error last line).

2- Is there a safer way to do it (without using pointers) ?

3- Do you think I'll hit a wall with this approach ?

4- Do you known any D libs that uses expression template for linear algebra ?

I thank you in advance for your help and wish you a nice weekend (and apologize for my bad english) :)

On 23/07/2016 11:05 PM, Etranger wrote:
[snip]

> ***************** start of code ******************
> import std.stdio;
> import std.traits;
> import std.conv;
>
> struct VecExpression(alias mixins) {
>   mixin (mixins);
>   VecSum!(typeof(this), VecExpression!(RHS)) opBinary(string op, alias
> RHS)(ref VecExpression!(RHS) rhs)
> {
>   static if (op == "+") return VecSum!(typeof(this),
> VecExpression!(RHS))(&this, &rhs);
> }
> }
>
>
> mixin template Vec_impl(int n) {
>   double[n] elems;
>
>   @disable this();
>
>   this(double[n] e){
>     elems=e;
>   }
>
>   double opIndex(int i) {
>     return elems[i];
>   }
> }
>
> alias Vec(alias n) = VecExpression!("mixin
> Vec_impl!("~to!string(n)~");"); //Vec(int n) does not work
>
> mixin template VecSum_impl(E1, E2) {
>   E1 * e1;
>   E2 * e2;
>
>   @disable this();
>
>   this(E1 * ee1, E2 * ee2){
>     e1=ee1;
>     e2=ee2;
>   }
>
>   double opIndex(int i) {
>     return (*e1)[i]+(*e2)[i];
>   }
> }
>
> alias VecSum(E1, E2) = VecExpression!("mixin
> VecSum_impl!("~fullyQualifiedName!E1~","~fullyQualifiedName!E2~");");
>
> void main()
> {
>   Vec!(3) v1 = Vec!(3)([5., 2., 3.]), v2 = Vec!(3)([1., 4., 3.]), v3 =
> Vec!(3)([3., 2., 1.]);
>   auto res = v1+v2;
>   for(int i=0; i < 3; ++i){
>     writefln("%f + %f = %f", v1[i], v2[i],  res[i]);
>   }
>   auto res2 = res+v3;
>   for(int i=0; i < 3; ++i){
>     writefln("%f + %f = %f", res[i], v3[i],  res2[i]);
>   }
>   writeln(res);
>   writeln(res2);
>
>   // VecExpression!(Vec_impl!(3)) ve; // Error: template instance
> Vec_impl!3 mixin templates are not regular templates
> }
>
> ***************** end of code ******************
>
> My questions:
>
> 1- Is there a cleaner way to do it ? I had to use struct because I want
> every thing to happen at compile time and on the stack (without gc). And
> I had to use string mixins because template mixin does not work the way
> I tried to use it ( see the error last line).
>
> 2- Is there a safer way to do it (without using pointers) ?
>
> 3- Do you think I'll hit a wall with this approach ?
>
> 4- Do you known any D libs that uses expression template for linear
> algebra ?
>
> I thank you in advance for your help and wish you a nice weekend (and
> apologize for my bad english) :)

My goodness that code is awful.

I have a fair idea what you are attempting to do here.
So I'm going to point you directly to gl3n. Its meant for game dev so won't provide you a 100% solution. But it should give you ideas of how to do it.

https://github.com/Dav1dde/gl3n/blob/master/gl3n/linalg.d#L49

If you have any questions and want somebody to talk with you instead of write, reply cycle. Please hop on to Freenode #d channel.

On Saturday, 23 July 2016 at 11:19:34 UTC, rikki cattermole wrote:
> On 23/07/2016 11:05 PM, Etranger wrote:
> [snip]
>
>> ***************** start of code ******************
>> import std.stdio;
>> import std.traits;
>> import std.conv;
>>
>> struct VecExpression(alias mixins) {
>>   mixin (mixins);
>>   VecSum!(typeof(this), VecExpression!(RHS)) opBinary(string op, alias
>> RHS)(ref VecExpression!(RHS) rhs)
>> {
>>   static if (op == "+") return VecSum!(typeof(this),
>> VecExpression!(RHS))(&this, &rhs);
>> }
>> }
>>
>>
>> mixin template Vec_impl(int n) {
>>   double[n] elems;
>>
>>   @disable this();
>>
>>   this(double[n] e){
>>     elems=e;
>>   }
>>
>>   double opIndex(int i) {
>>     return elems[i];
>>   }
>> }
>>
>> alias Vec(alias n) = VecExpression!("mixin
>> Vec_impl!("~to!string(n)~");"); //Vec(int n) does not work
>>
>> mixin template VecSum_impl(E1, E2) {
>>   E1 * e1;
>>   E2 * e2;
>>
>>   @disable this();
>>
>>   this(E1 * ee1, E2 * ee2){
>>     e1=ee1;
>>     e2=ee2;
>>   }
>>
>>   double opIndex(int i) {
>>     return (*e1)[i]+(*e2)[i];
>>   }
>> }
>>
>> alias VecSum(E1, E2) = VecExpression!("mixin
>> VecSum_impl!("~fullyQualifiedName!E1~","~fullyQualifiedName!E2~");");
>>
>> void main()
>> {
>>   Vec!(3) v1 = Vec!(3)([5., 2., 3.]), v2 = Vec!(3)([1., 4., 3.]), v3 =
>> Vec!(3)([3., 2., 1.]);
>>   auto res = v1+v2;
>>   for(int i=0; i < 3; ++i){
>>     writefln("%f + %f = %f", v1[i], v2[i],  res[i]);
>>   }
>>   auto res2 = res+v3;
>>   for(int i=0; i < 3; ++i){
>>     writefln("%f + %f = %f", res[i], v3[i],  res2[i]);
>>   }
>>   writeln(res);
>>   writeln(res2);
>>
>>   // VecExpression!(Vec_impl!(3)) ve; // Error: template instance
>> Vec_impl!3 mixin templates are not regular templates
>> }
>>
>> ***************** end of code ******************
>>
>> My questions:
>>
>> 1- Is there a cleaner way to do it ? I had to use struct because I want
>> every thing to happen at compile time and on the stack (without gc). And
>> I had to use string mixins because template mixin does not work the way
>> I tried to use it ( see the error last line).
>>
>> 2- Is there a safer way to do it (without using pointers) ?
>>
>> 3- Do you think I'll hit a wall with this approach ?
>>
>> 4- Do you known any D libs that uses expression template for linear
>> algebra ?
>>
>> I thank you in advance for your help and wish you a nice weekend (and
>> apologize for my bad english) :)
>
> My goodness that code is awful.
>
> I have a fair idea what you are attempting to do here.
> So I'm going to point you directly to gl3n. Its meant for game dev so won't provide you a 100% solution. But it should give you ideas of how to do it.
>
> https://github.com/Dav1dde/gl3n/blob/master/gl3n/linalg.d#L49
>
> If you have any questions and want somebody to talk with you instead of write, reply cycle. Please hop on to Freenode #d channel.

Hi and thanks for your quick replay.

A looked to the gl3n code, although the code is clear and clean, it is not really what I want to do in my example.

The gl3n is eager evaluation. If I write "v = v1+v2+v3;", then it will create a temporary variable "tmp1=v1+v2;" then a second temp "tmp2=tmp1+v3;" and finally "v=tmp2;".

what I'm trying to do to use lazy evaluation in order to create only one tmp that will that will directly hold the result of v1[i]+v2[i]+v3[i]. That concept is then generalized in order to perform more optimization based on entire expression trees, no just tow operands.

And thanks for the suggestion regarding the channel, I'll try to use it as soon as I have the time. I feel on advantage of the forum is that it stays for other people and not just for me.

best regards

On 24/07/2016 12:09 AM, Etranger wrote:
> On Saturday, 23 July 2016 at 11:19:34 UTC, rikki cattermole wrote:
>> On 23/07/2016 11:05 PM, Etranger wrote:
>> [snip]
>>
>>> ***************** start of code ******************
>>> import std.stdio;
>>> import std.traits;
>>> import std.conv;
>>>
>>> struct VecExpression(alias mixins) {
>>>   mixin (mixins);
>>>   VecSum!(typeof(this), VecExpression!(RHS)) opBinary(string op, alias
>>> RHS)(ref VecExpression!(RHS) rhs)
>>> {
>>>   static if (op == "+") return VecSum!(typeof(this),
>>> VecExpression!(RHS))(&this, &rhs);
>>> }
>>> }
>>>
>>>
>>> mixin template Vec_impl(int n) {
>>>   double[n] elems;
>>>
>>>   @disable this();
>>>
>>>   this(double[n] e){
>>>     elems=e;
>>>   }
>>>
>>>   double opIndex(int i) {
>>>     return elems[i];
>>>   }
>>> }
>>>
>>> alias Vec(alias n) = VecExpression!("mixin
>>> Vec_impl!("~to!string(n)~");"); //Vec(int n) does not work
>>>
>>> mixin template VecSum_impl(E1, E2) {
>>>   E1 * e1;
>>>   E2 * e2;
>>>
>>>   @disable this();
>>>
>>>   this(E1 * ee1, E2 * ee2){
>>>     e1=ee1;
>>>     e2=ee2;
>>>   }
>>>
>>>   double opIndex(int i) {
>>>     return (*e1)[i]+(*e2)[i];
>>>   }
>>> }
>>>
>>> alias VecSum(E1, E2) = VecExpression!("mixin
>>> VecSum_impl!("~fullyQualifiedName!E1~","~fullyQualifiedName!E2~");");
>>>
>>> void main()
>>> {
>>>   Vec!(3) v1 = Vec!(3)([5., 2., 3.]), v2 = Vec!(3)([1., 4., 3.]), v3 =
>>> Vec!(3)([3., 2., 1.]);
>>>   auto res = v1+v2;
>>>   for(int i=0; i < 3; ++i){
>>>     writefln("%f + %f = %f", v1[i], v2[i],  res[i]);
>>>   }
>>>   auto res2 = res+v3;
>>>   for(int i=0; i < 3; ++i){
>>>     writefln("%f + %f = %f", res[i], v3[i],  res2[i]);
>>>   }
>>>   writeln(res);
>>>   writeln(res2);
>>>
>>>   // VecExpression!(Vec_impl!(3)) ve; // Error: template instance
>>> Vec_impl!3 mixin templates are not regular templates
>>> }
>>>
>>> ***************** end of code ******************
>>>
>>> My questions:
>>>
>>> 1- Is there a cleaner way to do it ? I had to use struct because I want
>>> every thing to happen at compile time and on the stack (without gc). And
>>> I had to use string mixins because template mixin does not work the way
>>> I tried to use it ( see the error last line).
>>>
>>> 2- Is there a safer way to do it (without using pointers) ?
>>>
>>> 3- Do you think I'll hit a wall with this approach ?
>>>
>>> 4- Do you known any D libs that uses expression template for linear
>>> algebra ?
>>>
>>> I thank you in advance for your help and wish you a nice weekend (and
>>> apologize for my bad english) :)
>>
>> My goodness that code is awful.
>>
>> I have a fair idea what you are attempting to do here.
>> So I'm going to point you directly to gl3n. Its meant for game dev so
>> won't provide you a 100% solution. But it should give you ideas of how
>> to do it.
>>
>> https://github.com/Dav1dde/gl3n/blob/master/gl3n/linalg.d#L49
>>
>> If you have any questions and want somebody to talk with you instead
>> of write, reply cycle. Please hop on to Freenode #d channel.
>
> Hi and thanks for your quick replay.
>
> A looked to the gl3n code, although the code is clear and clean, it is
> not really what I want to do in my example.
>
> The gl3n is eager evaluation. If I write "v = v1+v2+v3;", then it will
> create a temporary variable "tmp1=v1+v2;" then a second temp
> "tmp2=tmp1+v3;" and finally "v=tmp2;".
>
> what I'm trying to do to use lazy evaluation in order to create only one
> tmp that will that will directly hold the result of v1[i]+v2[i]+v3[i].
> That concept is then generalized in order to perform more optimization
> based on entire expression trees, no just tow operands.
>
> And thanks for the suggestion regarding the channel, I'll try to use it
> as soon as I have the time. I feel on advantage of the forum is that it
> stays for other people and not just for me.
>
> best regards

If you evaluate it as v = a + b + c instead of v = a + (b + c) you will still have a temporary value. Remember structs are just the values they carry and are basically optimized out. Either way I recommend you not worry about it. Compilers can be smart and dmd is mostly good enough in this department.

If you use dynamic arrays and not fixed sized arrays, they will go into the heap, so be careful with what you do with them. Fixed sized arrays like gl3n does is fairly cheap since it goes on to the stack and in some cases only in registers.

Usually if we solve it on IRC we post the solution that we come up with on the post.

On Saturday, 23 July 2016 at 12:27:39 UTC, rikki cattermole wrote:
>
> If you evaluate it as v = a + b + c instead of v = a + (b + c) you will still have a temporary value. Remember structs are just the values they carry and are basically optimized out. Either way I recommend you not worry about it. Compilers can be smart and dmd is mostly good enough in this department.
>
> If you use dynamic arrays and not fixed sized arrays, they will go into the heap, so be careful with what you do with them. Fixed sized arrays like gl3n does is fairly cheap since it goes on to the stack and in some cases only in registers.
>
> Usually if we solve it on IRC we post the solution that we come up with on the post.

For small matrices of fixed size, lazy evaluation may not be worth it, but for large matrices and vectors, it is worth it. For example, for testing purpose, I wrote 2 versions of and MLP (multi-layered perceptron) lib, one that uses openBLAS and one with Eigen. The one with Eigen was significantly faster than the one with openBLAS tanks to lazy evaluation. No wonder all the most known c++ linear algebra lib use this method (Eigen, Armadillo, Blitz++, Boost uBlas, ...).

This link http://eigen.tuxfamily.org/dox/TopicLazyEvaluation.html explain better than me the advantages of this technique. Given the emphasize of D over generics, I think it is worth to explore this direction (I hope with better results than what I wrote !).

On Saturday, 23 July 2016 at 12:09:18 UTC, Etranger wrote:
> On Saturday, 23 July 2016 at 11:19:34 UTC, rikki cattermole wrote:
>> On 23/07/2016 11:05 PM, Etranger wrote:
>> [snip]
>>
>>> [...]
>>
>> My goodness that code is awful.
>>
>> I have a fair idea what you are attempting to do here.
>> So I'm going to point you directly to gl3n. Its meant for game dev so won't provide you a 100% solution. But it should give you ideas of how to do it.
>>
>> https://github.com/Dav1dde/gl3n/blob/master/gl3n/linalg.d#L49
>>
>> If you have any questions and want somebody to talk with you instead of write, reply cycle. Please hop on to Freenode #d channel.
>
> Hi and thanks for your quick replay.
>
> A looked to the gl3n code, although the code is clear and clean, it is not really what I want to do in my example.
>
> The gl3n is eager evaluation. If I write "v = v1+v2+v3;", then it will create a temporary variable "tmp1=v1+v2;" then a second temp "tmp2=tmp1+v3;" and finally "v=tmp2;".
>
> what I'm trying to do to use lazy evaluation in order to create only one tmp that will that will directly hold the result of v1[i]+v2[i]+v3[i]. That concept is then generalized in order to perform more optimization based on entire expression trees, no just tow operands.
>
> And thanks for the suggestion regarding the channel, I'll try to use it as soon as I have the time. I feel on advantage of the forum is that it stays for other people and not just for me.
>
> best regards

You should definitely have a look at mir - the upcoming library for numerical computing in D.
https://github.com/libmir/mir

There is a lot of working going on recently, for example Ilya is working on building the fattest BLAS library thanks to CTFE introspection etc.

You might know Ndslice (in Phobos), the first result of the mir project.

On Saturday, 23 July 2016 at 12:27:39 UTC, rikki cattermole wrote:
> Either way I recommend you not worry about it. Compilers can be smart and dmd is mostly good enough in this department.

he has something to worry about. remember, this is scientific department, where ours 4x4 matrices are child's play. one removed temporary can save hours of calculations there.

2OP: sorry, i can barely read that code. this has nothing to do with your skills, it is the topic -- i've never seen clean lazy evaluation code. after all, this is a hack.

still, i think that such a library worth at least some work.

as for "is my code/approach is good enough", i know only two answers.
1. yes, 'cause it works.
2. no, 'cause it doesn't work.
so if your code works, it is good. ;-)

i mean that such hacks will be somewhat ugly anyway, so just write a brief explanation in comments and go on.

sorry, it is probably not what you wanted to hear... ;-)

On Saturday, 23 July 2016 at 19:08:45 UTC, ketmar wrote:
> 2OP: sorry, i can barely read that code. this has nothing to do with your skills, it is the topic -- i've never seen clean lazy evaluation code. after all, this is a hack.
>
> still, i think that such a library worth at least some work.
>
> as for "is my code/approach is good enough", i know only two answers.
> 1. yes, 'cause it works.
> 2. no, 'cause it doesn't work.
> so if your code works, it is good. ;-)
>
> i mean that such hacks will be somewhat ugly anyway, so just write a brief explanation in comments and go on.
>
> sorry, it is probably not what you wanted to hear... ;-)

Oh no thanks ;-) I came here specially to see if there is some well known technique that I didn't hear about (I searched allot before posting here). If there is no such technique than it is just more interesting to explore !

Thanks all for your answers !

I thinks I'll continue exploring the lazy evaluation path and keep looking if there is a cleaner way to do it.
I'll specifically try to adapt lazy evaluation to ndsilces and see if I can have tangible results with benchmarks (by the way is there any good benchmarking lib like the #[bench] in rust that I can use ?) and study mir like it was suggested by Seb. I'll be very happy if I could contribute something useful for the D community :)

On 07/23/2016 01:05 PM, Etranger wrote:
> 1- Is there a cleaner way to do it ? I had to use struct because I want
> every thing to happen at compile time and on the stack (without gc). And
> I had to use string mixins because template mixin does not work the way
> I tried to use it ( see the error last line).

To avoid the string mixin, you can let VecExpression take an alias of the mixin template (Vec_impl/VecSum_impl) and the list of arguments:

----
struct VecExpression(alias mixinTemplate, mixinArgs ...)
{
  mixin mixinTemplate!mixinArgs;
  VecSum!(typeof(this), VecExpression!RHS) opBinary
    (string op, RHS ...)(ref VecExpression!RHS rhs)
  {
    /* ... */
  }
}
/* ... */
alias Vec(int n) = VecExpression!(Vec_impl, n);
/* ... */
alias VecSum(E1, E2) = VecExpression!(VecSum_impl, E1, E2);
----


Instead of mixing the fields into VecExpression, you could mix opBinary into distinct Vec/VecSum structs:

----
mixin template opBinaryMixin()
{
  VecSum!(typeof(this), RHS) opBinary(string op, RHS)(ref RHS rhs)
  {
    static if (op == "+") return typeof(return)(&this, &rhs);
  }
}

struct Vec(int n)
{
  double[n] elems;

  /* ... */

  mixin opBinaryMixin!();
}

struct VecSum(E1, E2)
{
  E1 * e1;
  E2 * e2;

  /* ... */

  mixin opBinaryMixin!();
}
----

A bit of type strictness is lost here, since opBinary accepts anything, not just Vec/VecSum. If that strictness is needed, you could check specifically for Vec/VecSum, or maybe mix in a little identifier along with opBinary and check for that.


You can also try std.typecons.scoped [1] to put classes on the stack. That would get around any kind of mixin.


[1] https://dlang.org/phobos/std_typecons.html#.scoped