Hi Ali (and anyone interested).

While implementing support for function attributes and parameter storage classes in 'openmethods', then working on my new 'ducktyping' library (golang-style interfaces), I came up with a module that helps with creating function mixins derived from an existing function. Since it is used in two distinct projects, I want to put it in a library of its own.

It seems to me that this is the sort of things that bolts is all about. What do you think of taking it in?

You can see some code here:
https://github.com/jll63/openmethods.d/blob/using-function-editor/source/openmethods_functioneditor.d
I suggest you take a look at the unit test at the bottom. Real usage here: https://github.com/jll63/openmethods.d/blob/using-function-editor/source/openmethods.d, search for occurrences of Editor.

It's still a work in progress but I am opening the discussion now because I will need permission from my employer. Which I will certainly get (they are very open-source friendly), but I will have to go through a bit of red tape and I will need to specify, in the ticket, whether I will be contributing to an existing open-source project (in case the module becomes part of bolts), or I will be leading a new project.

On Monday, 6 April 2020 at 20:52:56 UTC, Jean-Louis Leroy wrote:
> Hi Ali (and anyone interested).
>
> While implementing support for function attributes and parameter storage classes in 'openmethods', then working on my new 'ducktyping' library (golang-style interfaces), I came up with a module that helps with creating function mixins derived from an existing function. Since it is used in two distinct projects, I want to put it in a library of its own.
>
> It seems to me that this is the sort of things that bolts is all about. What do you think of taking it in?
>
> You can see some code here:
> https://github.com/jll63/openmethods.d/blob/using-function-editor/source/openmethods_functioneditor.d
> I suggest you take a look at the unit test at the bottom. Real usage here: https://github.com/jll63/openmethods.d/blob/using-function-editor/source/openmethods.d, search for occurrences of Editor.

Hey! At first look it does look like something bolts-y so I'm very open to discussing the usecases/problems to figure it out. And it initially looks like very cool stuff!

Am I correct in assuming that FunctionEditor is a way to "stringify" various attributes of a function, in order to create string versions of new functions, in order to mixin new functions?

Where being able to mock interfaces is one usecase?

What other usecases would there be?

I looked at the real usage you have, but I don't think I understood the dispatcher and discriminator

And, one last question:

If string interpolation was around would the interface you've designed be the same?

I opened an issue in github as well: https://github.com/aliak00/bolts/issues/9 just incase it makes more sense to discuss it there.

Cheers!

On Tuesday, 7 April 2020 at 11:50:10 UTC, aliak wrote:
> Am I correct in assuming that FunctionEditor is a way to "stringify" various attributes of a function, in order to create string versions of new functions, in order to mixin new functions?

Yes. In a first step the function is "meta-objectified", capturing all the function attributes (well not UDAs yet), parameter storage classes, and whether it is a member or a static function, in a template instantiation which has:
- accessors for querying all the attributes from a centralized place
- "mutators" that create new function meta-objects with altered attributes
- accessors that return strings suitable to be mixed in to produce function declarations or definitions (and I am going to add more for making a function pointer, a delegate, and maybe for adding template parameters)

> Where being able to mock interfaces is one usecase?

Yes.

> What other usecases would there be?

Any situation that involves wrapping a function from a template, across module boundaries - keeping in mind what Adam Ruppe explains here: https://stackoverflow.com/questions/32615733/struct-composition-with-mixin-and-templates/32621854#32621854 (and that I painfully re-discovered by myself).

For example, my new ducktyping library works with two templates: InterfaceVariable(Interface), a fat pointer that contains a void* to an object and a Vtbl* to a table of forwarder functions; and as(T, IV), that returns an InterfaceVariable and builds the forwarders and Vtbl for type T.

Thus:

  interface Geometry
  {
    @nogc nothrow void move(real dx, real dy);
  }

  alias IGeometry = InterfaceValue!Geometry;

  class Rectangle
  {
    @nogc nothrow void move(real dx, real dy) { ... }
  }

  IGeometry geo = new Rectangle(...).as!IGeometry;

  geo.move(dx, dy);

...translates (more or less) to:

  struct InterfaceVariable!Geometry
  {
    struct Vtbl
    {
      @nogc nothrow void function(void* obj, real a0, real a1) move0;
    }
    void* obj;
    Vtbl* vtbl;
    @nogc nothrow void move(real dx, real dy) { vtbl.move0(obj, a0, a1); }
  }

  InterfaceVariable!Geometry as!(Rectangle, InterfaceVariable!Geometry)(Rectangle r)
  {
    static @nogc nothrow void move(void* obj, real a0, real a1)
    {
      (cast(Rectangle) obj).move(a0, a1);
    }
    static InterfaceVariable(Geometry).Vtbl vtbl = { &move0 };
    return InterfaceVariable(Geometry)(cast(void*) r, &vtbl);
  }

Note that it is essential to preserve `@nogc nothrow` (and any parameter storage classes even if the example doesn't show this).

> I looked at the real usage you have, but I don't think I understood the dispatcher and discriminator

Given e.g.:

  module matrix;
  Matrix times(double a, virtual!Matrix b);
  @method DiagonalMatrix _times(double a, DiagonalMatrix b) { ... }

The library generates the following code:

  // in module matrix
  alias times = Method!(matrix, "times", 0).dispatcher;
  alias times = Method!(matrix, "times", 0).discriminator;

  // in Method!(matrix, "times", 0):
  double dispatcher(double a0, Matrix a1) { return resolve(a1)(a0, a1); }
  Method!(matrix, "times", 0) discriminator(MethodTag, double a0, Matrix a1);

  // also a pointer type:
  double function(double, Matrix) Spec;

  // and a code string to build a wrapper for a method specialization
  // it will be mixed in a context where 'Spec' is an alias to a specialization
  enum wrapper(Spec) = q{
    Matrix wrapper(double a0, Matrix a1) { return Spec(a0, cast(DiagonalMatrix) a1); }

Of course template Method is not allowed to use names like Matrix or DiagonalMatrix. Instead everything is channeled via what I call the "anchor", in this case essentially `__traits(getOverloads, Module, Name)[Index]`.

> If string interpolation was around would the interface you've designed be the same?

Yes. The whole point is to avoid manipulating string until the very end - when specifying the body. And indeed, at that point, interpolation will help the *client* code a lot.

> I opened an issue in github as well: https://github.com/aliak00/bolts/issues/9 just incase it makes more sense to discuss it there.

Let's discuss here for the time being, as others may jump in with useful comments and suggestions.

From the GH "issue":

> For structs and classes I may have had a prototype that went something like:

I haven't given a lot of thought to using this approach beyond functions. Probably manipulating enums and classes are less of a pain (no parameter storage classes), and may need much fewer string mixins.

However, I do like the idea of composing e.g. enums using a meta-objects, like:

  mixin(EnumEditor!"Fruit".addEnumerator!"Grapefruit".etc.etc.asString);

On a side note, I would like to have a very simple construct that turns an identifier into a string (see https://forum.dlang.org/post/ayzrsajzcelavxdbwbji@forum.dlang.org).

Finally I am not set on the template and module names. `FunctionEditor` is a bit errr flat. Transmogrify sounds much better ;-)

On Tuesday, 7 April 2020 at 16:58:12 UTC, Jean-Louis Leroy wrote:
[...]
>
> I haven't given a lot of thought to using this approach beyond functions. Probably manipulating enums and classes are less of a pain (no parameter storage classes), and may need much fewer string mixins.

Yeah, I actually forgot the use case I was trying to make it for. So I agree at this time.

>
> However, I do like the idea of composing e.g. enums using a meta-objects, like:
>
>   mixin(EnumEditor!"Fruit".addEnumerator!"Grapefruit".etc.etc.asString);
>
> On a side note, I would like to have a very simple construct that turns an identifier into a string (see https://forum.dlang.org/post/ayzrsajzcelavxdbwbji@forum.dlang.org).

Turn an identifier into a string?

int myint;
assert(__traits(identifier, myint) == "myint");

Like that?

The forum post made me think of something more like:

assert(isThisAValidIdentifier!("some string"))

?

>
> Finally I am not set on the template and module names. `FunctionEditor` is a bit errr flat. Transmogrify sounds much better ;-)

Hehe yeah, the name came from the Calvin and Hobbes comic books (https://calvinandhobbes.fandom.com/wiki/Transmogrifier)

But aaaaanyway, ok so thanks for the explanations :)

This is very cool stuff. I'm not exactly sure how the entire system can be seen yet - lots of questions , but if could build a static reflection system that is easy to use and allows for the mutation of compile time entities in order for derived entities to be mixed in, that "sounds" very cool.

I'm thinking use-cases can slowly be built up one by one, but the initial use-case you have is a great starting point. So, yeah, I'm all for it.

I opened a branch: https://github.com/aliak00/bolts/tree/reflection

And I put up an initial commit for variable reflection: https://github.com/aliak00/bolts/commit/5da534466285ae7357b56ccf9d60470a63d2ac61

At this time I have no idea how well that will work in the general sense but time will tell. It would be great if you could push small pieces of the function editor in to there (make a new file under the reflection module and just start with that?).

The entity traits should be integrated with the existing traits as well so that the source of truth remains the same - I find that's the hardest part about this library - there're so many ways to do things and it's already hard to keep track of which method is used where (even with a library as small as it is right now)

On Wednesday, 8 April 2020 at 12:03:23 UTC, aliak wrote:

> Turn an identifier into a string?
>
> int myint;
> assert(__traits(identifier, myint) == "myint");
>
> Like that?

No, the other way around. I don't like to use the string literal syntax for template arguments when those strings are meant to be identifiers. I.e. instead of:

  mixin makeFunction!("foo", ...);
  mixin makeFunction!("0ops", ...); // weird error deep down

I'd rather write:

  mixin makeFunction!(q.foo, ...);
  mixin makeFunction!(q.0ops, ...); // error reported here

Sorta Lisp-ish. It's such a tiny thing that it doesn't deserve its own dub package, except on Fool's Day maybe. But maybe in a `bolts.quote`?

> I opened a branch: https://github.com/aliak00/bolts/tree/reflection
> [...]
> At this time I have no idea how well that will work in the general sense but time will tell. It would be great if you could push small pieces of the function editor in to there (make a new file under the reflection module and just start with that?).

So right now I cannot send PRs - not until I get permission form my employer. I created a ticket for that but it will take some time.

But as part of `openmethods` it's OK, so what about I create a `bolts/reflection` subdir there and put stuff in it. You can watch it, we can discuss it, and you can send PRs against openmethods. It looks like dub doesn't object to having multiple `bolts` directories, from different packages. As soon as I get the green light, I will PR the entire `bolts/reflection` tree. In the meantime, someone who wants to play with this work-in-progress can dub depend on openmethods.

Since yesterday, I thought a lot about this. It looks like there is a path to making meta-programming a *lot* easier. I now think that replacing the __traits and std.traits caboodle with meta-objects is more important than creating string mixins, although of course we badly want to hide them as much as possible. So I am not going to focus only one code generation.

In last night's experiments I pushed the cumbersome nested `static foreach` loops on members and overloads to a mixin template in a new Aggregate meta-object. And it works beautifully.Here is the Mock(Interface) example again (from the unit test of bolts.reflection.metaaggregate):

  interface TheFullMonty
  {
    pure int foo() immutable;
    @nogc @trusted nothrow ref int foo(out real, return ref int, lazy int) const;
    @safe shared scope void foo(scope Object);
  }

  template Mocker(alias Function, int Index)
  {
    static if (is(Function.returnType.type == void)) {
      enum Mocker = Function.withBody!"";
    } else static if (Function.isRef) {
      enum Mocker = Function.withBody!q{
        static %s rv;
        return rv;
      }.format(Function.returnType.asString);
    } else {
      enum Mocker = Function.withBody!q{
        return %s.init;
      }.format(Function.returnType.asString);
    }
  }

  class Mock(Interface) : Interface
  {
    mixin Aggregate!Interface.forEachFunctionMixin!(Mocker);
  }

[Later] I renamed the branch to bolts-reflection, here: https://github.com/jll63/openmethods.d/tree/bolts-reflection/source/bolts/reflection