Good Evening,

As you know from my previous typefunction project I am trying to make CTFE more usable for proper metaprogramming (as opposed to just string manipulation and string mixins).
Since I do believe that working with structure and rich data-types is more useful than working with only strings essentially.

The goal is to be able to write regular procedural code, to introspect existing or generate new code.

This I want to achieve with core.reflect and core.codegen, which will be part of druntime and give access to compiler/CTFE integrated reflection capabilities.

in core.reflect.decl, for example you would see:

import core.reflect.node; // Empty base class is defined in there
enum DeclarationKind
{
    FunctionDeclatation,
    /+
    StructDeclaration,
    UnionDeclarartion,
    ClassDeclarartion,
    VariableDeclaration,
    AliasDeclaration,
    TemplateDeclaration,
    +/
}
class Declaration : Node
{
    string name;
    DeclarationKind kind;
    Attribute[] attributes;
    Linkage linkage;
    string comment;
}

Declaration[] DeclarationsFromString(string s);

Which can then be used like so:

pragma(msg, DeclarationsFromString(
 q{
  int var = 3;
  void* myFunc (void*) {};
 }
)[1].name);

and it would output the name if the second Declarartion which is:
"myFunc"

this functionality does already work as of a few hours ago.
However it runs into forward reference issues even more than type functions did.
And I assume I'll first have to make advances on dealing with complicated forward reference graphs before I can show anything neat.

On Sunday, 20 June 2021 at 22:51:06 UTC, Stefan Koch wrote:

+1

But this sounds much like macros, which I prefer much.

However, they are unlikely.

Stefan, could you first make a dip for any of these, or would you present one grand dip for all of these things?

Just one point, it would be nice to retrieve also the context of the whole project as AST enacting macros as true language plugins.

On Sunday, 20 June 2021 at 22:51:06 UTC, Stefan Koch wrote:

import core.reflect.node; // Empty base class is defined in there
enum DeclarationKind
{
    FunctionDeclatation,
    /+
    StructDeclaration,
    UnionDeclarartion,
    ClassDeclarartion,
    VariableDeclaration,
    AliasDeclaration,
    TemplateDeclaration,
    +/
}
class Declaration : Node
{
    string name;
    DeclarationKind kind;
    Attribute[] attributes;
    Linkage linkage;
    string comment;
}

Declaration[] DeclarationsFromString(string s);

pragma(msg, DeclarationsFromString(
 q{
  int var = 3;
  void* myFunc (void*) {};
 }
)[1].name);

I'm looking forward to this but I thing the same technics are always involved.
So whavetever is the magic trick used, there's actually no magic. Either 1 or 2.

1. you need to parse new code to produce an ast. In D this is currently like mixin which goes by the lexical stage or more simply the .d you pass to the compiler.

2. you need to clone an ast to run semantic on it with a specific scope. In D this is currently like template and that requires to clone an ast so that the semantic will decorate the ast in the right context scope (e.g expression.type).

so eventually this would be something like something new that allows stage 2 without stage 1 ? AST macros ?

On Monday, 21 June 2021 at 10:36:56 UTC, Basile B. wrote:

Ideally you would not need stage 2 either, at least not always.
If all you are doing is using core.reflect (for example to get debug info).
Then no modifications of the AST is required.

What I am doing is essentially runtime reflection (like java and go).
However since the divide between runtime and compile-time is broken via CTFE,
We can use this information to splice code into the existing AST.
(which is what mixins do)
Whether I have to copy the AST or can solve for a resolution schedule that doesn't conflict is currently to be explored.
It's not easy.
It is however a great way to debug inconsistencies in DMDs AST.

On Monday, 21 June 2021 at 11:00:42 UTC, Stefan Koch wrote:

ok I did not get that. Sorry for the off topic ;)

On Monday, 21 June 2021 at 11:37:31 UTC, Basile B. wrote:

I'm not sure how off topic it is...I had the same question. I didn't really understand what was going on...still not sure I do...

The main question I would have for Stefan would be how exactly this relates to type functions.

On Monday, 21 June 2021 at 11:42:47 UTC, jmh530 wrote:

For me it's a different take on type functions.
Type functions rely on a lot of un-inspectable compiler magic.
And while I think that they are intuitive you cannot easily see which introspection information you have access to.
Type functions are blending the syntax of types as types with types as values.
Creating hard to solve issues.
for example

type T = int;
pragma(msg, T.mangleof); // should this be the mangle of the Variable T ? or should it be the mangle of int?

core.reflect on the other hand declares what exactly you get back, what fields it has and which functions can be called on it.

On Sunday, 20 June 2021 at 22:51:06 UTC, Stefan Koch wrote:

If something like this gets through, would this allow things like converting range queries into SQL, for example, like how Entity Framework allows?

e.g. You can convert db.people.filter!(p => p.isMale) directly into a prepared statement similar to SELECT * FROM people WHERE isMale = TRUE?

On Monday, 21 June 2021 at 12:39:25 UTC, SealabJaster wrote:

Hmm yes I think so.
But the user-code for this would look quite horrid since you have to match a particular templates and so forth.
Also you cannot change the meaning of code that comes in.
So in that sense they're not actual macros.
You can however generate a new function/module which is exactly like the one you reflected on except that you do a SQL query rather than using the template.

On Monday, 21 June 2021 at 13:18:21 UTC, Stefan Koch wrote:

Exciting, even if the end code looks like a complete hack!

Another thing: Entity Framework, when you give it something like:

int age = 20; // Any external variable
MyDbContext.People.Where(person => person.Age >= age);

It's able to determine that age is an external variable, so essentially a parameter, and generates a prepared statement like:

SELECT * FROM people WHERE age >= $1;

And it will then automatically forward whatever's in age into parameter $1.

This is of course a very niche use-case, but it's one of the things that makes Entity Framework so fluent to use (i.e. your queries are written in normal C#, but are executed on the database instead of in the application's memory). Of course it falls back to client-side evaluation if it can't translate things.

This gets more complex the more conditions you tack on, as well as things like .Include.

This doesn't necessarily need the ability to rewrite the AST, but as you said, simply generate based off of the reflection of the AST.

And there's likely a lot more use cases outside of this example. But I'm now wondering what other things might be possible @_@