July 01

On Monday, 1 July 2024 at 13:00:55 UTC, ryuukk_ wrote:

>

please stick to what i wrote, i don't want string concatenation, i provide a reduced example from my project, everything should be a single template block, no extra functions other than the append() one

Mixin templates are a declaration scope, not a function scope, so they aren't necessarily analyzed top to bottom. This is why allowing partial string mixins would be complicated, consider this example:

mixin template implement()
{
    mixin("struct " ~ str);

    mixin("{");

    immutable string str = "T";

    mixin("}");
}

It's not obvious how this should be compiled, and this is before throwing static if and static foreach in the mix! If you want to procedurally build a type in sequential steps, you'll have to do that in a function scope.

If your concern is that such a function would add needless code generation, you can use an immediately invoked anonymous function like so:

mixin template implement(string typeName, string[] members)
{
    mixin(() {
        string result = "struct " ~ typeName ~ " {";
        foreach (name; members)
        {
            result ~= "int " ~ name ~ ";";
        }
        result ~= "}";
        return result;
    } ());
}

mixin implement!("S", ["x", "y", "z"]);

immutable s = S(x: 3, y: 5, z: 7);

You can use your fixed size array append function to try and improve CTFE performance, but I'd start with straightforward concatenation, and see if it's actually too slow. In that case, maybe see if you can reduce it to a self-contained example and post it on bugzilla as a performance bug.

July 02

On Monday, 1 July 2024 at 21:43:02 UTC, Dennis wrote:

>

On Monday, 1 July 2024 at 13:00:55 UTC, ryuukk_ wrote:

>

please stick to what i wrote, i don't want string concatenation, i provide a reduced example from my project, everything should be a single template block, no extra functions other than the append() one

Mixin templates are a declaration scope, not a function scope, so they aren't necessarily analyzed top to bottom. This is why allowing partial string mixins would be complicated, consider this example:

mixin template implement()
{
    mixin("struct " ~ str);

    mixin("{");

    immutable string str = "T";

    mixin("}");
}

It's not obvious how this should be compiled, and this is before throwing static if and static foreach in the mix! If you want to procedurally build a type in sequential steps, you'll have to do that in a function scope.

If your concern is that such a function would add needless code generation, you can use an immediately invoked anonymous function like so:

mixin template implement(string typeName, string[] members)
{
    mixin(() {
        string result = "struct " ~ typeName ~ " {";
        foreach (name; members)
        {
            result ~= "int " ~ name ~ ";";
        }
        result ~= "}";
        return result;
    } ());
}

mixin implement!("S", ["x", "y", "z"]);

immutable s = S(x: 3, y: 5, z: 7);

You can use your fixed size array append function to try and improve CTFE performance, but I'd start with straightforward concatenation, and see if it's actually too slow. In that case, maybe see if you can reduce it to a self-contained example and post it on bugzilla as a performance bug.

I said it 2 times already, i don't want string concatenation, i'll benchmark later, but not right now, right now i'm looking for a functioning code without string concatenation

July 02
On Tuesday, July 2, 2024 1:23:42 AM MDT ryuukk_ via Digitalmars-d-learn wrote:
> I said it 2 times already, i don't want string concatenation, i'll benchmark later, but not right now, right now i'm looking for a functioning code without string concatenation

D has two options for mixins, and both of them require complete statements.

Template mixins mix in the contents of a template rather than strings, but they only allow you to compile in declarations, because that's all that templates can contain:

https://dlang.org/spec/template-mixin.html

You can control what is compiled into a template mixin via conditional compilation based on the template arguments (e.g. with static if or static foreach), but the declarations do need to be complete just like they would for any template, so it will really only work for you if you just need a set of declarations and not if you're looking to build a list of statements to run.

For anything more complex, you'll need string mixins:

https://dlang.org/spec/statement.html#mixin-statement

A string mixin takes any string that's known at compile time and compiles it as code, but the string must contain a complete statement (or a sequence of statements). You can provide the mixin a string literal, an enum, a list of strings which are concatenated together, a function call, or any other expression so long as it evaluates to a string at compile time.

You can of course choose to not use string concatenation to create such a string, but unless you can provide a string literal with the exact piece of code that you need, you're going to need to actually build a string from pieces one way or another, and that normally means either appending to a string and/or concatenating strings together.

The only way that I can think of at the moment which would involve building a string at compile time but which would not involve explicit allocations would be to use a static array and set the unused characters to whitespace, since the compiler will let you mix in a static array of characters as a string, e.g.

mixin(()
      {
          char[1024] buffer = ' ';
          ... build the string by setting the characters in buffer ...
          return buffer;
      }());

However, given that this is CTFE we're talking about here, I'm pretty sure that this particular solution not only allocates, but it potentially allocates more than simply appending to a string would. IIRC, CTFE's current implementation doesn't actually use mutatation. Rather, it creates a new value every time that you mutate a variable. And if that is indeed the case, then each time you set a character in the static array, you'd basically be duping the array. CTFE does not work in a manner that pretty much anyone who hasn't actively worked on it is going to expect, and the generated code isn't necessarily anything like what would be generated for runtime code. So, not many D programmers are going to be very good at guessing what's going to be most efficient with CTFE.

Regardless, because a string mixin must be given an expression that evaluates to a string, your options are basically

1. Use string concatenation in an expression that you pass to mixin.

2. Call a function that returns a string (with the internals of that function doing whatever is required to generate that string and return it) and pass the return value to mixin.

3. Create an enum using either #1 or #2 and pass that to mixin.

So, if you want to build a statement from pieces instead of providing a known statement or known list of statements, and you don't want to use string concatenation, you basically have to write a function which builds and returns a string in whatever the most efficient fashion is that you can come up with.

And if you want to measure the performance of a particular function that builds a string during CTFE, you _will_ need to measure build times rather than testing it at runtime because of how differently CTFE behaves from running code at runtime. And honestly, I expect that you would have to be generating a lot of string mixins to be able to get a consistent, measurable difference in build times between different implementations, but if you care that much about build times, you'll need to experiment to see what the most efficient way to build a string is for your code. It is certainly true though that CTFE tends to be pretty inefficient, unfortunately. It really is a shame that "newCTFE" was never completed.

In general, best practice would be to just build the string in whatever the simplest, most maintainable fashion would be, particularly since it will have no impact on the runtime performance of the program, and you could easily spend more time trying to optimize your build times than you would ever save by trying generate the string more efficiently, but it's your code.

- Jonathan M Davis



July 03

On Tuesday, 2 July 2024 at 07:23:42 UTC, ryuukk_ wrote:

>

I said it 2 times already, i don't want string concatenation, i'll benchmark later, but not right now, right now i'm looking for a functioning code without string concatenation

Your buffer solution works, but you need to put it inside a function, not at declaration scope. What you wrote declares runtime variables, which wouldn't be usable at compile time (if you got it past the parser, which is where it was failing).

So for instance:

mixin template implement()
{
    mixin(() {
        // ctfe new array is basically the same as static array
        char[] buffer = new char[4096];
        int pos = 0;

        void append(string str)
        {
            buffer[pos .. pos + str.length] = str[];
            pos += str.length;
        }

        append("void* ctx;");
        return buffer[0 .. pos];
    }());
}

And yes, it is faster to do this than appending. But you have to do a lot of it to make a huge difference.

-Steve

July 03

On Wednesday, 3 July 2024 at 03:52:41 UTC, Steven Schveighoffer wrote:

>
mixin template implement()
{
    mixin(() {
        // ctfe new array is basically the same as static array
        char[] buffer = new char[4096];
        int pos = 0;

        void append(string str)
        {
            buffer[pos .. pos + str.length] = str[];
            pos += str.length;
        }

        append("void* ctx;");
        return buffer[0 .. pos];
    }());
}

And yes, it is faster to do this than appending. But you have to do a lot of it to make a huge difference.

What about creating an array of strings and then using join? One obvious problem with this solution is the requirement to specify the size of buffer.

July 03

On Wednesday, 3 July 2024 at 11:57:58 UTC, Lance Bachmeier wrote:

>

On Wednesday, 3 July 2024 at 03:52:41 UTC, Steven Schveighoffer wrote:

> >

And yes, it is faster to do this than appending. But you have to do a lot of it to make a huge difference.

What about creating an array of strings and then using join? One obvious problem with this solution is the requirement to specify the size of buffer.

It's important to remember that what happens at CTFE is not the same as what happens at runtime. CTFE is a tree-walking interpreter, and does weird things like allocate storage when you change a value. And there is no optimization.

So the best thing to do is try it and see what happens. You can't use your knowledge of how code compiles to judge CTFE performance.

My general feel is that the more code you give to the CTFE interpreter, the worse it gets. Like, try doing format in CTFE vs. simple appending it will be vastly different.

-Steve

July 06

On Wednesday, 3 July 2024 at 03:52:41 UTC, Steven Schveighoffer wrote:

>

On Tuesday, 2 July 2024 at 07:23:42 UTC, ryuukk_ wrote:

>

I said it 2 times already, i don't want string concatenation, i'll benchmark later, but not right now, right now i'm looking for a functioning code without string concatenation

Your buffer solution works, but you need to put it inside a function, not at declaration scope. What you wrote declares runtime variables, which wouldn't be usable at compile time (if you got it past the parser, which is where it was failing).

So for instance:

mixin template implement()
{
    mixin(() {
        // ctfe new array is basically the same as static array
        char[] buffer = new char[4096];
        int pos = 0;

        void append(string str)
        {
            buffer[pos .. pos + str.length] = str[];
            pos += str.length;
        }

        append("void* ctx;");
        return buffer[0 .. pos];
    }());
}

And yes, it is faster to do this than appending. But you have to do a lot of it to make a huge difference.

-Steve

That's annoying that it couldn't have been more straight forward.. but that'll do it for now, it works, thanks

August 12

On Tuesday, 2 July 2024 at 16:46:11 UTC, Jonathan M Davis wrote:

>

On Tuesday, July 2, 2024 1:23:42 AM MDT ryuukk_ via Digitalmars-d-learn wrote:

>

I said it 2 times already, i don't want string concatenation, i'll benchmark later, but not right now, right now i'm looking for a functioning code without string concatenation

D has two options for mixins, and both of them require complete statements.

Not a huge deal, but you forgot about the third option—mixin types—which does not require a complete statement: https://dlang.org/spec/type.html#mixin_types

August 12

On Tuesday, 2 July 2024 at 16:46:11 UTC, Jonathan M Davis wrote:

>

It really is a shame that "newCTFE" was never completed.

I didn’t know about this. I thought a new CTFE engine was actually implemented a few years ago. Is ‘newCTFE’ too old to be resurrected? Would it be feasible to make something similar from the ground up?

August 12

On Monday, 12 August 2024 at 11:26:50 UTC, IchorDev wrote:

>

On Tuesday, 2 July 2024 at 16:46:11 UTC, Jonathan M Davis wrote:

>

It really is a shame that "newCTFE" was never completed.

I didn’t know about this. I thought a new CTFE engine was actually implemented a few years ago. Is ‘newCTFE’ too old to be resurrected? Would it be feasible to make something similar from the ground up?

It's about 12.000 lines of code.
I am not sure how much has rotted away, I last touched it in 2021.

The relevant files would be:
  the bytecode interpreter:
    https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/bc.d
  the visitor that builds the bytodecode:
    https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/ctfe_bc.d

as you can see by the branch name it was rebased on dmd 2.093.
so that's a few versions old now.
As far as I know Max Haughton has tried to rebase it on master and improve it but I have not heard from him about that.