April 08, 2020
On Wednesday, 8 April 2020 at 09:48:25 UTC, Stefan Koch wrote:
> The only reason why I used assert (__ctfe) not static assert (__ctfe) is because that's impossible to express.
> since __ctfe is a magic runtime variable that's hardwired to be zero in the non-ctfe case.

Indeed, and that should make you think how you can properly express that instead of adding special cases to established language concepts making them mean an entirely different thing.

Imagine if D could not express quick sort, so programmers just use bubble sort instead out of necessity. Then to fix that, you let the compiler detect a common bubble-sort implementation, and replace it with a quick-sort implementation.

It is backwards-compatible and automatically works with existing code. What a fix!

Except even the slightest deviation of that pattern will break the optimization.
Also the quick-sort implementation is not stable unlike bubble sort, so the behavior is actually different making this an illegal optimization. What a hack!

Problems like this should be attacked at the root. This assert(__ctfe) hack may fix one thing in the short-term, but you gain a lot of technical debt. Next thing you'll stumble upon this:

```
import core.stdc.stdlib;
int* newIntPtr() @nogc {
    if (__ctfe) {
        return new int(); // Error: cannot use new in @nogc function
    } else {
        return cast(int*) malloc(4);
    }
}
```

What now, another hack? Or should we finally think of something to make __ctfe work with `static if` / `static assert` proper?
April 08, 2020
Am Wed, 08 Apr 2020 09:53:24 +0000 schrieb Atila Neves:


>>
>> Isn't there an issue for better C as well? If a function can't be compiled in betterC but is valid for CTFE, then this could be a potential solution.
>>
>> I don't know how this works today.
>>
>> -Steve
> 
> Good question, me neither.

The answer is simple. It simply does not work today ;-)

-- 
Johannes
April 08, 2020
Am Wed, 08 Apr 2020 08:42:52 +0000 schrieb Stefan Koch:


>>
>> My primary fear here is that the current implementation checks the function too late in semantic and therefore such checks (or the "ensure no non-ctfe function calls ctfe-only functions" check are much more difficult to implement than they should be. Using a pragma or attribute would easily solve this.
>>
>> Maybe you'd have to introduce something analog to "safetyInProgress" which is "ctfeOnlyInProgress" to make this assert(__ctfe) detection fully usable? But I think this adds way to much complexity. Even right now, the code of the PR could probably be reduced by 1/2 when just using a pragma.
> 
> The detection of assert __ctfe in the function body _can_ be done very
> early in the process I do not want to introduce a pragma or magic
> annotatation.
> When existing the language is perfectly able to express (only runs at
> ctfe).
> 
> Also the PR I have open is _only_ for that.
> I or someone else, might write code which does the detection earlier and
> then disables betterC checks, but this is only about turning off
> codegen.

The important point is that we do not bake in semantics now which might turn out to cause problems later. If we can do the assert detection before all betterC checking, nogc checking, and related checks, this is fine. But it should not be an afterthought, we should really be pretty sure that this will not cause issues later on and that this integrates well with all other language features. Especially betterC, nogc (if you have a @nogc: at module top, do you want to still enforce GC checks in CTFE only code, ...)

-- 
Johannes
April 08, 2020
On Wednesday, 8 April 2020 at 10:27:02 UTC, Dennis wrote:
>
> What now, another hack? Or should we finally think of something to make __ctfe work with `static if` / `static assert` proper?

I have tried doing that for a long time and gave up.
the problem is that static if and static assert get done before ctfe.
and therefore would always see the __ctfe being false.

A language which only allows you to express bubble-sort may have a good reason for that limitation.
Though I suspect it would come down to religious believes in mathematical concepts.

April 08, 2020
On 4/8/20 3:09 AM, Johannes Pfau wrote:
> Am Tue, 07 Apr 2020 17:13:54 -0400 schrieb Steven Schveighoffer:
> 
>> On 4/7/20 12:53 PM, Johannes Pfau wrote:
>>> The example above actually already compiles with -betterC, as the check
>>> is done in codegen phase! But some betterC checks seem to be done in
>>> semantic. This doesn't work yet:
>>>
>>> ------------------------
>>> string generateMixin(T)(string b)
>>> {
>>>       assert(__ctfe);
>>>       return typeid(T).stringof ~ " " ~ b ~ ";";
>>> }
>>>
>>> void main() @nogc {
>>>       mixin(generateMixin!string("b"));
>>> }
>>> ------------------------
>>> dmd -betterC test.d -c test.d(4): Error: TypeInfo cannot be used with
>>> -betterC
>>
>> Ironically, ctfe can't deal with TypeInfo today.
>>
>> In the past, the issue was for functions that use TypeInfo is that the
>> non-CTFE branch uses TypeInfo, and that can't be used in betterC, but
>> the CTFE branch doesn't use TypeInfo, but the compiler wants to build
>> the whole function.
>>
>> So while this is still a problem, I don't think this fix will change
>> that.
>>
> 
> I'm not sure what exactly you're referring to, but this code compiles
> perfactly fine:
> --------------------------
>   string generateMixin(T)(string b)
>   {
>        assert(__ctfe);
>        return "auto a = " ~typeid(T).stringof ~ ";";
>   }
>     void main() @nogc
>   {
>        mixin(generateMixin!string("b"));
>        pragma(msg, generateMixin!string("b"));
>   }
> --------------------------
> 

I misread your example! I thought you were doing typeid(T).toString. I had to fix an issue very recently that was similar to this.

TypeInfo does not work inside ctfe, but here you aren't actually using the TypeInfo object. If you switch to typeid(T).toString, even without -betterC you get:

Error: static variable typeid(string) cannot be read at compile time

But your example is indeed a bug with betterC. But even if that was fixed, the code would still not work. It seems more like a diagnostic error:

return "auto a = typeid(" ~ T.stringof ~ ");";

This is equivalent to your code and will work in betterC, but the mixin of it won't.

> But it does not copile with -betterC. The point here is that the "Error:
> TypeInfo cannot be used with -betterC" message does not apply to CTFE-
> only functions, so this PR should be able to disable the check.

In fact, there is a further error that TypeInfo isn't being used AT ALL, and it's still complaining.

> 
> My primary fear here is that the current implementation checks the
> function too late in semantic and therefore such checks (or the "ensure
> no non-ctfe function calls ctfe-only functions" check are much more
> difficult to implement than they should be. Using a pragma or attribute
> would easily solve this.

This is a legitimate concern. Indeed, it might not fix this issue just to not emit the code to the binary.

-Steve
April 08, 2020
On Tuesday, 7 April 2020 at 17:33:03 UTC, Johannes Pfau wrote:
>>> On Sunday, 5 April 2020 at 18:25:10 UTC, Petar Kirov [ZombineDev] wrote:
>>>>
>>>> Another, more radical and interesting option is to not perform codegen unless it's actually necessary. Instead of blacklisting `@ctfe` or `assert(__ctfe)` functions, have a whitelist of what functions are allowed to codegen-ed.
>>>>
>>>> [...]
>
> How would such a lazy compilation model work without explicit 'export' animations?

In my post [1] I said that the `export` storage class is the key to making this work. The way I envision this is similar to a copying garbage collector:
1. [MARK] Starting from the set of ~GC~ CG roots [2] the compiler traverses the static call graph [3] and marks any callees as `export` (i.e. the `export` inference step in my proposal). All functions determined to have the `export` attribute are added to the code generation queue
2. [COPY] Do code generation by poping function from the queue until it becomes empty.
3. (Optional) produce a report of all functions that were not added to the queue.

[1]: https://forum.dlang.org/post/xbllqrpvflazfpowizwj@forum.dlang.org
[2]: CG (call graph) roots: all functions marked explicitly as `export`, or implicitly under rules B) - F) in my post [1] (e.g. `main()`, `unittest`s, virtual functions, etc.)
[3]: After statements inserted from `static`, `version` and `mixin` declarations are resolved, the set of all functions that may be called from a given function. Even if a function is called conditionally at runtime, it's still part of the static call graph.

> If I wanted to build a proprietary library libfoo.so and provide only .di files with stripped function bodies, all functions have to be compiled into the .so file. But how does the compiler know the result is going to be a shared library in a separate compilation model, where the compiler produces only an object file?

To produce the libfoo.{so,dll,dylib}, I assume that you have a whole package of D modules that you want to compile.

---

(The case where libfoo is produced from a single D module is the trivial base case: only functions determined to be `export` will be part of the object file, from which libfoo will be produced.)

Also, as in C++ (AFAIU), templates can't be part of your dynamic library ABI, only specific template instances can be. If type templates participate in the function signature (the parameter types, or return type) of an `export`-ed function, then those type templates are inferred to be `export` as well (and all of their members). I suspect we can do better here, but it's easier to start pessimistically to relax the rules later when we have a better formal model an empirical experience.

---

So, finally, how would we produce a dynamic library?

1. In our implementation files (*.d) we would mark as `export` all public functions that we want to be part of our dynamic library API/ABI.

2. When the compiler produces *.di header files it would include the declarations of only public, protected or package symbols that are determined to be export.

3. When the object files are produced all `export` functions will be compiled (including private ones, i.e. private symbols, called from public exported ones).

4. The rest of the symbols that are not determined to be `export` are discarded.

---

All in all, what we're left with is:
1. A strict compilation mode in which only symbols determined to be `export` are included in libraries. This way we have a much more precise definition of ABI.
2. A lazy compilation mode, based on (dmd -i) in which the compiler automatically imports and compiles functions as needed, starting from `main()`. In a package-wise compilation (not separate compilation), regular free functions are treated like template functions - they're not compiled unless used.

Actually 1. and 2. are exactly the same compilation strategy, just very different use cases ;)

> I guess such a lazy compilation model might be quite interesting, but as it is a quite radical idea, it needs lots of thinking, proper specification and testing first. (Testing as you'll probably have to emit everything as weak symbols then, and you probably want to test how linkers deal with that if you make excessive use of it)

I don't think we should rely that much on linkers for object file-level GC. The compiler shouldn't produce functions that are not determined to be `export` in the first place.


Some examples:

--------------------
module app;

string generateMixin(T)(string b)
{
    assert(__ctfe);
    return T.stringof ~ " " ~ b ~ ";";
}

void main() @nogc
{
    mixin(generateMixin!string("b"));
}
--------------------

`dmd -c app.d` should produce an object file with a single (empty) function - `main()`;

---------------------------
module foo;

private void privateFunc() {}

public void publicTemplate(T)()
{
    privateFunc();
}
---------------------------

Q: How would this work?
A: Uner my proposal, the compiler should produce an **empty** object file.

What could work is the following:

---------------------------
module foo;

private void privateFunc() {}

public void publicTemplate(T)()
{
    privateFunc();
}

// the obvious way:
export void publicFunc() { publicTemplate!int(); }

// the syntax sugar way 1a:
export alias publicFunc = publicTemplate!int;

// the syntax sugar way 1b:
export
{
    alias publicFunc = publicTemplate!int;
    alias publicFunc = publicTemplate!string;
}

// the syntax sugar way 2:
export
{
    publicTemplate!int;
    publicTemplate!string;
}

// Note: the syntax sugar is not part of my proposal,
// as it's not important for now.
---------------------------

Or:

---------------------------
module foo;

private void privateFunc() {}

public void publicTemplate(T)()
{
    privateFunc();
}

----
module bar;

import foo;

export void actuallyPublicInterfaceFunction()
{
    publicTemplate!int();
}

---

dmd -i bar.d

---------------------------

In this way we tell the compiler to compile the exported function in `bar` as well as any other functions that they may call.




April 08, 2020
On 2020-04-07 21:33, tsbockman wrote:

> Catching an AssertError thrown by an assert statement is officially undefined behavior:
> https://dlang.org/spec/expression.html#assert_expressions

Ok, let me tweak the example a bit:

void foo()
{
    assert(__ctfe);
}

unittest @system
{
    try
        foo();
    catch (AssertError)
    {}
}

Then if you read the third item in the list:

"AssertExpression has different semantics if it is in a unittest or in contract."

Then you read the unittest spec [1], the third item, the second sentence:

"Unlike AssertExpressions used elsewhere, the assert is not assumed to hold, and upon assert failure the program is still in a defined state."

Now it's not undefined anymore.

[1] https://dlang.org/spec/unittest.html

-- 
/Jacob Carlborg
April 08, 2020
On 2020-04-07 22:43, Stefan Koch wrote:
> On Tuesday, 7 April 2020 at 17:52:44 UTC, Jacob Carlborg wrote:
>>
>> void foo()
>> {
>>     assert(__ctfe);
>> }
>>
>> void main() @system
>> {
>>     try
>>         foo();
>>     catch (AssertError)
>>     {}
>> }
> 
> firstly Does anybody actually do this?

Ok, let me tweak the example a bit:

void foo()
{
    assert(__ctfe);
}

unittest
{
    foo();
}

druntime catches _all_ exceptions:

https://github.com/dlang/druntime/blob/46867186035fdf5ec2596efbb770627518ff0919/src/core/runtime.d#L617

Regardless if anyone does it or not, it's still a breaking change. Then we can of course argue the impact of the breaking change, how much code will actually break.

-- 
/Jacob Carlborg
April 08, 2020
On 4/8/20 2:53 PM, Jacob Carlborg wrote:
> On 2020-04-07 22:43, Stefan Koch wrote:
>> On Tuesday, 7 April 2020 at 17:52:44 UTC, Jacob Carlborg wrote:
>>>
>>> void foo()
>>> {
>>>     assert(__ctfe);
>>> }
>>>
>>> void main() @system
>>> {
>>>     try
>>>         foo();
>>>     catch (AssertError)
>>>     {}
>>> }
>>
>> firstly Does anybody actually do this?
> 
> Ok, let me tweak the example a bit:
> 
> void foo()
> {
>      assert(__ctfe);
> }
> 
> unittest
> {
>      foo();
> }
> 
> druntime catches _all_ exceptions:
> 
> https://github.com/dlang/druntime/blob/46867186035fdf5ec2596efbb770627518ff0919/src/core/runtime.d#L617 
> 
> 
> Regardless if anyone does it or not, it's still a breaking change. Then we can of course argue the impact of the breaking change, how much code will actually break.
> 

I think it's a breaking change, just like flagging any obvious error case as a non-compilable error is a breaking change for code that wasn't aware of the problem.

I'm sure there was lots of code that did something equivalent to this before it was disallowed:

ref int foo()
{
  int x;
  return x;
}

That doesn't mean we shouldn't do it.

-Steve
April 08, 2020
On 4/8/20 2:53 PM, Jacob Carlborg wrote:
> Regardless if anyone does it or not, it's still a breaking change. Then we can of course argue the impact of the breaking change, how much code will actually break.

another interesting case:

void foo()
{
   assert(__ctfe);
}

void main(string[] args)
{
   if(args.length == 0)
       foo(0);
}

Arguably, the call will never happen. But it won't compile now.

As I said a minute ago, this is a breaking change, and it's one we should make.

-Steve