On Monday, 15 July 2019 at 12:57:46 UTC, Mike Franklin wrote:
> On Monday, 15 July 2019 at 12:27:22 UTC, Vladimir Panteleev wrote:
>
>> This isn't exactly true. The restriction is that core should not *import* rt. Have a look at all the extern(C) definitions in Druntime - using extern(C) functions to communicate between the compiler and rt, as well as core and rt, is not a "hack", but an established mechanism to invoke the low-level implementations in Druntime.
>
> We are trying to implement many of those `extern(C)` runtime hooks as templates.  Those templates need to be implicitly imported through object.d.  That means code that was in `rt` is converted to a template, and then moved to object.d.  However, as we do more and more of them object.d becomes unwieldy.
>
> I took the initiative to prevent object.d from turning into a more of a monstrosity that it already is, and moved those runtime templates (which used to reside in `rt`) back into `rt`.

This is not a problem, and not at all related to the issue we're discussing. The problem is that you chose to move them into `rt` instead of somewhere under `core`, which would respect existing conventions and avoid breakages like the ones we've seen reported in this thread.

On Monday, 15 July 2019 at 12:42:57 UTC, Mike Franklin wrote:
> ... and are the exception, not the rule.  I believe they should be moved to `rt`.

BTW, from this discussion it seems to me that you did not have a good overview of the situation and made a bad decision based on that. No problem there, however, throughout this discussion, you've made a number of claims that have been refuted, and instead of re-evaluating the situation, you've stuck to your position while presenting continuously more radical counter-arguments. At this point, I apologize and would like to ask you to carefully consider whether you are trying to neutrally argue for the best outcome to make D better, or merely to continue supporting your position.

On Monday, 15 July 2019 at 13:06:58 UTC, Vladimir Panteleev wrote:
> On Monday, 15 July 2019 at 12:42:57 UTC, Mike Franklin wrote:
>> ... and are the exception, not the rule.  I believe they should be moved to `rt`.
>
> BTW, from this discussion it seems to me that you did not have a good overview of the situation and made a bad decision based on that. No problem there, however, throughout this discussion, you've made a number of claims that have been refuted, and instead of re-evaluating the situation, you've stuck to your position while presenting continuously more radical counter-arguments. At this point, I apologize and would like to ask you to carefully consider whether you are trying to neutrally argue for the best outcome to make D better, or merely to continue supporting your position.

I'm not thoughtless and clueless, and I've acted in the best interest of D.

I asked for input from other developers before moving forward.  They helped me understand that `rt` is where the core language features are implemented.  The code that I moved into `rt/array` initially came from `rt`, depends on implementations in `rt` and is the core implementation of D's built-in arrays, so it is perfectly logical for it to be in `rt`.  I submitted PRs that were scrutinized by the reviewers and merged.  Even after considering your arguments, I still believe I made the right decision.  I'm sorry you disagree and don't find my explanations convincing.

Mike

On Monday, 15 July 2019 at 13:27:39 UTC, Mike Franklin wrote:
> I asked for input from other developers before moving forward.  They helped me understand that `rt` is where the core language features are implemented.

Assuming it was the discussion linked in this thread, it did not seem like the finer details which we covered today have been considered back then.

There are also two developers in this thread which are arguing the contrary, and it would be unfair to count only those participating in the prior discussion if you would make this a matter of numbers.

> I submitted PRs that were scrutinized by the reviewers and merged.

Bad PRs get merged all the time. It happens. We need to recognize mistakes and do our best to address them.

> The code that I moved into `rt/array` initially came from `rt`, depends on implementations in `rt` and is the core implementation of D's built-in arrays, so it is perfectly logical for it to be in `rt`.

That's a subjective assessment, and it would be perfectly fine to rely on subjective opinions to make decisions which would otherwise have no consequence, but it is not the case here. The decision was an avoidable breaking change.

> I'm sorry you disagree and don't find my explanations convincing.

There are a few arguments remaining in this thread to which you have not responded.

On Monday, 15 July 2019 at 13:00:08 UTC, Vladimir Panteleev wrote:

>> We are trying to implement many of those `extern(C)` runtime hooks as templates.  Those templates need to be implicitly imported through object.d.  That means code that was in `rt` is converted to a template, and then moved to object.d.  However, as we do more and more of them object.d becomes unwieldy.
>>
>> I took the initiative to prevent object.d from turning into a more of a monstrosity that it already is, and moved those runtime templates (which used to reside in `rt`) back into `rt`.
>
> This is not a problem, and not at all related to the issue we're discussing. The problem is that you chose to move them into `rt` instead of somewhere under `core`, which would respect existing conventions and avoid breakages like the ones we've seen reported in this thread.

It is related.  If I follow your suggestion to move these implementations to `core.internal` and continue with the objective of converting all runtime hooks to templates, the vast majority of `rt` will end up being moved to `core.internal`.  Is that what you're suggesting?

`rt` is the language implementation.  `core.internal` contains the utilities used internally by druntime and "privately" imported by Phobos.  Following that established convention, I made the right decision.

I'm sorry it broke digger, but digger is not how we typically build DMD, druntime, and Phobos.

Mike

On Monday, 15 July 2019 at 14:00:23 UTC, Mike Franklin wrote:
> On Monday, 15 July 2019 at 13:00:08 UTC, Vladimir Panteleev wrote:
>
>>> We are trying to implement many of those `extern(C)` runtime hooks as templates.  Those templates need to be implicitly imported through object.d.  That means code that was in `rt` is converted to a template, and then moved to object.d.  However, as we do more and more of them object.d becomes unwieldy.
>>>
>>> I took the initiative to prevent object.d from turning into a more of a monstrosity that it already is, and moved those runtime templates (which used to reside in `rt`) back into `rt`.
>>
>> This is not a problem, and not at all related to the issue we're discussing. The problem is that you chose to move them into `rt` instead of somewhere under `core`, which would respect existing conventions and avoid breakages like the ones we've seen reported in this thread.
>
> It is related.  If I follow your suggestion to move these implementations to `core.internal` and continue with the objective of converting all runtime hooks to templates, the vast majority of `rt` will end up being moved to `core.internal`.  Is that what you're suggesting?
>
> `rt` is the language implementation.  `core.internal` contains the utilities used internally by druntime and "privately" imported by Phobos.  Following that established convention, I made the right decision.
>
> I'm sorry it broke digger, but digger is not how we typically build DMD, druntime, and Phobos.
>
> Mike

The point is that we don't ship the sources of rt to the user. That's the separation. With templates sources must be made available to the user, s.t. the compiler can instantiate them. However, rt doesn't get shipped to the user as it is compiled only.

As probably more tools and distributions are affected I suggest we move all source code that needs to be instantiated by the user to core.internal.

Am Mon, 15 Jul 2019 12:40:50 +0000 schrieb Vladimir Panteleev:

> On Monday, 15 July 2019 at 12:36:14 UTC, Mike Franklin wrote:
>> Many of the implementations in `rt/array` are templates, so the entire implementation should be available through object.d, not just declarations.
> 
> The amount of templated code is still finite, otherwise you would have needed to include all of rt into the set of files to be made importable.

Which is due to the limited time available in GSoC. Right now, the implementations simply call typeid and forward to the TypeInfo based implementations. But all of these implementations are ultimately meant to be fully templated, and then you'll need access to lots of rt functions.

So why should we we now move the functions to core, so that we can later move them back to rt?

> 
>> In `core.internal`, I see utilities that appear to be intended for use
>> only within runtime, or "privately" imported by Phobos.
>>  I do not see implementations for fundamental language features
>> as can be found in `rt`.  The code in `rt/array` implementations for
>> fundamental language features, not utilities to be used privately with
>> druntime.
> 
> Please have a closer look:
> 
> - core.internal.hash contains the implementation of hashing routines
> used for associative arrays.
> - core.internal.arrayop contains the implementation of array vector
> operations. This one doesn't seem to be too far from your work in
> question.
> 

Both don't have any real dependencies. Maybe all the array implementation stuff could be moved to core, but until that happens moving only parts of the implementation seems to be a bad idea. Especially considering this:

>> "rt can import core, but core can't import rt." makes it pretty clear to me.


It's probably a mistake that we even have both rt and core.internal. core.internal seems to be a much more recent addition (2013), probably inspired by std.internal. It's no wonder there's duplication and no clearly defined scope for the packages.

And why would anyone think it's a good idea not to install the rt headers? What do you gain from this, except from a few KB saved disk space?

-- 
Johannes

Am Mon, 15 Jul 2019 12:27:22 +0000 schrieb Vladimir Panteleev:

> On Monday, 15 July 2019 at 12:14:16 UTC, Mike Franklin wrote:
>> Many of the implementations in `rt/array` require importing or
>> referencing other implementations in `rt` (e.g. `rt.lifetime`).
>>  If they were moved to `core.internal` they would require
>> importing `rt` or peeking into `rt` with various hacks, which exactly
>> what you've said should not be done.
> 
> This isn't exactly true. The restriction is that core should not *import* rt. Have a look at all the extern(C) definitions in Druntime - using extern(C) functions to communicate between the compiler and rt, as well as core and rt, is not a "hack", but an established mechanism to invoke the low-level implementations in Druntime.


Grepping for extern in core.internal yields one result outside of pareoptions.d. If you count parseoptions.d, 6 results.

I wonder how you can advertise this as a good idea: You have to manually keep declarations in sync, you have to be very careful to get the attributes right, module constructor evaluation order guarantees don't hold, no mangling (no type safety), you pollute the C namespace, no inlining, no templates.

This is an established workaround at best, in no way it's a good solution.

-- 
Johannes

Am Mon, 15 Jul 2019 14:00:23 +0000 schrieb Mike Franklin:

> On Monday, 15 July 2019 at 13:00:08 UTC, Vladimir Panteleev wrote:
> 
>>> We are trying to implement many of those `extern(C)` runtime hooks as templates.  Those templates need to be implicitly imported through object.d.  That means code that was in `rt` is converted to a template, and then moved to object.d. However, as we do more and more of them object.d becomes unwieldy.
>>>
>>> I took the initiative to prevent object.d from turning into a more of a monstrosity that it already is, and moved those runtime templates (which used to reside in `rt`) back into `rt`.
>>
>> This is not a problem, and not at all related to the issue we're discussing. The problem is that you chose to move them into `rt` instead of somewhere under `core`, which would respect existing conventions and avoid breakages like the ones we've seen reported in this thread.
> 
> It is related.  If I follow your suggestion to move these implementations to `core.internal` and continue with the objective of converting all runtime hooks to templates, the vast majority of `rt` will end up being moved to `core.internal`.  Is that what you're suggesting?
> 
> `rt` is the language implementation.  `core.internal` contains the utilities used internally by druntime and "privately" imported by Phobos.  Following that established convention, I made the right decision.
> 
> I'm sorry it broke digger, but digger is not how we typically build DMD, druntime, and Phobos.
> 
> Mike

I agree here: rt is code deeply tied to the language / compiler. core.internal is largely code which is useful as standalone module (abort, convert, parts of dassert, lifetime, traits, string).

However, the structure is not really clear: rt.util (older than core.internal) should probably rather be part of core.internal and some code in core.internal (arrayop, assert) should be in rt.

Either way, dictating a code structure on druntime only because of build system aspects (these files are installed, these are not) seems to be a very bad idea. The code should be structured in a way to minimize cross- module dependencies and to seperate compiler specific from generic code.

In addition, the build system shipped as part of druntime is the authoritative way to build the project. Even though digger is an important tool, we can't really compromise on code quality in druntime only to stay compatible to build systems using undocumented internals of the runtime build process.

-- 
Johannes

Am Mon, 15 Jul 2019 17:59:25 +0000 schrieb Seb:

> On Monday, 15 July 2019 at 14:00:23 UTC, Mike Franklin wrote:
>> On Monday, 15 July 2019 at 13:00:08 UTC, Vladimir Panteleev wrote:
>>
>>>> We are trying to implement many of those `extern(C)` runtime hooks as templates.  Those templates need to be implicitly imported through object.d.  That means code that was in `rt` is converted to a template, and then moved to object.d. However, as we do more and more of them object.d becomes unwieldy.
>>>>
>>>> I took the initiative to prevent object.d from turning into a more of a monstrosity that it already is, and moved those runtime templates (which used to reside in `rt`) back into `rt`.
>>>
>>> This is not a problem, and not at all related to the issue we're discussing. The problem is that you chose to move them into `rt` instead of somewhere under `core`, which would respect existing conventions and avoid breakages like the ones we've seen reported in this thread.
>>
>> It is related.  If I follow your suggestion to move these implementations to `core.internal` and continue with the objective of converting all runtime hooks to templates, the vast majority of `rt` will end up being moved to `core.internal`.  Is that what you're suggesting?
>>
>> `rt` is the language implementation.  `core.internal` contains the utilities used internally by druntime and "privately" imported by Phobos.  Following that established convention, I made the right decision.
>>
>> I'm sorry it broke digger, but digger is not how we typically build DMD, druntime, and Phobos.
>>
>> Mike
> 
> The point is that we don't ship the sources of rt to the user. That's the separation. With templates sources must be made available to the user, s.t. the compiler can instantiate them. However, rt doesn't get shipped to the user as it is compiled only.
> 

But why is that? What's the benefit here? And do we skip to this
convention forever, only for legacy reasons?
We always shipped rt in gdc btw. Nobody ever complained.


If we decide to move that code to core.internal, I'm with Mike that we should simply move all array code out of rt. In the long term, we may even end up moving anything out of rt: Modern D code is template heavy, template code needs the sources available. Inlining needs the sources as well. The more we get rid of TypeInfo and the more we modernize the compiler/runtime interface, this weill become an issue.

And duplicating extern(C) declarations, syncing them manually, ... is a safety liability and maintainance nightmare (see my other post). So in no way should we start to add more such functions interfacing rt to core.internal.


-- 
Johannes