Hey everyone!
I have been meaning to send these questions earlier, but then school happened.

Small introduction about me:
I am Dan Printzell and I am currently studying for a master degree in computer
science in Sweden. I'm one of the maintainers for the D package on Arch Linux[1]
and I am also writing a operating system from scratch in D called PowerNex[2].
Development of it has slowed down due to school and bugs but I'm trying to get
out a new release this year.

I am planning on applying to GSoC but I'm having a bit of a problem trying to
decide on what I should do. My goal is to try and do something that I will
personally use in PowerNex and it is something that will help the whole D
community, this is why I am looking at dmd and betterC topics.

For dmd I am primarily interested in implementing DIP1014[3], as this will help
me with the development of PowerNex. In the past I had to workaround that this
feature was missing. From the outside is seems like this would not take too long
to implement, at least not a whole GSoC. But as I'm not too familiar with dmd's
source code, I only did a few small patches to allow dmd to support PowerNex as
a target[4].
So my question for this topic would be: How long time do you guys think that
this will take to implement, test and get it merged? Is a month about correct?

For the betterC topic I would be interested in implementing a bare metal D
runtime[5]. This is something I already have some experience as I have written a
betterC runtime for PowerNex, and I also used to update/maintain Adam D. Ruppe's
minimal.zip[6] when I used that as the runtime for PowerNex.
My question would be: What features should this runtime contain? From the wiki I
see that basic libc functionality is requested, like the str* functions and
memory management. From what I can understand, the goals it to have a runtime
that can be used on multiple targeted environments, without any external
dependencies.

Other questions regarding this topic: how separate should it be from druntime?
Should this be a separate library, like a druntime-lite? or is the goal to allow
druntime to be used for a bare metal target? Also, should this only be betterC
or should it support regular D as well?


Depending on the implementation size of DIP1014 and the bare metal D runtime,
it seems like both could be implemented during a GSoC. But I am probably just
being time optimistic here.

Best Regards,
Dan Printzell

[1] https://www.archlinux.org/people/trusted-users/#wild
[2] https://github.com/PowerNex/PowerNex
[3] https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1014.md
[4] https://github.com/PowerNex/powernex-dmd I will probably submit some of
these upstream when PowerNex gets to a better, stable state
[5] https://wiki.dlang.org/GSOC_2019_Ideas#Baremetal_D_runtime
[6] http://arsdnet.net/dcode/minimal.zip

I'm sure you'd be of great help. :)

On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:
> For dmd I am primarily interested in implementing DIP1014[3], as this will help
> me with the development of PowerNex. In the past I had to workaround that this
> feature was missing. From the outside is seems like this would not take too long
> to implement, at least not a whole GSoC. But as I'm not too familiar with dmd's
> source code

Wrt. DIP1014, I tried to point out in the discussion that it probably needs a D ABI change (I know that it definitely will for LDC). We're not talking about a trivial minor adaptation, but a change affecting all non-POD arguments passed by value - don't copy in registers or on the stack anymore and let callEE destruct it, but have callER create a temporary, pass it by ref and then destruct it by callER (also when callEE throws), just like C++. As a nice side-effect, this will also fix C++ interop in this regard.

At least for LDC, there are some implicit moves emitted by the compiler in the glue-layer between front- and backend; I assume that's also the case for DMD.

I don't think a full 3 months of GSoC would be needed for a working implementation, but the effort is easily underestimated IMO. And sadly, it's not a pure front-end thing, so other compilers will have to reimplement some/most? parts.

> For the betterC topic I would be interested in implementing a bare metal D
> runtime[5]. This is something I already have some experience as I have written a
> betterC runtime for PowerNex, and I also used to update/maintain Adam D. Ruppe's
> minimal.zip[6] when I used that as the runtime for PowerNex.
> My question would be: What features should this runtime contain? From the wiki I
> see that basic libc functionality is requested, like the str* functions and
> memory management. From what I can understand, the goals it to have a runtime
> that can be used on multiple targeted environments, without any external
> dependencies.
>
> Other questions regarding this topic: how separate should it be from druntime?
> Should this be a separate library, like a druntime-lite? or is the goal to allow
> druntime to be used for a bare metal target? Also, should this only be betterC
> or should it support regular D as well?

IMO, it should indeed be something like a druntime-lite without any external dependencies, and druntime build on top of it. Containing the 'absolutely essential runtime bits for the core features' (to be defined ;) - e.g., (most?) runtime functions implicitly invoked by the compiler, e.g., for array copying/initializing). As a goal, it should be fairly simple to add support for any new target, e.g., WebAssembly and a new OS like your PowerNex.

Am Thu, 21 Mar 2019 19:50:49 +0000 schrieb Dan Printzell:

> For the betterC topic I would be interested in implementing a bare metal
> D runtime[5]. This is something I already have some experience as I have
> written a betterC runtime for PowerNex, and I also used to
> update/maintain Adam D. Ruppe's minimal.zip[6] when I used that as the
> runtime for PowerNex.
> My question would be: What features should this runtime contain?
>  From the wiki I
> see that basic libc functionality is requested, like the str* functions
> and memory management. From what I can understand, the goals it to have
> a runtime that can be used on multiple targeted environments, without
> any external dependencies.
> 
> Other questions regarding this topic: how separate should it be from
> druntime?
> Should this be a separate library, like a druntime-lite? or is the goal
> to allow druntime to be used for a bare metal target? Also, should this
> only be betterC or should it support regular D as well?
> 
> 

I think what we should do for betterC is to first replace the extern(C) hooks with a defined API. Right now, the compiler does not know if a runtime has feature X, it simply calls an extern(C) function _featureX(). If this functions is not in the runtime, this fails at link time only. A better approach would be to use a D function featureX in object.d. The compiler can then import object.d and see if there is a function featureX. If not, the runtime does not support the used feature and the compiler should emit an error.

This way, we can implement a truly modular runtime. Just implement some feature and the compiler will allow using it, do not implement it and the compiler will error if you try to use it. In the extreme case, this means we can use a D compiler without any runtime and it will gracefully degrade to a reduced subset.

Ideally, we can then implement different parts of the runtime in dub sub-
packages. We could then ship a gdc for embedded targets without any
default runtime and it would just work for some basic runtime. If you
need classes, constructors, ... include microdrt:core. If you need
unittests include microdrt:unittest (:exceptions, :gc, ...). For the C
library, we should have some extra package stdc. We might also want a
package for device-independent, portable code (volatile wrapper,
traits, ...). We could then also use different configurations for
microdrt:core, where the default 'ansic' would just use ansic features
and be portable everywhere where a C lib is available, 'native' would
work without a C lib, etc.

https://wiki.dlang.org/GSOC_2019_Ideas#GDC_.28GCC-based_D_compiler.29

Whether this should replace druntime in the long term is probably a very difficult question. I'd say start small, with a minimal core runtime which is suitable even for the smallest 8bit microcontrollers. Getting the full featureset of druntime in there in a modular runtime is probably too much work for GSoC, but as a long-term goal it would be prefereable to only have one runtime. However, druntime is not the place for such radical restructurings.

-- 
Johannes

On Thursday, 21 March 2019 at 21:54:12 UTC, Johannes Pfau wrote:
> I think what we should do for betterC is to first replace the extern(C) hooks with a defined API. Right now, the compiler does not know if a runtime has feature X, it simply calls an extern(C) function _featureX(). If this functions is not in the runtime, this fails at link time only. A better approach would be to use a D function featureX in object.d. The compiler can then import object.d and see if there is a function featureX. If not, the runtime does not support the used feature and the compiler should emit an error.

These hooks are what I meant by 'functions implicitly invoked by the compiler', and the defined API being the (or a subset of) the public exports of the stripped-down runtime.

How do you propose to solve the `object.d` issue of having to accomodate for absence/availability of all subpackages with a highly modular approach? I know that types (TypeInfo, ModuleInfo) are currently handled like this (compiler checks if they are defined in object.d and only emits those symbols if they are), but that only works because they reside themselves in an already way too bloated object.d.

On Thursday, 21 March 2019 at 22:30:32 UTC, kinke wrote:
> How do you propose to solve the `object.d` issue of having to accomodate for absence/availability of all subpackages with a highly modular approach?

Ah, dub's predefined versions `Have_*` should work indeed.

On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:
> Other questions regarding this topic: how separate should it be from druntime?
> Should this be a separate library, like a druntime-lite? or is the goal to allow
> druntime to be used for a bare metal target? Also, should this only be betterC
> or should it support regular D as well?

I'm honestly not sure what the scope of the project is, but one option is to make the normal D runtime work with Newlib or some other portable libc for embedded systems.  That would create a lot of possibilities (even if not all D runtime features are supported).  It would also help define the minimal requirements for the D runtime, making pure-D bare metal easier.

BTW, PowerNex is a cool project.

On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:

> I am planning on applying to GSoC but I'm having a bit of a problem trying to
> decide on what I should do. My goal is to try and do something that I will
> personally use in PowerNex and it is something that will help the whole D
> community, this is why I am looking at dmd and betterC topics.

Hi Dan,

I've been a fan of your work, benefit greatly from your Arch Linux contributions, and admire what you've accomplished so far.  (￣^￣)ゞ  <-- That's supposed to be a salute.

I have been dreaming about this kind of work for years and did a lot of work in the 2017~2018 time-frame to move it along, but opportunity cost forced me to, at least temporarily, to focus my efforts elsewhere.  You're probably a much better D programmer and software engineer than I am, but I believe I can help you quite a bit in your quest if you're willing to engage me and hear me out.

Here are a few suggestions I have that I think will have the most beneficial impact on your existing goals and D as a whole

# Suggestion 1 - Replace Runtime Hooks with Templates
IMO the most important thing you can do to help D become more scalable to bare-metal-like platforms is to replace the runtime hooks [1] with templates.  I think this is what Johannes was alluding to in his reply.  This will not only remove the runtime/artificial dependency on `TypeInfo` (and therefore a dependency on classes as a whole), but it will also make druntime a "header-only" library that only requires libc as a dependency.  Since -betterC only imports druntime, but doesn't link to it, it also will expand D language feature support for -betterC (though if this work were done, -betterC could even become obsolete).  See my description at [2] and ask away if you'd like more details.  I'm not really qualified to mentor such a project, but I believe I can significantly reduce the barrier to entry should you wish to pursue this.

# Suggestion 2 - Create a No-dependency D Library
There are a lot of great utilities in Phobos that do not require the druntime (e.g. std.meta, std.traits, std.conv, and probably more).  What we need is a library that has NO DEPENDENCIES WHATSOEVER that can be imported into druntime and DMD to aid in their development.  Manu hinted on this with a proposal for `core.traits` [3], but I don't think we should stop there, and I had more to say about in a bit of a passionate brainstorm [4].  This no-dependency-library, by its inherent nature, would bring a lot of the great features of Phobos and D to all kinds of programming, including bare-metal-like platforms.  It would also make DMD and druntime development much more productive and idiomatic D.

# Suggestion 3 - Port the Entire Runtime to ARM Cortex-M, RISCV, X64, {Insert your favorite hardware architecture}
IMO, Suggestion 1 and 2 are prerequisites for this, but if you really want to, try to get druntime, in its entirety, ported to a bare-metal platform of your choice.  You'll have to find some way to implement dynamic memory allocation and threads, but once that is done, I think it should be possible.  Though I recommend making it easy on yourself and implement Suggestion 1 or 2 first.

# Suggestion 4 - Replace Software Building Blocks with Implementations in D.
By software building blocks, I mean `memcpy`, `memcmp`, `malloc`, `free` etc.  This is a very unpopular topic, but if you were to work on Suggestion 1 and 2, you'd likely see this as a practical and worthwhile goal. If you're like most, however, you're probably thinking this is a waste of resources, so I'll defer elaborating on this for now.  Though, if you do happen to see what I see, go for it.

I am by no means an expert in DMD and druntime, but I did stumble around enough in the code in the 2017~2018 time-frame to know how to get started.  I'm not qualified to mentor such a project but I do believe I can help reduce the barrier to entry.  Let me know if you're interested in engaging with me and I'll do my best to help.  You can reach me on Slack under the handle JinShil, and I think you're clever enough to get my e-mail through the same handle at GitHub, should you choose to contact me that way.  Or we can just go nuts in this thread if you'd like.

Good Luck!

Mike

[1] - Runtime Hooks on the Wiki - https://wiki.dlang.org/Runtime_Hooks
[2] - Info on Replacing Runtime Hooks with Template - https://forum.dlang.org/post/hequvmostuhwnvcgrcge@forum.dlang.org
[3] - Move `std.traits` to druntime - https://forum.dlang.org/post/mailman.6241.1546722791.29801.digitalmars-d@puremagic.com
[4] - Me Dreaming About a More Scalable D - https://forum.dlang.org/post/dvbxqrcmyltgnikxdweb@forum.dlang.org

On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:

I'll attempt to answer your specific questions below.

> For dmd I am primarily interested in implementing DIP1014[3], as this will help
> me with the development of PowerNex. In the past I had to workaround that this
> feature was missing. From the outside is seems like this would not take too long
> to implement, at least not a whole GSoC. But as I'm not too familiar with dmd's
> source code, I only did a few small patches to allow dmd to support PowerNex as
> a target[4].
> So my question for this topic would be: How long time do you guys think that
> this will take to implement, test and get it merged? Is a month about correct?

I'm afraid I haven't looked into this and I don't have anything to share. Sorry.

> For the betterC topic I would be interested in implementing a bare metal D
> runtime[5]. This is something I already have some experience as I have written a
> betterC runtime for PowerNex, and I also used to update/maintain Adam D. Ruppe's
> minimal.zip[6] when I used that as the runtime for PowerNex.
> My question would be: What features should this runtime contain?

As I see it there are 3 different levels:

Level-0:  No dependencies whatsoever. Level-0 would likely only include utilities, but more features could be added if you chose to import `memcpy`, `memset`, and `memcmp` from libc.  However, I would prefer if those software building blocks were implemented in D.  See https://github.com/JinShil/memcpyD for an unfinished exploration.  Level-0 would not include dynamic heap memory allocation, but could utilize dynamic stack memory allocation.  LDC and GDC have alloc out of the box, but DMD doesn't unfortunately, so you might have to hack something together with static arrays and runtime error-checking. See more about this at https://forum.dlang.org/post/yyylugjalnlawjvxaqfp@forum.dlang.org

Level-1:  Imports Level-0, but also adds dynamic heap memory allocation and any D language feature implementations that depend on it.  You could import libc for `malloc`, `free`, and friends, or they too could be implemented in D super extra credit.

Level-2:  Imports Level-1, but also adds threads.  Once threads are implemented D features such as thread local storage and the GC become possible.  Subsequently, with the GC, you can now implement classes and the rest of druntime.  I think you could implement the GC in Level-1, but I haven't thought it through yet.

If those three levels were distributed separately, a user could potentially opt into whatever level they desire for their application.

> From the wiki I
> see that basic libc functionality is requested, like the str* functions and
> memory management. From what I can understand, the goals it to have a runtime
> that can be used on multiple targeted environments, without any external
> dependencies.

Yes, no dependencies on anything non-D is my preference.  It is possible to replace the facilities currently being leveraged from libc, but will take a lot of work, and a significant project itself.  For the short-term, however, we can tolerate a libc dependency, or you could make replacing these facilities your project.

> Other questions regarding this topic: how separate should it be from druntime?
> Should this be a separate library, like a druntime-lite? or is the goal to allow
> druntime to be used for a bare metal target? Also, should this only be betterC
> or should it support regular D as well?

I don't really think about it as being separate from druntime.  I see it as just an alternate implementation of druntime; one that is more scalable and pay-as-you-go than what we currently have.

> Depending on the implementation size of DIP1014 and the bare metal D runtime,
> it seems like both could be implemented during a GSoC. But I am probably just
> being time optimistic here.

Sound WAY too optimistic for me, but who am I to judge?

I'll restate, briefly, what I said in my other post, that I think there are some fundamental changes to DMD and druntime (e.g. replacing runtime hooks with templates) that should probably be implemented first as prerequisites to all of this, and I can help with that.

Mike

On Thursday, 21 March 2019 at 21:54:12 UTC, Johannes Pfau wrote:

> I think what we should do for betterC is to first replace the extern(C) hooks with a defined API. Right now, the compiler does not know if a runtime has feature X, it simply calls an extern(C) function _featureX(). If this functions is not in the runtime, this fails at link time only. A better approach would be to use a D function featureX in object.d. The compiler can then import object.d and see if there is a function featureX. If not, the runtime does not support the used feature and the compiler should emit an error.
>
> This way, we can implement a truly modular runtime. Just implement some feature and the compiler will allow using it, do not implement it and the compiler will error if you try to use it. In the extreme case, this means we can use a D compiler without any runtime and it will gracefully degrade to a reduced subset.

Yes, this is very similar to what I've been thinking, with regard to having the runtime inform the compiler about what features of the language are and aren't supported by the imported runtime.  I've already implemented something very similar to that for `ModuleInfo`, `TypeInfo`, and `Exception` and that code in the compiler today.

https://github.com/dlang/dmd/pull/7395
https://github.com/dlang/dmd/pull/7768
https://github.com/dlang/dmd/pull/7799
https://github.com/dlang/dmd/pull/7786

It allows the compiler to use design by introspection to determine what has and hasn't been imported, and generate code accordingly, or emit an error if something is required but can't be found.

It's no different than how we work with code and imported modules in our editor.  For example, if you write a call to a function, but you failed to import it; the compiler emits an error telling you it can't be found.  If you try to use a language feature that isn't implemented and imported from druntime, the compiler can let you know, at compile-time, that you're trying to utilize a feature that doesn't have an implementation.

But, it's more powerful than that.  If the code passing through the compiler isn't utilizing certain features like `ModuleInfo`, `TypeInfo` and `Exceptions`, then there's no reason for the compiler to complain if that stuff doesn't exist in the imported runtime. That's what the above pull requests fixed.  Prior to those PRs we had to implement a bunch unused boilerplate just to get a build, and none of that code ever ended up in the resulting binary anyway.  Ridiculous.

Best of all, Walter seemed to like it, which was a significant morale boost. https://github.com/dlang/dmd/pull/7395#issuecomment-349200847

Mike

Am Thu, 21 Mar 2019 22:46:14 +0000 schrieb kinke:

> On Thursday, 21 March 2019 at 22:30:32 UTC, kinke wrote:
>> How do you propose to solve the `object.d` issue of having to accomodate for absence/availability of all subpackages with a highly modular approach?
> 
> Ah, dub's predefined versions `Have_*` should work indeed.

Something like that or we could do a conditional, public import (static if(__traits(compiles, import ...)). Alternatively we could of course split the API into different modules but that would be more difficult to get upstreamed.

-- 
Johannes