On Thursday, 4 April 2019 at 13:56:08 UTC, Stefanos Baziotis wrote:

> I'm mainly interested on the Baremetal D runtime project, as I am mostly comfortable
> with low-level C programming. I have developed some allocators [1]
> and have some SIMD experience as I have written a small image convolution MPI
> program that uses AVX extensions [2].

I'm sorry for responding late; I didn't see this thread until just a short while ago.

I think there are probably a few different ways to define a "bare metal runtime in D".  The way I define it is "port the existing D runtime to a processor that is not running an operating system".  The problem with that is D's runtime expects an underlying operating system.  So, one is left with a bit of a paradox:  which comes first the operating system or the runtime.

I would argue that if one is porting the existing D runtime to a processor without an operating system, the runtime becomes the operating system.  Therefore, one needs to implement dynamic memory allocation, threads, exceptions, thread-local storage, the garbage collector, etc. all in the runtime, and all unique to a specific processor or processor family.  Basically, one needs to build an operating system into the D runtime.
In a way, that's pretty cool, as D itself would then come with an operating system included, but is that really practical?

Perhaps what users are actually looking for is a minimal subset of D that doesn't require a runtime. That would likely mean no dynamic memory allocation, no threads, no thread-local storage, no GC, no classes, no exceptions, or some variant of that. That's more or less what -betterC provides.  Perhaps one could use -betterC to build the OS on which the full D runtime would be ported to.

What I would like to see is neither of those, but what Andrei recently proposed with this comment:

> Oh, and druntime must go.
>
> The whole distinction between the runtime library and the standard library is clowny and has more to do with poorly ported tradition from other languages, than with anything else.
>
> We need one standard library that is entirely pay-as-you-go (e.g. empty main() means no library file is ever needed for linking) and that offers an opt-in continuum.

In my opinion, that is smack dab right on and said better and more succinctly than I ever could.  But, how do we get there?

As I see it, we actually have the vast majority of it already implemented, but with all the dependencies on C, C++, operating system APIs, and a design philosophy the operating system and C/C++ toolchain would always be there from the beginning, we have a tangled mess.

So, we need to untangle everything and reduce the implementations down to individual, modular components with minimal dependencies that can be imported without bringing in everything else.  Here's Andrei delegating a few tasks to me:  https://www.youtube.com/watch?v=uUQeQ7E8hOU

A great help will be if Dan Printzell succeeds in eliminating the dependency of many runtime hooks on TypeInfo.  Since TypeInfo is a class, his project will help eliminate the dependency on classes, and since classes more-or-less require the GC, it removes the dependency of those runtime hooks on the GC as well. It will also decouple the compiler from the runtime allowing it to generate code with fewer dependencies.

However, those runtime hooks still require implementations in terms of memcpy, memcmp, malloc, free, etc.  Having those written in D as strongly-typed templates would further reduce dependencies on a C/C++ toolchain and standard library.  With Dan's work completed, the benefits of having memcpy, malloc and friends implemented in D would become much more apparent, as the new runtime template lowerings could then be implemented in D all the way down the stack utilizing CTFE, design by introspection, and all of D's compiler guarantees such a safety and purity.

Finally, I think it would help greatly if we could detangle and pull out those features of Phobos that don't require any runtime support (e.g. std.traits, std.meta) that would provide a base, lightweight, no-dependency utility library for all D platforms including bare metal.

Mike

On Saturday, 6 April 2019 at 18:56:42 UTC, Stefanos Baziotis wrote:

> Unfortunately, I know too little about how the GSoC works from the organization-side, but
> from what I know, it should be available for potential mentors through the Summer of Code platform (if it's not, it is very unfortunate as I'm waiting 2 days :P ).
> I suppose I can send it here too, as I've seen another guy do it.
> In any case, this is the proposal: https://docs.google.com/document/d/1B3YbjY70iVgGT7hq_16tBriLuCAUtjFR0gf0xkxmQ5c/edit?usp=sharing
>
> Any comments are welcome!

I think the Bare Metal Runtime suggestion on the GSoC ideas page conflated a few separate projects into one.  My other post in this thread elaborates on that.

I don't have much, if anything, to offer, as I've never participated in GSoC or written a similar proposal.  It looks like an awful lot of work, but I hope the project gets selected and is a success.  I think it would be great for D in the long run.

I'm JinShil, the author of memcpyD and some of the other projects linked on the GSoC ideas page.  My interest is mostly in microcontrollers and embedded systems, and I've only dabbled in memcpy and malloc/free implementations, but you're welcome to contact me if you have anything you'd like to discuss.

Good Luck!

Mike

On Saturday, 6 April 2019 at 19:48:42 UTC, Mike Franklin wrote:
> I'm sorry for responding late; I didn't see this thread until just a short while ago.

No problem at all, I responded late as well.

> Basically, one needs to build an operating system into the D runtime.
> In a way, that's pretty cool, as D itself would then come with an operating system included, but is that really practical?

While that would be a very interesting hobby project, it
is as you say not very practical and most of all, it would
probably not help the community that much.

> As I see it, we actually have the vast majority of it already implemented, but with all the dependencies on C, C++, operating system APIs, and a design philosophy the operating system and C/C++ toolchain would always be there from the beginning, we have a tangled mess.
>

I totally agree with all of these. What I roughly got from the project
description was to achieve what Andrei said but without the dependencies to
so much existing C/C++ code.

> A great help will be if Dan Printzell succeeds in eliminating the dependency of many runtime hooks on TypeInfo.  Since TypeInfo is a class, his project will help eliminate the dependency on classes, and since classes more-or-less require the GC, it removes the dependency of those runtime hooks on the GC as well. It will also decouple the compiler from the runtime allowing it to generate code with fewer dependencies.
>

Actually Dan started on the Baremetal project at first (you probably
know that) and then moved to that project after discussion
in the forum. The fact that Dan works on that makes me doubly
excited to work on this. I think the result would much better than
simply -betterC with a couple more utilities.

> However, those runtime hooks still require implementations in terms of memcpy, memcmp, malloc, free, etc.  Having those written in D as strongly-typed templates would further reduce dependencies on a C/C++ toolchain and standard library.  With Dan's work completed, the benefits of having memcpy, malloc and friends implemented in D would become much more apparent, as the new runtime template lowerings could then be implemented in D all the way down the stack utilizing CTFE, design by introspection, and all of D's compiler guarantees such a safety and purity.
>
> Finally, I think it would help greatly if we could detangle and pull out those features of Phobos that don't require any runtime support (e.g. std.traits, std.meta) that would provide a base, lightweight, no-dependency utility library for all D platforms including bare metal.
>
> Mike

Great proposals Mike. My answers might seem short but really you solved
most of my quetsions. So what do you believe is the best way I can help?

- Stefanos

On Saturday, 6 April 2019 at 20:07:34 UTC, Mike Franklin wrote:
> I think the Bare Metal Runtime suggestion on the GSoC ideas page conflated a few separate projects into one.  My other post in this thread elaborates on that.
>
> I don't have much, if anything, to offer, as I've never participated in GSoC or written a similar proposal.  It looks like an awful lot of work, but I hope the project gets selected and is a success.  I think it would be great for D in the long run.

I think too and I would like to work on that. Sadly, there's still
no proposed mentor.

> I'm JinShil, the author of memcpyD

It provided a lot of inspiration mate!

> you're welcome to contact me if you have anything you'd like to discuss.
>
> Good Luck!
>
> Mike

I hope we talk soon. Thanks for the help.

On Saturday, 6 April 2019 at 18:52:04 UTC, Stefanos Baziotis wrote:
> On Saturday, 6 April 2019 at 17:50:39 UTC, IGotD- wrote:
>> On Thursday, 4 April 2019 at 13:56:08 UTC, Stefanos Baziotis wrote:
>>> [GSoC 2019] Interested in Baremetal D Runtime and project ideas
>>>
>>> Hello,
>>>
>>> [...]
>>
>> What is in the runtime we need for baremetal projects that betterC doesn't provide?
>
> There are a number of routines for which D currently calls the C standard library equivalent. For example, memcpy, memmove etc. and also malloc(), free() etc.
> The idea is that these should be implemented in D to provide a more baremetal
> environment for D, for limited resources (aka embedded systems). Note that the C standard library provides a lot of benefits for the general case, mainly that it has been used and tested for years.
>
> https://wiki.dlang.org/GSOC_2019_Ideas#Baremetal_D_runtime

memcpy, memmove and such are heavily architecture dependent and needs an implementation for each architecture (even on sub architecture level) in assembler in order to be optimized. At least this what I've seen for the C/C++ compilers. We can have a fallback in D though. Correct me if I'm wrong, doesn't LLVM provide with such primitives, at least partly?

When it comes to malloc and free, I would suggest we have just hooks for these. Baremetal people have very different opinions what allocator suits their needs. Example allocators are of course welcome but being able to change to whatever allocator you want is essential for baremetal.

On Saturday, 6 April 2019 at 17:51:31 UTC, Mike Franklin wrote:
> On Saturday, 6 April 2019 at 16:23:28 UTC, Stefanos Baziotis wrote:
>> On Thursday, 4 April 2019 at 13:56:08 UTC, Stefanos Baziotis wrote:
>>> [GSoC 2019] Interested in Baremetal D Runtime and project ideas
>>>
>>
>> I have submitted a proposal (before 2 days) for D Baremetal Runtime. Any feedback is very welcome. :)
>
> Where can one find this proposal?
>
> Mike

You need to be a GSoC "Dementor" to see the proposal. I just sent you an invitation ;-)

On Saturday, 6 April 2019 at 21:31:19 UTC, IGotD- wrote:
> memcpy, memmove and such are heavily architecture dependent and needs an implementation for each architecture (even on sub architecture level) in assembler in order to be optimized. At least this what I've seen for the C/C++ compilers. We can have a fallback in D though. Correct me if I'm wrong, doesn't LLVM provide with such primitives, at least partly?

I believe that you are correct about LLVM [1]. It seems though
that it calls the C standard library.

I would be glad to listen to the creators of the project idea
about the clear goals of this project, but this is my take:

1) Independency in the C standard library. Being dependent on the
C standard library means that users have to have a C toolchain.

2) It tries to target limited resources systems and as such
the focus will be in those archictures.

3) C versions are nowadays quite complex, so they might want
simpler versions.

4) As Mike states in github, take advantage of the templates and
compile-time execution for potentialy more efficient code and also
the strong type system to leverage features such as @safe.

> When it comes to malloc and free, I would suggest we have just hooks for these. Baremetal people have very different opinions what allocator suits their needs. Example allocators are of course welcome but being able to change to whatever allocator you want is essential for baremetal.

I think that the reasons to implement malloc() and free() are
the same with those of memcpy etc.

Any enhancement on the reasons is welcome.

[1] https://llvm.org/docs/LangRef.html#standard-c-library-intrinsics

- Stefanos

On Saturday, 6 April 2019 at 22:11:51 UTC, Seb wrote:
>
> You need to be a GSoC "Dementor" to see the proposal. I just sent you an invitation ;-)

Seb, do you know if anyone will mentor this?

On Saturday, 6 April 2019 at 21:31:19 UTC, IGotD- wrote:

> memcpy, memmove and such are heavily architecture dependent and needs an implementation for each architecture (even on sub architecture level) in assembler in order to be optimized. At least this what I've seen for the C/C++ compilers.

This has been my observation as well.  However, DMD only supports Intel architectures.  There's still a wide range of feature differences in the Intel family, but those can all be managed with information like https://dlang.org/phobos/core_cpuid.html

I assume that this GSoC project would focus only on Intel architectures that DMD supports, and even then it could always fall back to C's `memcpy`, `memcmp`, etc. for those architectures that the implementation has not been optimized for.

For LLVM and GDC, they'll probably have to do their own thing for all the architectures they support, but I believe DMD could set a precedent.

I encourage this project to focus on the fact that this D, not C, and there's no reason to follow the C way.  For example, `memcpy` in D would probably manifest itself as a template like `memcpy(T)(T* dest, const T* src)` or something like that. `void*` should go the way of the dodo if at all possible.  Recognize also that users will likely never need to call `memcpy` directly in D.  They would use the wealth of features already in the D language, and the compiler should lower those features to the `memcpy` implementation on behalf of the user.

> When it comes to malloc and free, I would suggest we have just hooks for these. Baremetal people have very different opinions what allocator suits their needs. Example allocators are of course welcome but being able to change to whatever allocator you want is essential for baremetal.

The implementations in druntime need to be ported to bare-metal with facilities like `version(X86_64)` and `version(FreeStanding)`, both of which already exist at https://dlang.org/spec/version.html  This could be the default implementation, and it should be possible to build something in that would allow the user to override the default.

Mike

On Saturday, 6 April 2019 at 20:22:19 UTC, Stefanos Baziotis wrote:

> I totally agree with all of these. What I roughly got from the project
> description was to achieve what Andrei said but without the dependencies to
> so much existing C/C++ code.

I don't see that Andrei objects much to the C/C++ dependency.  Afterall, he recently approved adding the C++ standard library to the druntime: https://github.com/dlang/druntime/pull/2286

In fact, I find I'm in the minority when it comes to wanting to replace `memcpy` and friends with templates.  It's hard to advocate for it right now because there isn't actually a way to utilize it.  That's why Dan's project is so important.  Once Dan is done and the other runtime hooks are converted to templates the benefit of replacing `memcpy` and friend will become more apparent.  Until that work happens, we can't really utilize the templatized versions of `memcpy` and friends and so it will be difficult to measure real-world benefits.  Once done however, could replace `memcpy` and friends in the runtime and run some real-world benchmarks on the DMD compiler, Phobos, and other D projects to measure real benefit.  And that's not to mention turtles-all-the-way-down safety and purity guarantees provided by the compiler.

I'm not sure who posted the original GSoC idea, so they'd have to speak for themselves, but it appears to be a synthesis of things I have been advocating for for a long time.  It's not really about creating a bare metal runtime, it's mostly just about being able to use D for bare-metal programming in a pay-as-you-go fashion.

> So what do you believe is the best way I can help?

Where to begin?  There's so much work that needs to be done, I'm not even sure where to start.  At the moment, I'm just trying to clean up some unfinished business in the compiler like converting the last remaining C files in DMD to D, following through on deprecations so we can be liberated from some technical debt, and hopefully getting around to finishing https://github.com/dlang/dmd/pull/9068 which is a prerequisite to refactoring druntime enabling much of what I've been talking about.

Probably the best thing you can do is focus on your proposal and make it great.  I hope those reviewing the proposals will be able to see the relationship and natural progression with Dan's proposal and yours and both will be selected.

For the past 5 years this has been my experience working on D:  https://www.youtube.com/watch?v=AbSehcT19u0

If you're anxious to get your hands dirty, you can start by choosing a not-yet-converted runtime hook from https://wiki.dlang.org/Runtime_Hooks and have at it.  See https://github.com/dlang/druntime/pull/2508 for another recent example of another contributor getting busy.

https://github.com/dlang/dmd/pull/9068 is a low barrier to entry task that would help us out a lot.

You can also start stripping out utilities from Phobos (e.g. std.traits, std.meta, std.conv, etc.) to create a no-dependency utility library.  Basically my "Suggestion 2" here: https://forum.dlang.org/post/qooasdrdlrqdhghjwmus@forum.dlang.org  It could potentially serve as a starting point for std.v2 and the opt-in continuum.  See https://forum.dlang.org/post/mailman.7799.1553797857.29801.digitalmars-d@puremagic.com for the thread where that was first proposed.

Mike