Read on GitHub: https://github.com/JinShil/minimal_druntime_experiment

There was recently a discussion about how we could create a portable, pay-as-you-go, D runtime to help bring the promise of D to free-standing platforms and devices with tight resource constraints (e.g. microcontrollers).  Thread started here:  http://forum.dlang.org/post/mhrs4p$31id$1@digitalmars.com

The primary motivation is to create an arm-none-eabi GDC cross-compiler toolchain for programming ARM Cortex-M microcontrollers in D, but I think there's a way to achieve broader platform support by delegating implementation details down the "supply chain".  I hope to articulate that strategy in this post.

To prove the concept, provide a place to start, and discuss ideas, I created the most minimal D runtime I could:  https://github.com/JinShil/minimal_druntime_experiment.  I've also included Phobos, so we could still have `std.stdio.write` and `std.stdio.writeln` for console output, as every device needs a console for development.

Overview
**********
d
├── phobos
│   └── std
└── runtime
    └── rt
        └── typeinfo

ports
├── arm
│   └── cortexm4
│       ├── phobosWe
│       │   └── phobosPort.d
│       └── runtime
│           └── runtimePort.d
├── posix
│   └── linux
│       ├── phobos
│       │   └── phobosPort.d
│       └── runtime
│           ├── c_types.d
│           └── runtimePort.d


There are two main folders: "d" and "ports".  "d" provides the patform-agnostic code, or code that is relevant to a large number of platforms.  The "ports" directory provides the platform-specific implementation.  Building simply requires importing "d/runtime", "d/phobos", and your platform's hierarchy in the "ports" folder.  At the moment, I've only implemented a Linux 64-bit port and an ARM Cortex-M4 port to illustrate the concept.  This is roughly how I wish the official runtime could be structured in the spirit of Issue 11666.

The official runtime includes platform-specific bindings in `core.sys` and `stdc`, but I think that is a design anomaly.  While a port may find it convenient to use those bindings, they should not be part of the D language's public API.  Rather, they should be deported to Deimos, and if needed, imported privately by a platform's port.  For the Linux port, I chose to write the platform's system calls in assembly to illustrate that it is not even necessary to use those bindings if one wants to do the due diligence in D.

Porting to a New Platform
******************************
The platform-agnostic code in "d" delgates implementation details to the platform-specific code using `extern(C) extern _d_sys_name` "system calls" (for lack of a better term).  This is how the newlib C library delegates platform-specific implementations:  See http://wiki.osdev.org/Porting_Newlib#newlib.2Flibc.2Fsys.2Fmyos.2Fsyscalls.c

At the moment, for the sake of demonstration, only 2 system calls have been defined:

* `__d_sys_write` - Equivalent to C's `write` system call
* `__d_sys_exit` - Equivalent to C's `exit` system call

These two system calls allow us to create a simple Hello World.  "runtimePort.d" implements `__d_sys_exit` and "phobosPort.d" implements the `__d_sys_write`.

Putting it all Together
************************
Users are not expected to use this code directly.  Rather, I envision toolchain, silicon, and board vendors will use this code as a small, but essential part of, their platform's D programming package.  For example, a package for the ARM Cortex-M family of MCUs might contain the following:
* arm-none-eabi GDC cross-compiler
* a library containing compiled code for the "d" folder in this repository
* a library containing the compiled code for the "ports/arm/cortexm" folders in this repository
* cortex-m core startup files
* the newlib C library
* the C library bindings from Deimos
* multilibs from the GNU toolchain.

A silicon vendor like ST, may then take that package and add more platform-specific for their family of products:
* startup files with interrupt vectors for their peripherals
* linker scripts for each of their MCUs
* flash memory programmer
* library for on-dye peripherals
* etc..

A board vendor may choose to create yet another layer on top of the silicon vendor's package.
* library for peripherals external to the MCU (external RAM, IO expanders, gyroscope, LCD, etc...)

In short, this repository just provides just the foundation to "get D working", and delegates to toolchain, silicon, and board vendors to fill in the details for their products.

RFC
*****
For those who have stake in this code base (kernel developers, embedded developers, toolchain and package maintainers, etc...) your constructive criticism and ideas are most welcome.  I am unsure yet how this will play out, and what roles players will assume, but let's give it a try.

Plea to Compiler Implementers
***********************************
We need better control over codegen.  TypeInfo and dead-code removal is preventing me from making progress (http://forum.dlang.org/post/quemhwpgijwmqtpxukiv@forum.dlang.org).  I've resorted to compiling to assembly, using sed to hack the assembly, and then compiling the assembly.  Things like that make me want to not use D at all.  Here are some ideas from myself and others:
*  Move TypeInfo to the runtime: https://issues.dlang.org/show_bug.cgi?id=12270
*  Move D runtime hooks out of the compiler and into *.di files so they can be decorated with attributes and `version`ed so the compiler can see what features of D are supported by the port and generate smarter code:  http://forum.dlang.org/post/psssnzurlzeqeneagora@forum.dlang.org
*  Add -fno-rtti compiler switch
*  Add attribute support so programmer can choose which types to generate runtime-time info for.

I don't know what the right solution is.  Let's have that discussion.

LDC Folks:  https://github.com/ldc-developers/ldc/issues/781 is currently preventing me from supporting LDC with this runtime.

Mike

"Mike" <none@none.com> writes:

> Read on GitHub: https://github.com/JinShil/minimal_druntime_experiment
>
> There was recently a discussion about how we could create a portable, pay-as-you-go, D runtime to help bring the promise of D to free-standing platforms and devices with tight resource constraints (e.g. microcontrollers).  Thread started here: http://forum.dlang.org/post/mhrs4p$31id$1@digitalmars.com

This is cool stuff.  Embedded has been the bulk of my software life and is most fun.  I see there has been a lot of discussion that I'll have to read up on before I say much more though.

> * the newlib C library

newlib has treated me well in the past, especially being so highly customizable.  I think it is a good choice.  It would be fun to rewrite it in D, but that perhaps is much work for little immediate benefit.

>
> Plea to Compiler Implementers
> ***********************************
> We need better control over codegen.  TypeInfo and dead-code removal
> is preventing me from making progress
> (http://forum.dlang.org/post/quemhwpgijwmqtpxukiv@forum.dlang.org). I've
> resorted to compiling to assembly, using sed to hack the assembly, and
> then compiling the assembly.  Things like that make me want to not use
> D at all.

Yeah, this should be improved.

How about a -disable-tls option so that when there are no threads and thus no TLS, you can compile exising D code as-is.  Otherwise you have to rewrite normal variables to __gshared everywhere (I actually have this option in a ldc fork).

> LDC Folks:  https://github.com/ldc-developers/ldc/issues/781 is currently preventing me from supporting LDC with this runtime.

I looked at this and found a workaround and noted it in the above issue link.
-- 
Dan Olson

On 5 May 2015 at 08:39, Dan Olson via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> "Mike" <none@none.com> writes:
>
>> Read on GitHub: https://github.com/JinShil/minimal_druntime_experiment
>>
>> There was recently a discussion about how we could create a portable, pay-as-you-go, D runtime to help bring the promise of D to free-standing platforms and devices with tight resource constraints (e.g. microcontrollers).  Thread started here: http://forum.dlang.org/post/mhrs4p$31id$1@digitalmars.com
>
> This is cool stuff.  Embedded has been the bulk of my software life and is most fun.  I see there has been a lot of discussion that I'll have to read up on before I say much more though.
>
>> * the newlib C library
>
> newlib has treated me well in the past, especially being so highly customizable.  I think it is a good choice.  It would be fun to rewrite it in D, but that perhaps is much work for little immediate benefit.
>
>>
>> Plea to Compiler Implementers
>> ***********************************
>> We need better control over codegen.  TypeInfo and dead-code removal
>> is preventing me from making progress
>> (http://forum.dlang.org/post/quemhwpgijwmqtpxukiv@forum.dlang.org). I've
>> resorted to compiling to assembly, using sed to hack the assembly, and
>> then compiling the assembly.  Things like that make me want to not use
>> D at all.
>
> Yeah, this should be improved.
>
> How about a -disable-tls option so that when there are no threads and thus no TLS, you can compile exising D code as-is.  Otherwise you have to rewrite normal variables to __gshared everywhere (I actually have this option in a ldc fork).
>

Configure GDC with --disable-tls --disable-threads to get the desired behaviour (disabling only TLS just makes codegen fallback to emulated thread support).

On Tuesday, 5 May 2015 at 06:56:52 UTC, Iain Buclaw wrote:
>
> Configure GDC with --disable-tls --disable-threads to get the desired behaviour (disabling only TLS just makes codegen fallback to emulated thread support).

I didn't know this.  You mean if I configure with those options,
TLS variables will be treated as __gshared?

Mike

On Tuesday, 5 May 2015 at 06:39:42 UTC, Dan Olson wrote:

>
>> LDC Folks:  https://github.com/ldc-developers/ldc/issues/781 is
>> currently preventing me from supporting LDC with this runtime.
>
> I looked at this and found a workaround and noted it in the above issue
> link.

Many Thanks!

I've gotten further thanks to your help, but now I'm stuck on
https://github.com/ldc-developers/ldc/issues/925.

Mike

On 5 May 2015 at 09:48, Mike via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Tuesday, 5 May 2015 at 06:56:52 UTC, Iain Buclaw wrote:
>>
>>
>> Configure GDC with --disable-tls --disable-threads to get the desired behaviour (disabling only TLS just makes codegen fallback to emulated thread support).
>
>
> I didn't know this.  You mean if I configure with those options, TLS variables will be treated as __gshared?
>
> Mike

I have no control over what the backend does, but if it generates control variables for them (emulated tls), then __get_address would incur a small heap cost when first retrieved.

If the cost is too much (ie: there is no heap), then we should be able to stop emutls from kicking in by preventing the D frontend from returning true on isThreadLocal calls.

"Mike" <none@none.com> writes:

> On Tuesday, 5 May 2015 at 06:39:42 UTC, Dan Olson wrote:
>
>>
>>> LDC Folks:  https://github.com/ldc-developers/ldc/issues/781 is currently preventing me from supporting LDC with this runtime.
>>
>> I looked at this and found a workaround and noted it in the above
>> issue
>> link.
>
> Many Thanks!
>
> I've gotten further thanks to your help, but now I'm stuck on https://github.com/ldc-developers/ldc/issues/925.
>
> Mike

Mike, Just updated the issue (you probably got an email). It is caused by a change from 2.066 to 2.067 in DMD front end versions, not specific to LDC.

Iain Buclaw via Digitalmars-d <digitalmars-d@puremagic.com> writes:

> If the cost is too much (ie: there is no heap), then we should be able to stop emutls from kicking in by preventing the D frontend from returning true on isThreadLocal calls.

Iain, I think something like this would be good in GDC.  Makes it so much easier to compile for no thread environments.

Am Tue, 05 May 2015 08:41:13 -0700
schrieb Dan Olson <zans.is.for.cans@yahoo.com>:

> Iain Buclaw via Digitalmars-d <digitalmars-d@puremagic.com> writes:
> 
> > If the cost is too much (ie: there is no heap), then we should be able to stop emutls from kicking in by preventing the D frontend from returning true on isThreadLocal calls.
> 
> Iain, I think something like this would be good in GDC.  Makes it so much easier to compile for no thread environments.

I'm not really sure about this. It's trivial to implement but it generates a semantic difference which harms code portability (e.g. if you have code where you want a global variable, don't mark it as __gshared and then move to an environment with threads).

Am Tue, 05 May 2015 02:26:26 +0000
schrieb "Mike" <none@none.com>:

> Read on GitHub: https://github.com/JinShil/minimal_druntime_experiment
> 
> There was recently a discussion about how we could create a portable, pay-as-you-go, D runtime to help bring the promise of D to free-standing platforms and devices with tight resource constraints (e.g. microcontrollers).  Thread started here: http://forum.dlang.org/post/mhrs4p$31id$1@digitalmars.com
> 
> The primary motivation is to create an arm-none-eabi GDC cross-compiler toolchain for programming ARM Cortex-M microcontrollers in D, but I think there's a way to achieve broader platform support by delegating implementation details down the "supply chain".  I hope to articulate that strategy in this post.
> 
> To prove the concept, provide a place to start, and discuss ideas, I created the most minimal D runtime I could: https://github.com/JinShil/minimal_druntime_experiment.  I've also included Phobos, so we could still have `std.stdio.write` and `std.stdio.writeln` for console output, as every device needs a console for development.
> 
> Overview
> **********
> d
> ├── phobos
> │   └── std
> └── runtime
>      └── rt
>          └── typeinfo
> 
> ports
> ├── arm
> │   └── cortexm4
> │       ├── phobosWe
> │       │   └── phobosPort.d
> │       └── runtime
> │           └── runtimePort.d
> ├── posix
> │   └── linux
> │       ├── phobos
> │       │   └── phobosPort.d
> │       └── runtime
> │           ├── c_types.d
> │           └── runtimePort.d
> 
> 
> There are two main folders: "d" and "ports".  "d" provides the patform-agnostic code, or code that is relevant to a large number of platforms.  The "ports" directory provides the platform-specific implementation.  Building simply requires importing "d/runtime", "d/phobos", and your platform's hierarchy in the "ports" folder.  At the moment, I've only implemented a Linux 64-bit port and an ARM Cortex-M4 port to illustrate the concept.  This is roughly how I wish the official runtime could be structured in the spirit of Issue 11666.
> 
> The official runtime includes platform-specific bindings in `core.sys` and `stdc`, but I think that is a design anomaly. While a port may find it convenient to use those bindings, they should not be part of the D language's public API.  Rather, they should be deported to Deimos, and if needed, imported privately by a platform's port.  For the Linux port, I chose to write the platform's system calls in assembly to illustrate that it is not even necessary to use those bindings if one wants to do the due diligence in D.
> 
> Porting to a New Platform
> ******************************
> The platform-agnostic code in "d" delgates implementation details
> to the platform-specific code using `extern(C) extern
> _d_sys_name` "system calls" (for lack of a better term).  This is
> how the newlib C library delegates platform-specific
> implementations:  See
> http://wiki.osdev.org/Porting_Newlib#newlib.2Flibc.2Fsys.2Fmyos.2Fsyscalls.c
> 
> At the moment, for the sake of demonstration, only 2 system calls have been defined:
> 
> * `__d_sys_write` - Equivalent to C's `write` system call
> * `__d_sys_exit` - Equivalent to C's `exit` system call
> 
> These two system calls allow us to create a simple Hello World. "runtimePort.d" implements `__d_sys_exit` and "phobosPort.d" implements the `__d_sys_write`.
> 
> Putting it all Together
> ************************
> Users are not expected to use this code directly.  Rather, I
> envision toolchain, silicon, and board vendors will use this code
> as a small, but essential part of, their platform's D programming
> package.  For example, a package for the ARM Cortex-M family of
> MCUs might contain the following:
> * arm-none-eabi GDC cross-compiler
> * a library containing compiled code for the "d" folder in this
> repository
> * a library containing the compiled code for the
> "ports/arm/cortexm" folders in this repository
> * cortex-m core startup files
> * the newlib C library
> * the C library bindings from Deimos
> * multilibs from the GNU toolchain.
> 
> A silicon vendor like ST, may then take that package and add more
> platform-specific for their family of products:
> * startup files with interrupt vectors for their peripherals
> * linker scripts for each of their MCUs
> * flash memory programmer
> * library for on-dye peripherals
> * etc..
> 
> A board vendor may choose to create yet another layer on top of
> the silicon vendor's package.
> * library for peripherals external to the MCU (external RAM, IO
> expanders, gyroscope, LCD, etc...)
> 
> In short, this repository just provides just the foundation to "get D working", and delegates to toolchain, silicon, and board vendors to fill in the details for their products.
> 
> RFC
> *****
> For those who have stake in this code base (kernel developers,
> embedded developers, toolchain and package maintainers, etc...)
> your constructive criticism and ideas are most welcome.  I am
> unsure yet how this will play out, and what roles players will
> assume, but let's give it a try.
> 

I would probably split the runtime into at least two, probably three parts:

* Compiler support library (object.d, gcc/*.d, exception
  implementation) (module rt/ no module name)
* Higher level library (portable, micro)
* Hardware specific library (not portable, avr/ stm/)

I think some basic portability between microcontrollers is useful. The compiler support library should be only a few 100 lines of code and should be without external dependencies. This way it's also useful for very small platforms (8bit) or use-cases where you inject code into other processes. It's trivial to port: we should have an ansi-c port and native ports should be implementable in 1-2 hours.


The higher level library should contain stuff like 'core.time.Duration',
emplace, allocator framework, Volatile!T, register wrapper types, etc.
The implementation could be hardware specific, but it should provide a
common interface (e.g. a delay!Duration function has got a common
interface but different implementation, malloc is similar).
Namespace micro => import micro.time; import micro.memory;
So everything in the micro namespace is portable.


The hardware library the provides access to hardware specific peripherals. In reality the high-level-library might require the hardware library or they could actually be merged. The important part is portable vs platform specific module API for user code.


A radically different approach instead of using ports directories is using git features: Have a base repository with a master branch which only includes the interfaces, probably with static assert wherever platform specific code is necessary. Then have AVR/STM32/LPC/... branches where you simply implement these functions. (We could also have different repositories instead of branches)

+ you can modify all code
+ you already start with a common interface
+ changes are easy to compare by comparing git branches
+ changes have descriptions (in their git commits)
+ it's easy to merge further generic changes

Phobos and core.stdc should then also be separate libraries.

> Plea to Compiler Implementers
> ***********************************
> We need better control over codegen.  TypeInfo and dead-code
> removal is preventing me from making progress
> (http://forum.dlang.org/post/quemhwpgijwmqtpxukiv@forum.dlang.org).
>   I've resorted to compiling to assembly, using sed to hack the
> assembly, and then compiling the assembly.  Things like that make
> me want to not use D at all.  Here are some ideas from myself and
> others:
> *  Move TypeInfo to the runtime:
> https://issues.dlang.org/show_bug.cgi?id=12270
> *  Move D runtime hooks out of the compiler and into *.di files
> so they can be decorated with attributes and `version`ed so the
> compiler can see what features of D are supported by the port and
> generate smarter code:
> http://forum.dlang.org/post/psssnzurlzeqeneagora@forum.dlang.org
> *  Add -fno-rtti compiler switch

This is something I could probably finish up in a few hours next weekend.

> *  Add attribute support so programmer can choose which types to generate runtime-time info for.

I can add an @attribute(notypeinfo) in GDC but it'll be implemented in the backend. This means we can prevent TypeInfo output but we can't reliably warn on TypeInfo usage. You'll get linker errors instead.