Am Sun, 01 Feb 2015 02:11:42 -0800
schrieb Walter Bright <newshound2@digitalmars.com>:

> On 2/1/2015 1:38 AM, Timo Sintonen wrote:
> > The one of major issues is: how to access hardware. We need a language feature to access hardware registers. This has been discussed twice. Both time you rejected anything but your own idea of library functions. You rejected anything anybody said. No serious programmer will write code that way. It worked in 80's when we had an uart with three registers 8 bit each. Now an usb or ethernet peripheral may have 100 registers 32 bit each.
> 
> core.bitop.volatileLoad() and volatileStore() are implemented, and do
> the job. They are compiler intrinsics that result in single
> instructions. They support 8, 16, 32 and 64 bit loads and stores.

I think everybody agreed that these low-level primitives can't be used in end-user code. We can generate nice wrappers (nicer than C code), which perform as well as C code, but only with force-inline _and_ enabled optimizations (we essentially need heavy constant folding).

We also need a pragma(address) to complement pragma(mangle).

Here's some proof-of concept running D on an 8bit AVR: https://github.com/D-Programming-microD/avr-playground/blob/master/src/test.d

WIP implementation of Volatile!T and a register string mixin: http://pastebin.com/sb58UW00

Volatile!T: Line 16
Volatile!T usage: Line 73

generateRegisterType: Line 150
usage: Line 353
sample output: 356


So I'd say there are not too many language problems, the main problem is runtime/compiler interaction:

* If you don't want to use any runtime at all that's actually the
  easier part. We'd need to implement a little more of betterC but this
  can be done easily. Mike would prefer the compiler to autodetect the
  capabilities of the runtime (implemented hooks) instead of compiler
  switches. That'd be better but some more work.

* Using only part of druntime is ugly. The one thing most people would
  probably like to strip out is the GC, but keep exception handling,
  threads, ... But the GC is everywhere: core.demangle doesn't work
  without, backtraces, exceptions, threads. Right now you either use
  all of druntime or nothing but it's not possible to use parts of
  druntime only, it's not modular enough.

Am Sat, 31 Jan 2015 22:37:19 -0800
schrieb Walter Bright <newshound2@digitalmars.com>:

> On 1/31/2015 9:21 PM, Mike wrote:
> > Is D's core team genuinely interested in this domain?
> 
> Yes.
> 
> 
> > If you are genuinely interested, are you committed?  And if so, what direction would you like to take?  So far, my ideas have been very unpopular and I'm growing weary fighting the current. How can I contribute in a way that gets us both where we want to go?
> 
> I don't recall what you've suggested in this vein that was very unpopular - can you please post an example?

Mike once complained that the runtime heavily relies on some high-level OS features not available on embedded devices (e.g. threads). That's OK if you actually have an OS, but druntime should be modular enough to run on systems without thread support.

He got some very direct responses that told him that if an OS doesn't have thread-support etc there's no use to run D on that. Responses like that obviously demotivate people.

IIRC he also proposed moving more of TypeInfo implementation to the runtime so TypeInfo layout can be modified by the runtime or even completely avoided by not implementing it in the runtime. The main response was that it complicates compiler code too much for little benefit. Obviously bare-metal programmers might disagree on the little benefit part and as Mike points out rust seems to do similar things.

On Sunday, 1 February 2015 at 11:22:04 UTC, Johannes Pfau wrote:
> which perform as well as C code, but only with force-inline

why is this still not part of the language? I'm not sure of anything else that has been repeatedly asked for without any good counterarguments.

On Sunday, 1 February 2015 at 06:37:27 UTC, Walter Bright wrote:
>
> I don't recall what you've suggested in this vein that was very unpopular - can you please post an example?

These are not in any particular order.  They are just how I remember them.

I don't care if everyone disagrees with them, but Rust has shown the benefits of a minimal, nimble runtime with modular language implementation [7], and I don't see why D can't compete and offer something even better.

Issue 11666 - Separate each platform's port to its own folder/file [1]
----------------------------------------------------------------------
Originally Ian Buclaw's idea, but the Bugzilla issue was filed by me.  The problem is it is difficult to see the abstractions in the runtime, because one has to navigate through hierarchies of `version`ing, making it difficult to port D to new platforms.  The solution is to organize it by platform/architecture separating each into its own folder.  While there were a few pull requests attempting to address this, they all failed, mostly due to the fact that participants could not agree how to organize the Linux/Posix platform headers.

That made me wonder: What in the world are the the platform bindings doing in druntime anyway, and why should they interfere 11666?  druntime is supposed to be the language implementation.  If platform bindings are needed, fine, get them from Deimos, and encapsulate them.  If users need platform bindings, they can get the from Deimos too, and the linker can sort things out.

I brought this up when the community was considering adding C++ standard library language bindings to druntime as well (yikes!).  I tried to intervene, and took a beating [2].  I was criticised for whining instead of contributing with pull requests.  Well, I did submit a pull request for the most unintrusive, uncontroversial, trivial thing I could think of to get the ball rolling [3].  But I was told I should discuss it on the forum.  I decided to stop the ridiculous cycle of discussion and debate, and did not go back to the forum, as it was clear where that would have ended up.

If druntime was simply the language implementation, 11666 would have been resolved without debate, and we'd have a well-structured code base for the community to gradually and incrementally bring D to more platforms and architectures, without having to go their own way with their own runtime and their own tools.

I think the stuff in core.sys.* needs to be deported to Deimos, and privately imported by any ports that may need it.  The current architecture of druntime is not very portable or scalable, especially to the bare-metal architectures.

Issue 13605 - Add ability to `version` a module declaration [4]
---------------------------------------------------------------
One of the issues brought up in 11666 was that using public imports introduces new namespaces.  So, in my own experiments, I found that if I only had a way to `version` a module, I could potentially achieve better encapsulation of a given port.  So, I filed this enhancement request. Although this enhancement changes nothing for anyone, and simply removes an arbitrary limitation, it was still controversial.

Issue 12270 - Move TypeInfo to the D Runtime [5]
------------------------------------------------
If you remember from my and Adam Ruppe's DConf talk, we both mentioned the need to stub out TypeInfo in our runtime.  It's a silly, but effective hack.  Adam proposed an idea to move TypeInfo to the runtime, and I decided to log it in Bugzilla because I thought it was a great idea.  But more importantly it's a great example for future precedent.

There seems to be something inherently wrong with the way the compiler is coupled to the runtime.  I can't put my finger on it, but perhaps you know what I'm talking about.  It seems the compiler needs to decouple from and delegate more to the runtime.  Issue 12270 is a great start in that direction, but unfortunately I seem to be the only one of two actually interested in it, and I'm disappointed to say I don't know how to implement it.  Furthermore, I suspect this may actually break a few things if it were implemented, and given the aversion to change in this community, my current belief is it would not be accepted even if it were implemented.

Moving runtime hook declarations to .di files in the runtime [6]
----------------------------------------------------------------
I want my runtime to inform the compiler which language features it supports have the compiler generate errors, at compile-time, if the user attempts to use a language feature that is not implemented.

Frustrated with the push-back from the larger community, I decided to take a lower profile and see if I could have more impact from the bottom up. I found GDC's code base a little easier to understand than DMD's, and the few bare-metal folks there are in this community all seem to hang out at D.GNU anyway.  So, I made a few propositions there, one of them being to move runtime hook declarations to the runtime (see the link).  None of them seemed to light anyone's fire, however, and I have found that I still don't understand the compiler well enough to do any of the implementations.  Iain has offered to help when he gets time, but he's certainly under no obligation to do so, and with the community largely silent on the idea, I'm unsure if it will be embraced by the community if it were implemented.

Bottom line is some features in D are too high-level and I need a way to inform users (at compile time) what language features are supported in the runtime port, and which features aren't.  Also, I think users should be able to make similar choices even if an implementation in the runtime exists.

Final Thoughts
--------------
The D language needs to be more nimble, pay-as-you-go, scalable, and portable.  For bare-metal programming, it appears the language designers have not really given much thought to it.  As I study Rust, I see that someone there is thinking about it and making design choices to support it rather elegantly. [7]

*  The D language itself should not be dependent on libc or any other library.  Ports of the D runtime may need those libraries, but the language itself should not.  libc bindings below in Deimos.
*  Those porting D to new platforms shouldn't have to port the entire feature set of D, especially for resource-constrained bare-metal ports.  They should be able to implement only what is applicable for their platform (of course I recognize there is a limit).  That is going to look very different for each platform, so D should be more modular and malleable.
*  Users should be able to make decisions about what features of D they want or don't want in their programs, even if a runtime implementation exists, and have those decisions enforced at compile-time.
*  Users and those porting D to new platforms should have some control over the code generation to make efficient binaries, especially for resource constrained systems.  But I also think the linker needs to take some responsibility here too.
*  I have more, but I'm already getting carried away.

I don't want to file more Bugzilla issues because I don't necessarily have the right answers here, and I think Bugzilla issues need to be specific and actionable.  Also, I want to help make them happen, and not just throw them in Bugzilla to rot.  Instead, I'd like some dialogue with those willing to work with me to find a reasonable solution and help with the implementation.  But I also need to know what the core team will welcome, and what they will not, before I decide to jump in with both feet.  The aversion to change has me very sceptical.

Mike

[1] - Separate each platform's port to its own folder/file - https://issues.dlang.org/show_bug.cgi?id=11666
[2] - core.spp - http://forum.dlang.org/post/sdrjfagsayomsngmeywp@forum.dlang.org
[3] - Move core.stdc to Deimos - https://github.com/D-Programming-Deimos/libc/pull/2
[4] - Add ability to `version` a module declaration - https://issues.dlang.org/show_bug.cgi?id=13605
[5] - Move TypeInfo to D Runtime - https://issues.dlang.org/show_bug.cgi?id=12270
[6] - Move runtime hook declarations to .di file in runtime - http://forum.dlang.org/post/psssnzurlzeqeneagora@forum.dlang.org
[7] - Writing Safe Unsafe and Low-Level Code in Rust - http://doc.rust-lang.org/0.11.0/guide-unsafe.html

On Sunday, 1 February 2015 at 11:22:04 UTC, Johannes Pfau wrote:
>
> So I'd say there are not too many language problems, the main problem
> is runtime/compiler interaction:
>
> * If you don't want to use any runtime at all that's actually the
>   easier part. We'd need to implement a little more of betterC but this
>   can be done easily. Mike would prefer the compiler to autodetect the
>   capabilities of the runtime (implemented hooks) instead of compiler
>   switches. That'd be better but some more work.
>
> * Using only part of druntime is ugly. The one thing most people would
>   probably like to strip out is the GC, but keep exception handling,
>   threads, ... But the GC is everywhere: core.demangle doesn't work
>   without, backtraces, exceptions, threads. Right now you either use
>   all of druntime or nothing but it's not possible to use parts of
>   druntime only, it's not modular enough.

Yes, I totally agree with this assessment.

Mike

On 1 February 2015 at 09:38, Timo Sintonen via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Sunday, 1 February 2015 at 06:37:27 UTC, Walter Bright wrote:
>>
>> On 1/31/2015 9:21 PM, Mike wrote:
>>>
>>> Is D's core team genuinely interested in this domain?
>>
>>
>> Yes.
>>
>>
>>> If you are genuinely interested, are you committed?  And if so, what
>>> direction
>>> would you like to take?  So far, my ideas have been very unpopular and
>>> I'm
>>> growing weary fighting the current. How can I contribute in a way that
>>> gets us
>>> both where we want to go?
>>
>>
>> I don't recall what you've suggested in this vein that was very unpopular - can you please post an example?
>
>
> The one of major issues is: how to access hardware. We need a language feature to access hardware registers. This has been discussed twice. Both time you rejected anything but your own idea of library functions. You rejected anything anybody said. No serious programmer will write code that way. It worked in 80's when we had an uart with three registers 8 bit each. Now an usb or ethernet peripheral may have 100 registers 32 bit each.
>
> There are workarounds:
> - disable optimization. the code works but is 3 times bigger and slower
> - GDC marks shared variables as volatile. This works mostly but is
> unstandard and unstable and may be removed at any time.
> - Have an separate templated data type which has its own operators. While
> this seems to work, it is a little complicated way to do things.
>
> ----
> There are some areas that might be developed.
>
> There are not many people here that work with microcontollers besides Mike and I. Many people make just web services. This is ok, the business is there. If bare metal is a goal for D then it is impostant that we processor people do not feel ourselves as second class people. Our needs and concerns should at least be noticed and respected by leaders.
>
> More practioal thing is the runtime library. Mike, Adam and some others have started with empty runtime and added only what they need. This does not get very far because mamy real D features need runtime functions. I may the only one to start with full runtime and remove anything that does not work. I have many D features working now, like exceptions.
>
> The runtime needs:
> - NoSystem should be a supported platform and not a build failure. The build
> system should leave out files and features that are not available.
> - Unrelated things should be split in separate files and unnecessary imports
> should be removed. Object.d is pulling in most of the library with its
> imports. Imports that are used only in unittests should be in unittest
> blocks. I know this is worked on.
> - Runtime library functions should not use gc and free all resources they
> use. I know this is also being worked on.
>

Talk is cheap, need more actions.

https://github.com/D-Programming-Language/dmd/pull/4367 https://github.com/D-Programming-Language/dlang.org/pull/881

Iain.

On 1 February 2015 at 11:28, Johannes Pfau via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> Am Sat, 31 Jan 2015 22:37:19 -0800
> schrieb Walter Bright <newshound2@digitalmars.com>:
>
>> On 1/31/2015 9:21 PM, Mike wrote:
>> > Is D's core team genuinely interested in this domain?
>>
>> Yes.
>>
>>
>> > If you are genuinely interested, are you committed?  And if so, what direction would you like to take?  So far, my ideas have been very unpopular and I'm growing weary fighting the current. How can I contribute in a way that gets us both where we want to go?
>>
>> I don't recall what you've suggested in this vein that was very unpopular - can you please post an example?
>
> Mike once complained that the runtime heavily relies on some high-level OS features not available on embedded devices (e.g. threads). That's OK if you actually have an OS, but druntime should be modular enough to run on systems without thread support.
>

The core library part needs some love in this respect.  But I feel at least the rt and gcc library in GDC allow easier transition.

For instance, I'm still happy at how gcc.gthreads is handled and how
you can control the thread support at configure time - despite being
only used in rt.monitor_ and rt.criticial_.
https://github.com/D-Programming-GDC/GDC/blob/61682dd6adada34bc5bbff5903954a6af45cca92/libphobos/configure.ac#L342
https://github.com/D-Programming-GDC/GDC/tree/master/libphobos/libdruntime/gcc/gthreads

Though it's clearly not enough, and I think there should be a push for
more configurations here:
https://github.com/ibuclaw/GDC/blob/9e03425dd5b3229d9333837631d19ad2d5c9b150/libphobos/libdruntime/gcc/config.d.in

Iain

On 2/1/15 1:38 AM, Timo Sintonen wrote:
> The one of major issues is: how to access hardware. We need a language
> feature to access hardware registers.

What features do Rust and Nim provide for such?

> - NoSystem should be a supported platform and not a build failure. The
> build system should leave out files and features that are not available.

I really like this notion that not defining a system (or defining a special one) builds a minimal freestanding implementation.


Andrei

On 2/1/15 3:28 AM, Johannes Pfau wrote:
> IIRC he also proposed moving more of TypeInfo implementation to the
> runtime so TypeInfo layout can be modified by the runtime or even
> completely avoided by not implementing it in the runtime. The main
> response was that it complicates compiler code too much for little
> benefit.

Interesting. I naïvely believe it would simplify rather than complicate the compiler. -- Andrei

On Sunday, 1 February 2015 at 15:28:25 UTC, Andrei Alexandrescu wrote:
> On 2/1/15 1:38 AM, Timo Sintonen wrote:
>> The one of major issues is: how to access hardware. We need a language
>> feature to access hardware registers.
>
> What features do Rust and Nim provide for such?
>
> Andrei

I was not the one who compared the languages and I have never used rust or nim.  I just pointed out that d had no way to access registers properly. DMD and GDC optimize very heavily and may cache, reorder and remove any access they think is not needed.
Other languages may not have such optimizations and registers can be accessed just with normal memory oprations.