Hi,

What's the current situation of using D without Phobos/Druntime?

Sometimes, it's quite useful to be able to use D to create tiny EXEs/DLLs. For example, create simple DLL plugins that will be loaded into non-D programs, or even not have a 50-line program take half a meg. Additionally, some tasks require injecting a DLL systemwide (into every running process), so it is critical to minimize the impact by being as unintrusive as possible and having a minimal overhead.

I've started a "framework" for this purpose a while ago[1], but I heard of people starting similar projects since then and I think they've gotten further than me. Would there be any interest in collaborating and turning this into a community project?

I think some goals would be:
- A "runtime" (object.d) which provides either stubs for the current compiler hooks, or implementations with minimal dependencies
- A "standard library" which does not require runtime support, but can also be used when using Druntime (as a GC-free alternative to Phobos)
- Using MSVCRT on 32-bit as well
- Options to enable/disable various runtime features? For example, a project may want to use exceptions (which requires ClassInfo) but not the GC.
- Support for the Unilink linker? I think it is better at generating smaller binaries that OPTLINK.

There is also some ongoing work on the -betterC switch[2], which I'm quite excited about, but it seems to be currently blocked on waiting for the next DMD release as the changes[3] might break a lot of code.

 [1]: https://github.com/CyberShadow/SlimD
 [2]: https://d.puremagic.com/issues/show_bug.cgi?id=11881
 [3]: https://github.com/D-Programming-Language/dmd/pull/2561

On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev wrote:
> Hi,
>
> What's the current situation of using D without Phobos/Druntime?

I created a minimal, no runtime, "hello world" program [1], and I think Iain Buclaw answered your question when he commented on my example as follows:

"that's not D! That's some D, a bit of some extended inline assembly, and a custom linker script."

To paraphrase, if you really want the best of what D has to offer, you'll need to implement a good part of the runtime.  But I argue, not all of it.

I'm working on a bare-metal ARM Cortex-M _very minimal_ port of the D runtime with an emphasis on "minimal". I just created my repository and started documenting my progress [2].  It's in the very early stages, and I am quite new to D, so I have quite a ways to go, but my goals align with your stated intent.

I'm able to do structs with just a few line of stubbed TypeInfo stuff in my object.d.  I've reduced it to 21 lines [3].

I've also implemented a simple malloc/free and am able to do classes, but I haven't tested any D feature thoroughly yet.

>
> Sometimes, it's quite useful to be able to use D to create tiny EXEs/DLLs. For example, create simple DLL plugins that will be loaded into non-D programs, or even not have a 50-line program take half a meg. Additionally, some tasks require injecting a DLL systemwide (into every running process), so it is critical to minimize the impact by being as unintrusive as possible and having a minimal overhead.
>
> I've started a "framework" for this purpose a while ago[1], but I heard of people starting similar projects since then and I think they've gotten further than me. Would there be any interest in collaborating and turning this into a community project?

Didn't know about SlimD (catchy name by the way).  I'll definitely be taking a look as this aligns with my goals.

As the Cortex-M microcontrollers have very little FLASH memory, it is also important for me to reduce the executable size, and I'm focusing on that in my project.  I've been able to get my "Hello, World!" program down to 68 bytes, with 15 of those bytes being the "Hello, World!\r\n" string constant.

>
> I think some goals would be:
> - A "runtime" (object.d) which provides either stubs for the current compiler hooks, or implementations with minimal dependencies

I would really like to see a more modular D runtime in which features of the D language can be selectively chosen.  This is one of my goals.  I'd also like to have the D runtime "reorganzied" to make hardware/OS/C-library abstraction more obvious.  I submitted an enhancement to facilitate this [4]:

I thin, a modular, well-organized, well-abstracted, D runtime will facilitate both your stated intention and my goals to bring D to the microcontroller world, and will also facilitate bringing D to a variety of other platforms as well.  But, I really think the D runtime needs some refactoring to achieve this.

In my project, I'm also trying to do everything in D ("Look, Walter!,  No C!").  I'd like to see all the references to C's baggage left separate from the rest of the D runtime, included "size_t".  Let's not track C's doo-doo into D's house, if it can be avoided.

> - A "standard library" which does not require runtime support, but can also be used when using Druntime (as a GC-free alternative to Phobos)

Phobos will not be useful to me unless I can find some way to make it lighter and GC-Free.  I'll probably just end up taking pieces of it.  I would _really_ like to have the whole concept of a GC decoupled from D.  There are many suitable memory models to use with D, and the memory model seems like it should be a platform feature, not a language feature.  But, I don't know how one could accomplish that without some changes to the language itself, and it seems the core language folks have their hands full.

> - Options to enable/disable various runtime features? For example, a project may want to use exceptions (which requires ClassInfo) but not the GC.

Yes, this definitely aligns with my goals, see my comment above about a modular D runtime.  But I wouldn't necessarily be in favor of adding switches to the compiler, as I don't like to be at the mercy of the compiler implementers.  I'd prefer some way where I can pass a user-defined variable to my compiler flags in concert with the version() feature.  And if something is needed, but not implemented, the compiler/linker will throw an error saying so.

> - Support for the Unilink linker? I think it is better at generating smaller binaries that OPTLINK.
>

Does D have to support Unilink, or does Unilink have to support D?

> There is also some ongoing work on the -betterC switch[2], which I'm quite excited about, but it seems to be currently blocked on waiting for the next DMD release as the changes[3] might break a lot of code.
>
>  [1]: https://github.com/CyberShadow/SlimD
>  [2]: https://d.puremagic.com/issues/show_bug.cgi?id=11881
>  [3]: https://github.com/D-Programming-Language/dmd/pull/2561

There was an interesting discussion, recently, that might relevant to this.  I encourage you to check it out and maybe post your thoughts [5]. It's more about the required TypeInfo stuff, but nevertheless related to the -betterC idea.

LCD has some pragmas that seem to align with the general idea of what you're looking for [6], and GDC has the -fno-emit-moduleinfo which is essential for what I'm doing.

I'd love to contribute to something like this in anyway I can, but I'd probably be asking more questions than providing help.  At the moment, my contribution is in the form of my repository [2].  I hope it will have some influence in the long run, and maybe when I come up to speed on the language I'll be able to contribute more.

Mike

[1] http://wiki.dlang.org/Minimal_semihosted_ARM_Cortex-M_%22Hello_World%22
[2] https://github.com/JinShil/D_Runtime_ARM_Cortex-M_study
[3] https://github.com/JinShil/D_Runtime_ARM_Cortex-M_study/blob/structs/source/object.d
[4] https://d.puremagic.com/issues/show_bug.cgi?id=11666
[5] http://forum.dlang.org/post/jynxfglpulguvqbivrms@forum.dlang.org
[6] http://wiki.dlang.org/LDC-specific_language_changes

On Wednesday, 22 January 2014 at 04:13:41 UTC, Mike wrote:
> Didn't know about SlimD (catchy name by the way).
Would be even better if D2 had been named "E" :-D

On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev wrote:
> What's the current situation of using D without Phobos/Druntime?

With LDC I had to figure out how to make the compiler happy, but eventually got the job done.

> Sometimes, it's quite useful to be able to use D to create tiny EXEs/DLLs. For example, create simple DLL plugins that will be loaded into non-D programs, or even not have a 50-line program take half a meg. Additionally, some tasks require injecting a DLL systemwide (into every running process), so it is critical to minimize the impact by being as unintrusive as possible and having a minimal overhead.
>
> I've started a "framework" for this purpose a while ago[1], but I heard of people starting similar projects since then and I think they've gotten further than me. Would there be any interest in collaborating and turning this into a community project?

A framework doesn't seem like a good idea. You can have either big framework or small executable, not both.

My code works with stock object.d just fine. In order to make injectable dll you only have to disable GC (and probably tls), and write otherwise normal D code.

On Wednesday, 22 January 2014 at 09:37:00 UTC, Dominikus Dittes Scherkl wrote:
> On Wednesday, 22 January 2014 at 04:13:41 UTC, Mike wrote:
>> Didn't know about SlimD (catchy name by the way).
> Would be even better if D2 had been named "E" :-D

I still think that the original name "Mars" was far better, never mind the fact that it would've been far easier to Google. Ah well.

Atila

Hi Mike, thanks for the extensive reply.

On Wednesday, 22 January 2014 at 04:13:41 UTC, Mike wrote:
> To paraphrase, if you really want the best of what D has to offer, you'll need to implement a good part of the runtime.  But I argue, not all of it.

Well, even C with D's metaprogramming would be a big win. But picking-and-choosing runtime-supported features would be even better.

> I'm working on a bare-metal ARM Cortex-M _very minimal_ port of the D runtime with an emphasis on "minimal". I just created my repository and started documenting my progress [2].  It's in the very early stages, and I am quite new to D, so I have quite a ways to go, but my goals align with your stated intent.
>
> I'm able to do structs with just a few line of stubbed TypeInfo stuff in my object.d.  I've reduced it to 21 lines [3].
>
> I've also implemented a simple malloc/free and am able to do classes, but I haven't tested any D feature thoroughly yet.
>
> As the Cortex-M microcontrollers have very little FLASH memory, it is also important for me to reduce the executable size, and I'm focusing on that in my project.  I've been able to get my "Hello, World!" program down to 68 bytes, with 15 of those bytes being the "Hello, World!\r\n" string constant.

This is quite cool.

I noticed you're using GNU make. On Windows, D uses Digital Mars make, which is quite minimalistic. I've considered writing a custom build tool for SlimD, but now that I think about it, I think GNU make will suffice.

> I would really like to see a more modular D runtime in which features of the D language can be selectively chosen.  This is one of my goals.
>
> I thin, a modular, well-organized, well-abstracted, D runtime will facilitate both your stated intention and my goals to bring D to the microcontroller world, and will also facilitate bringing D to a variety of other platforms as well.  But, I really think the D runtime needs some refactoring to achieve this.

So, you think it would be better to make the standard D runtime more customizable in this regard, instead of putting together a new one?

I believe the biggest factor for why someone would want to use a lighter runtime is the garbage collector. I think a lot of the code in Druntime still assumes that a GC is present... and then I'm not sure the D maintainers would want to maintain support for such rarely-needed features such as a custom entry point. (Case in point: pragma(entrypoint, symbol) has been broken since DMD 2.061.)

> In my project, I'm also trying to do everything in D ("Look, Walter!,  No C!").  I'd like to see all the references to C's baggage left separate from the rest of the D runtime, included "size_t".

What's wrong with size_t?

On Windows, it is quite necessary if you want to target both 32-bit and 64-bit (and if you want to do system-wide DLL injection, that's an absolute requirement).

> Phobos will not be useful to me unless I can find some way to make it lighter and GC-Free.  I'll probably just end up taking pieces of it.  I would _really_ like to have the whole concept of a GC decoupled from D.  There are many suitable memory models to use with D, and the memory model seems like it should be a platform feature, not a language feature.  But, I don't know how one could accomplish that without some changes to the language itself, and it seems the core language folks have their hands full.

There have been discussions of a compiler switch or @attribute to make implicit allocations (such as array literals, or heap closures) generate compiler errors.

> Yes, this definitely aligns with my goals, see my comment above about a modular D runtime.  But I wouldn't necessarily be in favor of adding switches to the compiler, as I don't like to be at the mercy of the compiler implementers.

I agree that hacking or forking the compiler should be avoided and left as a last resort. By "switches" I meant per-project (e.g. makefile) options.

> Does D have to support Unilink, or does Unilink have to support D?

Well, neither. The "framework" would need to support linking your project with Unilink if you indicate that preference in your project's build configuration.

> There was an interesting discussion, recently, that might relevant to this.  I encourage you to check it out and maybe post your thoughts [5]. It's more about the required TypeInfo stuff, but nevertheless related to the -betterC idea.

Well, as long as the linker throws away all compiler-emitted TypeInfo objects if your code doesn't use them, there doesn't seem to be any practical problem with them, is there?

> LCD has some pragmas that seem to align with the general idea of what you're looking for [6], and GDC has the -fno-emit-moduleinfo which is essential for what I'm doing.

ModuleInfo IS a problem, though. It is like a parasite of interdependence: it points towards all class information (for Object.factory), which itself pulls in all virtual methods of all classes in your program (through the Vtable). That means that anything that any virtual method in your program uses, regardless if you use that class or not, WILL end up in the executable. Ugh!

> I'd love to contribute to something like this in anyway I can, but I'd probably be asking more questions than providing help.  At the moment, my contribution is in the form of my repository [2].  I hope it will have some influence in the long run, and maybe when I come up to speed on the language I'll be able to contribute more.

Maybe we could join efforts, even though our goals target different platforms :)

What do you say to this repository structure (inspired by the Windows DDK):

Users start a new SlimD (or whatever name) project by adding a Makefile to their project directory. The makefile configures the project's requirements (target platform, used D features, linker of choice) and source files (*.d by default), and includes the main SlimD makefile, which does all the work based on the given configuration.

On Wednesday, 22 January 2014 at 11:13:53 UTC, Kagamin wrote:
> On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev wrote:
>> I've started a "framework" for this purpose a while ago[1], [...]
>
> A framework doesn't seem like a good idea. You can have either big framework or small executable, not both.

I don't think you and I mean the same things ;) That's why I put "framework" in quotes. It was the most suitable word that came to mind - I basically meant "standard library and build system for your program". My goal is definitely not to replace one kitchen-sink library with another.

> My code works with stock object.d just fine. In order to make injectable dll you only have to disable GC (and probably tls), and write otherwise normal D code.

"Normal D code" involves plenty of explicit and implicit allocations. The practical and aesthetic considerations of hundreds of KB of useless baggage remain.

On Thursday, 23 January 2014 at 22:39:38 UTC, Vladimir Panteleev wrote:
> Well, even C with D's metaprogramming would be a big win. But picking-and-choosing runtime-supported features would be even better.

Quite right!  I have actually delayed implementing classes a little bit because I found I could do so much with just structs, mixins, and templates.  In fact, if there weren't so many types in the runtime implemented as classes, I could delay it even further.

> I noticed you're using GNU make. On Windows, D uses Digital Mars make, which is quite minimalistic. I've considered writing a custom build tool for SlimD, but now that I think about it, I think GNU make will suffice.

Using GNU make was really an arbitrary decision for me.  I really don't like GNU make and hate my own makefiles even more.  Personally, I would prefer to write scripts in D, so I could automate whatever I wanted including builds.

My project, however, is just an exercise.  I expect to go through this exercise a couple times and fine tune what I want.  Then I'll make a real repository with just the good stuff :-)

> So, you think it would be better to make the standard D runtime more customizable in this regard, instead of putting together a new one?

At the moment, I think throwing it all away and starting from scratch is the best approach, especially with the "minimal/modular" mindset, but I would like to get to a point where the two could be integrated (either I join D, or D joins me).  I would prefer not to fragment D into "D", "D for embedded", "D for smartphones", "D for tablets", etc..  I think if the D runtime were modular and designed with platform ubiquity in mind, it would simply require a customized makefile like you've already alluded to, or something other approach (see my comments below)

> What's wrong with size_t?

There's nothing wrong with the concept of size_t.  The problem is the "_t" suffix.  That is C's baggage, and is inconsistent with other types in D.  D doesn't use string_t, uint_t, etc..., so I don't see why this convention is being employed in D.


>> There was an interesting discussion, recently, that might relevant to this.  I encourage you to check it out and maybe post your thoughts [5]. It's more about the required TypeInfo stuff, but nevertheless related to the -betterC idea.
>
> Well, as long as the linker throws away all compiler-emitted TypeInfo objects if your code doesn't use them, there doesn't seem to be any practical problem with them, is there?

Yeah, that's the conclusion I eventually came to.  It was just a major inconvenience for me to have to spend hours studying and implementing all of this TypeInfo stuff, and all the stuff the TypeInfo uses, only to have it thrown away at link time.  It was a huge waste of effort, but all that TypeInfo stuff may be needed in the end anyway, so I'm holding back too much harsh judgement at the moment.

> ModuleInfo IS a problem, though. It is like a parasite of interdependence: it points towards all class information (for Object.factory), which itself pulls in all virtual methods of all classes in your program (through the Vtable). That means that anything that any virtual method in your program uses, regardless if you use that class or not, WILL end up in the executable. Ugh!

That sounds very undesirable.  I still don't even understand what purpose modules and ModuleInfo really serve.  Right now, I'm just using modules for namespace scope and encapsulation.  If you know some documentation that helps demystify ModuleInfo and what its purpose is (besides the source code) please point me to it.

> Maybe we could join efforts, even though our goals target different platforms :)
>
> What do you say to this repository structure (inspired by the Windows DDK):
>
> Users start a new SlimD (or whatever name) project by adding a Makefile to their project directory. The makefile configures the project's requirements (target platform, used D features, linker of choice) and source files (*.d by default), and includes the main SlimD makefile, which does all the work based on the given configuration.

This is very much like what I had in mind, and I like it.  However, I'm considering other methods, too.

1) Features/platforms can simply be provide at link time by adding a certain *.a or *.o files to the linker command.  And if a needed feature is missing, linker errors will result.

For example, exception handling could be provide with seh.o, eh.o, sjlj.o, or no_eh.o depending on which model the user wanted to use. (This example may not make any sense, as I know nothing about how to implement exception handling.  I just couldn't think of better illustration of my thoughts).

2) Templates can be used to provide modularity/specialization in one's source code at compile time.  There's a C++ graphics library [1] that uses this approach to create platform independent rendering pipelines.  For example:

//pseudo code
typedef color_t
typedef pixel_t
typedef path_t
typedef surface<pixel_t> surface_t
typedef rasterizer<path_t> rasterizer_t
typedef renderer<rasterizer_t, surface_t> renderer_t.

//Then draw like this
renderer_t r;
r.DrawLine()
r.DrawCircle()
...

So, surface_t could be defined as...

typedef surface<DirectX11> surface_t
typedef surface<OpenGL> surface_t
typedef surface<X11> surface_t

... to provide some platform specialization at compile time, and the user of the code doesn't know the difference.

This is a pretty cool way to build a graphics library, and something along these lines could be used to provide some specialization to the D runtime too.  User's would simply have to include a certain *.di file containing the required templated aliases/typedefs for a given platform.  If user's want to port to a different platform, or choose a different feature subset, they just create a different set of templated aliases/typedefs.

I'm brainstorming as I write this, so sorry for being so verbose.  As you can see I don't have any opinion yet.  Perhaps you could provide your opinion.

Nevertheless, your proposed solution does basically what I'm looking for, and would be happy to help in any way I can.

Mike

[1] - http://www.antigrain.com/

On 2014-01-24 06:17, Mike wrote:

> That sounds very undesirable.  I still don't even understand what
> purpose modules and ModuleInfo really serve.  Right now, I'm just using
> modules for namespace scope and encapsulation.  If you know some
> documentation that helps demystify ModuleInfo and what its purpose is
> (besides the source code) please point me to it.

ModuleInfo contains, for example, module constructors:

module foo;

static this () { // run once before every thread is started }
shared static this () { // run once before main is called }

It also provides a way to iterate all modules, at runtime. From each module it's possible to iterate all local classes and all imported modules. Object.create which will return a new object of a class given by its fully qualified name, as a string. Object.create is implemented using ModuleInfo.

-- 
/Jacob Carlborg

On Thursday, 23 January 2014 at 22:39:38 UTC, Vladimir Panteleev wrote:
> I believe the biggest factor for why someone would want to use a lighter runtime is the garbage collector. I think a lot of the code in Druntime still assumes that a GC is present... and then I'm not sure the D maintainers would want to maintain support for such rarely-needed features such as a custom entry point. (Case in point: pragma(entrypoint, symbol) has been broken since DMD 2.061.)

The GC seems to be the biggest issue here, so I wanted to comment separately.

Each target/platform/application will have different requirements and those requirements will dictate the type of memory management implementation to use.

Ideally, I don't see why D has to be a garbage collected language.  Can't it be agnostic to the memory management implementation?

D seems to offer several options to the programmer, Scoped, RefCounted, new/destroy, and can even create their own garbage collection with the gcstub (or so I've heard).  This is great!, but it's not idea as it forces the programmer to change their idioms.  It appears that D is very tightly coupled to the memory management implementation, but does it need to be?

Couldn't the runtime provide some hooks that can be used by the runtime implementation to know when a class is instantiated, when it goes out of scope, when it's assigned to another reference, etc... and use this to provide a variety of different memory management implementations without the programmer having to change their idioms?  I'm hypothesizing that if these runtime hooks existed, one could create a transparent reference-counted implementation that would be mostly transparent to the programmer, for example. I still think the programmer will have to write code with their memory management implementation in mind, but they could at least stick to the same idioms.

I'm not very optimistic, but I'm hoping something like this can be achieved, and that's what I'm shooting for.  Absent this possibility, I'm not sure what to do.

I believe there is probably a GC implementation that would be suitable for a my target platform (ARM Cortex-M microcontrollers).  I can think of at least one:  Some firmware for microcontrollers simply need to allocate all of their objects at the start of the program, but have no need to free memory: free is the power switch.  For applications like this only gc.allocate() needs to be implemented, and the GC's background thread and gc.free() do not.  I think this would allow the programmer to stick with traditional D idioms, but they have to keep in mind gc.free() will never be and should never be called.  Some compile-time guarantee here would be nice.

I could go on and on about some of my ideas, but I think I've said enough for now.  I've got a lot to learn first, anyway.

Mike