In an attempt to put some impetus behind an idea that I've proposed multiple times on the forum, I've resurrected my utiliD repository:  https://github.com/JinShil/utiliD  I decided to resurrect the repository after this brief discussion:  https://forum.dlang.org/post/bjycwgrifumsfrhprjho@forum.dlang.org

The idea behind the library is that it would not depend on druntime, phobos, C standard library, or anything else but would still offer many of the features that those libraries provide.  To utilize the library, one would only need a D compiler. It could be used in bare-metal programming, -betterC builds, or as a fundamental utility library for implementing DMD, druntime, and phobos themselves.

It's what I envision as a potential seed for Andrei's opt-in continuum (https://forum.dlang.org/post/q7j4sl$17pe$1@digitalmars.com).

I don't know what will ultimately happen with this library, if anything, but even if all it does is facilitate brainstorming of ideas, or as the genesis of some better idea, it will be a success.

I don't have much time to work on it right now, as I'm currently preoccupied fixing the compiler and the druntime to make something like utiliD possible, but if others grok the idea and want to help make it a reality, your help is most welcome.

Also, if any of you have already started something with the same goals, I'll be happy to drop this repository and join you.

You can find me on Slack and Discord using the handle JinShil if you wish to have a dialog about this.

Mike

On Sunday, 5 May 2019 at 03:45:41 UTC, Mike Franklin wrote:
> In an attempt to put some impetus behind an idea that I've proposed multiple times on the forum, I've resurrected my utiliD repository:  https://github.com/JinShil/utiliD  I decided to resurrect the repository after this brief discussion:  https://forum.dlang.org/post/bjycwgrifumsfrhprjho@forum.dlang.org
>
> The idea behind the library is that it would not depend on druntime, phobos, C standard library, or anything else but would still offer many of the features that those libraries provide.  To utilize the library, one would only need a D compiler. It could be used in bare-metal programming, -betterC builds, or as a fundamental utility library for implementing DMD, druntime, and phobos themselves.
>
> It's what I envision as a potential seed for Andrei's opt-in continuum (https://forum.dlang.org/post/q7j4sl$17pe$1@digitalmars.com).
>
> I don't know what will ultimately happen with this library, if anything, but even if all it does is facilitate brainstorming of ideas, or as the genesis of some better idea, it will be a success.
>
> I don't have much time to work on it right now, as I'm currently preoccupied fixing the compiler and the druntime to make something like utiliD possible, but if others grok the idea and want to help make it a reality, your help is most welcome.
>
> Also, if any of you have already started something with the same goals, I'll be happy to drop this repository and join you.
>
> You can find me on Slack and Discord using the handle JinShil if you wish to have a dialog about this.
>
> Mike

Hi Mike,

you may remember that I'm working on a library named "tanya" (https://github.com/caraus-ecms/tanya). It is now almost phobos-free and I reimplemented some routines from libc for x86-64 linux. Ideally I'd like to get rid of libc for some platforms. While the library isn't interesting for you since it's too high-level, it could be based on something like utilD. So, as for me, I'd be very much interested in collective effort in this direction and can contribute.

On 2019-05-05 05:45, Mike Franklin wrote:

> The idea behind the library is that it would not depend on druntime, phobos, C standard library, or anything else but would still offer many of the features that those libraries provide. To utilize the library, one would only need a D compiler. It could be used in bare-metal programming, -betterC builds, or as a fundamental utility library for implementing DMD, druntime, and phobos themselves.

Might be interesting to write a tool that enforces the rules. It would use DMD as a library.

-- 
/Jacob Carlborg

On Sunday, 5 May 2019 at 05:23:26 UTC, Eugene Wissner wrote:

> you may remember that I'm working on a library named "tanya" (https://github.com/caraus-ecms/tanya). It is now almost phobos-free and I reimplemented some routines from libc for x86-64 linux. Ideally I'd like to get rid of libc for some platforms. While the library isn't interesting for you since it's too high-level, it could be based on something like utilD. So, as for me, I'd be very much interested in collective effort in this direction and can contribute.

Excellent!  Yes, I remember seeing tanya.  As you can tell I have very few details worked out with regard to utiliD.  You obviously have some more experience creating such a library.

I see that https://github.com/caraus-ecms/tanya/tree/master/arch/x64/linux/memory has what appears to be something equivalent to memcpy, memcmp, and memset.  I am very interested in having D implementations of those (inline assembly is D) but I also want to explore the idea keeping things strongly-typed (as least as long as possible) and utilize design-by-introspection to branch the implementation.

I'd be interested in hearing more about what you have in mind.

Thanks,
Mike

- Memcmp, memcpy, memmove and memset are named equal, copy, copyBackward and fill respectively. I just wanted to create native implementations that are bit safer than their C counterparts. So they do the same job, but accept void[] instead of pointers. There are also templated functions with the same names, that work with ranges.

I‘m not very comfortable with GCC‘s inline asm and it doesn‘t support naked asm as DMD does, so I put asm in the .S files and compile them separately. But I‘m fine with inline asm too. A problem with the inline asm is that it should be written in several versions since DMD uses different calling conventions (unless we use extern(C)) and GDC and LDC use different asm syntax.

Tanya contains pretty much stuff now and I‘m just thinking to split it in a smaller parts (of a reasonable size), that are probably interesting for other people, who is ready to contribute, so I don‘t have to maintain everything myself. I don‘t know exactly what goes into this more „low-level“ library, we can always talk about it.

- OS API

Not sure if it belongs to the scope of utilD. Some time ago it became clear to me, that while C has functions for dynamic memory management, it uses them internally very seldom. Instead it lets the user to allocate the memory. So there functions like:

char *if_indextoname(unsigned int ifindex, char *ifname);

that take an output buffer as the last argument. The same can be done with output ranges in D, so these system functions can be rewritten in D with a better interface. Whereby I should say that tanya‘s range definitions differ from Phobos.

- meta

Another thing probably interesting for utilD library is meta-programming. Tanya has „tanya.meta“ package which contains templates similar to to std.traits and std.meta + some nice extras like Union/Intersection/Difference working on sets of types, that are inspired by Boost Hana. This part is completely independent (from Phobos and the rest of tanya) and can even be a separate library.

On Friday, 10 May 2019 at 05:20:59 UTC, Eugene Wissner wrote:
> - Memcmp, memcpy, memmove and memset are named equal, copy, copyBackward and fill respectively. I just wanted to create native implementations that are bit safer than their C counterparts. So they do the same job, but accept void[] instead of pointers. There are also templated functions with the same names, that work with ranges.
>
> I‘m not very comfortable with GCC‘s inline asm and it doesn‘t support naked asm as DMD does, so I put asm in the .S files and compile them separately. But I‘m fine with inline asm too. A problem with the inline asm is that it should be written in several versions since DMD uses different calling conventions (unless we use extern(C)) and GDC and LDC use different asm syntax.

Yeah, that is indeed unfortunate, and something I'll have to consider. I have had to write 3 different inline-asm implementations for some of my exporations, and didn't find it to be too bad.  I very much prefer to read the D with the inline-asm than a straight assembly file.  I've studied the ARM implementation of memcpy a little, and it's quite hard to follow.  I'd like for the D implementations to make such code easier to understand and maintain.

> Tanya contains pretty much stuff now and I‘m just thinking to split it in a smaller parts (of a reasonable size), that are probably interesting for other people, who is ready to contribute, so I don‘t have to maintain everything myself. I don‘t know exactly what goes into this more „low-level“ library, we can always talk about it.

Yes, I'm still working that out too.  If you apply the rule that it should not require anything from druntime, the C standard library, or dynamic memory allocation, it does eliminate quite a bit, and narrows the scope.  What I'm trying to do now is just focus on the obvious, and hopefully with that out of the way the rest will begin to reveal themselves.

> - OS API
>
> Not sure if it belongs to the scope of utilD. Some time ago it became clear to me, that while C has functions for dynamic memory management, it uses them internally very seldom. Instead it lets the user to allocate the memory. So there functions like:
>
> char *if_indextoname(unsigned int ifindex, char *ifname);
>
> that take an output buffer as the last argument. The same can be done with output ranges in D, so these system functions can be rewritten in D with a better interface. Whereby I should say that tanya‘s range definitions differ from Phobos.

Yes, I like that.  The buffer and memory management is then delegated outside of the library.  That, IMO, makes the library more broadly useful.  Fundamental algorithms (e.g. from std.algorithm, std.range, etc.) that can operate on memory buffers/ranges in this way would be good candidates for utiliD, IMO.  But I'd want them to meet the criteria of being fundamental and broadly useful; not too specialized.  Specialized algorithms that are designed for specific problem domains should probably go in a library designed for that problem domain.

> - meta
>
> Another thing probably interesting for utilD library is meta-programming. Tanya has „tanya.meta“ package which contains templates similar to to std.traits and std.meta + some nice extras like Union/Intersection/Difference working on sets of types, that are inspired by Boost Hana. This part is completely independent (from Phobos and the rest of tanya) and can even be a separate library.

I'm thinking metaprogramming modules and packages are good candidates for utilitD as long as they are broadly useful.  I see them more as extensions of the language than a library.  Though, in a way, that's basically what libraries are too.  I'll have to think about this some more but at the moment I'm leaning towards inclusion in utiliD.

Mike

On Friday, 10 May 2019 at 17:16:24 UTC, Mike Franklin wrote:
> On Friday, 10 May 2019 at 05:20:59 UTC, Eugene Wissner wrote:
>> - Memcmp, memcpy, memmove and memset are named equal, copy, copyBackward and fill respectively. I just wanted to create native implementations that are bit safer than their C counterparts. So they do the same job, but accept void[] instead of pointers. There are also templated functions with the same names, that work with ranges.
>>
>> I‘m not very comfortable with GCC‘s inline asm and it doesn‘t support naked asm as DMD does, so I put asm in the .S files and compile them separately. But I‘m fine with inline asm too.

Why would you use inline assembly ? (generalizing but: extremely bad portability, bad performance, bad readability)

Recent discussion on LLVM mailinglist about the problem of the optimizer recognizing memcmp implementation and substituting it with a call to memcmp: https://lists.llvm.org/pipermail/llvm-dev/2019-April/131973.html

cheers,
  Johan

On Fri, May 10, 2019 at 05:16:24PM +0000, Mike Franklin via Digitalmars-d-announce wrote: [...]
> I've studied the ARM implementation of memcpy a little, and it's quite hard to follow.  I'd like for the D implementations to make such code easier to understand and maintain.
[...]

I'm not 100% sure it's a good idea to implement memcpy in D just to prove that it can be done / just to say that we're independent of libc. Libc implementations of fundamental operations, esp. memcpy, are usually optimized to next week and back for the target architecture, taking advantage of the target arch's quirks to maximize performance. Not to mention that advanced compiler backends recognize calls to memcpy and can optimize it in ways they can't optimize a generic D function they fail to recognize as being equivalent to memcpy. I highly doubt a generic D implementation could hope to beat that, and it's a little unrealistic, given our current manpower situation, for us to be able to optimize it for each target arch ourselves.


> On Friday, 10 May 2019 at 05:20:59 UTC, Eugene Wissner wrote:
[...]
> > Whereby I should say that tanya‘s range definitions differ from Phobos.
[..]

I'm a bit uncomfortable with having multiple, incompatible range definitions.  While the Phobos definition can be argued whether it's the best, shouldn't we instead be focusing on improving the *standard* definition of ranges, rather than balkanizing the situation by introducing multiple, incompatible definitions just because?  It's one thing for Andrei to propose a std.v2 that, ostensibly, might have a new, hopefully improved, range API, deprecating the current definition; it's another thing to have multiple alternative, competing definitions in libraries that user code can choose from.  That would be essentially inviting the Lisp Curse.


T

-- 
Life would be easier if I had the source code. -- YHL

On Friday, 10 May 2019 at 17:55:53 UTC, Johan Engelen wrote:

> Why would you use inline assembly ? (generalizing but: extremely bad portability, bad performance, bad readability)

The only reason to use inline assembly is to achieve something that can't be achieved directly with D.  For example, prior to the introduction of `volatileLoad` and `volatileStore` inline assembly was required to achieve `volatile` semantics.

For memcpy and memcmp, one would first attempt to write a good implementation in straight D, but if the compiler doesn't generate good code for it, it would be appropriate to take control and provide an implementation in inline assembly.

I don't know how a proper assembly implementation would not be performant.  Perhaps you could elaborate.

> Recent discussion on LLVM mailinglist about the problem of the optimizer recognizing memcmp implementation and substituting it with a call to memcmp: https://lists.llvm.org/pipermail/llvm-dev/2019-April/131973.html

Yes, that bit me a while back when I was doing some bare-metal ARM Cortex-m development https://forum.dlang.org/post/clptjumxigcozfcyhzzx@forum.dlang.org

For compilers that already provide an optimized intrinsic implementation for memcpy none of this is necessary; one could simply add a naive implementation, the compiler would recognize it, and replace it with their optimized version.  DMD, to my understanding, is not one of those compilers.

One of the goals is to no longer require a C toolchain to build D programs.  If the compiler already provides intrinsics without needed a C standard library, Great!

The other goal is to explore what D could improve upon with its design-by-introspection features and compiler guarantees (e.g. `pure` and `@safe`).  My initial exploration into that can be found at https://github.com/JinShil/memcpyD  I find it much easier to read D code like that than something like this:  https://github.com/opalmirror/glibc/blob/c38d0272b7c621078d84593c191e5ee656dbf27c/sysdeps/arm/memcpy.S

Mike

On Friday, 10 May 2019 at 23:51:56 UTC, H. S. Teoh wrote:

> I'm not 100% sure it's a good idea to implement memcpy in D just to prove that it can be done / just to say that we're independent of libc. Libc implementations of fundamental operations, esp. memcpy, are usually optimized to next week and back for the target architecture, taking advantage of the target arch's quirks to maximize performance. Not to mention that advanced compiler backends recognize calls to memcpy and can optimize it in ways they can't optimize a generic D function they fail to recognize as being equivalent to memcpy. I highly doubt a generic D implementation could hope to beat that, and it's a little unrealistic, given our current manpower situation, for us to be able to optimize it for each target arch ourselves.

I understand that point of view.  Indeed we have to demonstrate benefit.  One benefit is to not have to obtain a C toolchain when building D programs.  That is actually quite an inconvenient barrier to entry when cross-compiling (e.g. for developing microcontroller firmware on a PC).

I'm also hoping that a D implementation would be easier to comprehend than something like this:  https://github.com/opalmirror/glibc/blob/c38d0272b7c621078d84593c191e5ee656dbf27c/sysdeps/arm/memcpy.S  The D implementation still has to handle all of those corner cases, but I'd rather read D code with inline assembly sprinkled here and there than read the entire thing in assembly.  The goal with the D implementation would be to minimize the assembly.

For compilers that already do something special with memcpy and don't require a C standard library, there's no reason to do anything.  My initial exploration into this has shown that DMD is not one of those compilers.

>> On Friday, 10 May 2019 at 05:20:59 UTC, Eugene Wissner wrote:
> [...]
>> > Whereby I should say that tanya‘s range definitions differ from Phobos.
> [..]
>
> I'm a bit uncomfortable with having multiple, incompatible range definitions.  While the Phobos definition can be argued whether it's the best, shouldn't we instead be focusing on improving the *standard* definition of ranges, rather than balkanizing the situation by introducing multiple, incompatible definitions just because?  It's one thing for Andrei to propose a std.v2 that, ostensibly, might have a new, hopefully improved, range API, deprecating the current definition; it's another thing to have multiple alternative, competing definitions in libraries that user code can choose from.  That would be essentially inviting the Lisp Curse.

Agreed.  We should decide on one consistent definition.  I don't know what that looks like right now.  I'm more focused on low-level details right now.  I do, however, like the idea of delegating the memory management (allocation/deallocation) outside of the library.  If that's not feasible for some reason, then I would suggest it not be included in utiliD.  I don't want dynamic memory allocation in utiliD; that should go into a higher-level library that may import utiliD.

Mike