This is what a good system programming standard library should provide:

1. Types needed to specify library APIs.

2. Functionality for accessing hardware in a non-emulated fashion.

3. Functionality that most _libraries_ need to build on (like arrays/iterators/ranges).

4. Functionality that is tied to the individual compiler (like intrinsics).

5. Memory management.


The primary goal of the standard library should not be to support building applications, but to enable building frameworks and libraries and the interaction between them.


So if one user says "My app needs to read WAV which is RIFF, therefore RIFF should be in the standard library" then the sensible response is: "Do most _libraries_ need RIFF? Why don't you use the recommended audio library which has optimized WAV support.".

Surely everyone uses string processing on a daily basis?

No. I personally almost never do string processing in system level programming languages, what I need is binary serializing support. Or Collada support. Or audio file format support.

I'd be happy to use a recommended string procesessing library when or if I need it.

But a standard library MUST have great support for SIMD intrinsics, i.e. interfacing the hardware,  that is much more critical for system level programming and is tied to the specific compiler.

On Thursday, 28 January 2016 at 10:00:59 UTC, Ola Fosheim Grøstad wrote:
> This is what a good system programming standard library should provide:
>
> 1. Types needed to specify library APIs.
>
> 2. Functionality for accessing hardware in a non-emulated fashion.
>
> 3. Functionality that most _libraries_ need to build on (like arrays/iterators/ranges).
>
> 4. Functionality that is tied to the individual compiler (like intrinsics).
>
> 5. Memory management.
>
>
> The primary goal of the standard library should not be to support building applications, but to enable building frameworks and libraries and the interaction between them.
>
>
> So if one user says "My app needs to read WAV which is RIFF, therefore RIFF should be in the standard library" then the sensible response is: "Do most _libraries_ need RIFF? Why don't you use the recommended audio library which has optimized WAV support.".
>
> Surely everyone uses string processing on a daily basis?
>
> No. I personally almost never do string processing in system level programming languages, what I need is binary serializing support. Or Collada support. Or audio file format support.
>
> I'd be happy to use a recommended string procesessing library when or if I need it.
>
> But a standard library MUST have great support for SIMD intrinsics, i.e. interfacing the hardware,  that is much more critical for system level programming and is tied to the specific compiler.

I do like the building-block idea you suggest, but one must think about the deeper reasons for why things are owned by which people.  (I have found the Coase theorem and work on industrial organisation to be quite stimulating in thinking about this question).

In theory it's completely irrelevant as to whether is something is in the standard library or can just be imported via dub or a git clone, but in practice that's not the case.

As you yourself have mentioned, the size of the D community as it stands today presents some impediment to the possible maintenance and stability of alternative libraries.  If something is in Phobos you know that you can depend on it, that bugs will get fixed, and that changes to the language won't stop the code from compiling (since it will be fixed).  Being in Phobos is a focal point, whereas there simply isn't for now any kind of focal point for external libraries.  The bus factor is high for external libraries - people get sick, divorced, change interest, and so on.

There are also benefits from coherence, since the code will tend to converge on a similar style.

I appreciate that your own interests are quite different, but perhaps you can see that they are only part of what's important for the community as a whole.

It would be nice to have SIMD intrinsics, and in the time you have spent arguing over the years perhaps it would have been possible to help the process of having a high quality and consistent implementation along.  People would also be much more inclined to listen to what you say because then it comes from someone with serious skin in the game.  (Walter himself has spoken about the importance of listening to people who like your stuff and use it already over those who say if only you did X we could use it on a large scale - and that's also plain good business sense).  The gadfly can play a useful role in a community, but not if he continually diminishes the worth of what people are working hard trying to do (in inevitably imperfect human ways).

It really doesn't do any good to worry about what the crowd says and the applause you receive.  If you focus on doing good work (and have some reinforcement along the way because people are using your work to do serious things) then recognition will come.

The work that dlangscience is doing (and one could see ndslice as a part of that) opens up new possibilities that I am quite excited about.  I should say that John Colvin is a contractor for me, but I am not talking up dlangscience because he is helping me, but rather I asked him to help me because I was impressed by what he is doing.

It strikes me as a bit silly to get hung up over language specifications as some kind of mystical talisman.  C was used for years in a serious way before the ANSI spec.  Same for Ruby.  I couldn't immediately see what the situation was for Python.  There isn't the manpower to go that route yet, and nor would it be constructive when certain aspects are not yet as polished as they will eventually be.

Ultimately the test of a language is in whether people can use it to do good work.  Compound growth can be very powerful, and more complex things take time to mature to the point where they are useful in certain domains.  I couldn't have used D for work five years ago, but I can now.  There are these threshold effects, just as Christensen described in the Innovator's Dilemma.  D continues to grow and isn't going to disappear any time soon.  It would be much better to put one's energy into actually writing code that can improve things (even as a proof of concept) than debating what might or might not happen if things don't change (when they always do).  Kingsley hadn't been using D long, liked his IDE, so wrote a D module for it.  It's not perfect yet, but it's a beginning and will mature in time.  That kind of thing will get us much further than debating about what people should do, when there is no army of troops that can be commanded to implement things they are not interested in doing anyway.

Walter clearly doesn't decide everything, any more than a prince does in a monarchy of old.  But we are lucky that he does have such an important influence because he has good taste and a rare skill set.  One gets a higher quality design that way than if you have something that is the pure creation of a committee.

At some point maybe there'll be a need for a committee and a language spec.  That's a higher quality problem to have, but it surely isn't the main concern today.

On Thursday, 28 January 2016 at 11:25:08 UTC, Laeeth Isharc wrote:
> I do like the building-block idea you suggest, but one must think about the deeper reasons for why things are owned by which people.

It is much easier to get motivated if you have a certain level autonomy. Clearly, the "D foundation" should have criterions for making a recommendation, like API guidelines, Boost license and commitment to maintaining it for a period of time.

For instance, Ponce has made some efforts on audio. You and others have made some effort on economic modelling (?). I am interested in audio. Then we have an opportunity to form a team that builds a shared numerics library, with some input from some Phobos coordinator so that ranges are supported in a consistent matter.

But here is the key point: we don't have to get input from all D programmers. We can form consensus and take ownership and make faster decisions. And we can scrap it in two years, learn from our mistakes and create a much better API.

I think that Sonke received too much "negative motivation" for his contributions recently, if I had been in his shoes I'd probably found working on vibe.d more fun. IRRC Ruppe also have voiced that he want to work on libraries which he has more freedom with.

An frankly, as APIs have to be vetted on large applications in maintenance mode a lot of the effort put into arguing the design "perfect Phobos libraries" most likely will be in vain.

>  (I have found the Coase theorem and work on industrial organisation to be quite stimulating in thinking about this question).

I noticed you and the other economic theory guys in the thread mentioning some interesting models. I'd like to look at those later when I have more time :-). Always nice to find some shared language for expressing system dynamics.

> In theory it's completely irrelevant as to whether is something is in the standard library or can just be imported via dub or a git clone, but in practice that's not the case.

Well, but if something is in the standard library, other libraries will build upon it and add dependencies to it even if they don't really have to. Meaning, they will pull in dependencies and bloat and make Phobos costly to maintain as you will end up with std.json, std.json2, std.json3 etc.

> As you yourself have mentioned, the size of the D community as it stands today presents some impediment to the possible maintenance and stability of alternative libraries.

No, I don't think so. I think that D has nurtured the idea that Phobos should provide everything and therefore the modus operandi is to complain that Phobos does not provide it.

> The bus factor is high for external libraries - people get sick, divorced, change interest, and so on.

Recommended libraries should have at least 3-6 people behind them

> It would be nice to have SIMD intrinsics, and in the time you have spent arguing over the years perhaps it would have been possible to help the process of having a high quality and consistent implementation along.

SIMD isn't critical to me at this point in time. I am currently using C++ for what I would have liked to use D for. D needs more compiler developers, a strategy to get there and someone to take D there with focused leadership.

The main challenge is not code, it is process.

> It really doesn't do any good to worry about what the crowd says and the applause you receive.  If you focus on doing good

I don't worry about applause. Describe the key issues and explain it from various angels whenever it comes up and people start to develop a shared understanding of what needs to be done.

Without a timeline, why invest? The vision for 2016 looks pretty good, but there it lacks the details of what and when. What are the estimates. What are the explicit goals? How many man hours (min/max) are needed. Goals and estimates.

> excited about.  I should say that John Colvin is a contractor for me, but I am not talking up dlangscience because he is helping me, but rather I asked him to help me because I was impressed by what he is doing.

That's great! :-)

> It strikes me as a bit silly to get hung up over language specifications as some kind of mystical talisman.

It brings unfortunate inconsistencies and implementation effects to the foreground so that they can be addressed. It is near impossible to discuss the design without one.

> debating about what people should do, when there is no army of troops that can be commanded to implement things they are not interested in doing anyway.

Attract more developers.

1. Find out why they don't come.
2. Find out why they leave.
3. Address the issues they have with D.

In my case and for many others the reason go along the lines of: memory management, language inconsistencies, a refusal to add builtin tuples and platform support.

On Thursday, 28 January 2016 at 11:25:08 UTC, Laeeth Isharc wrote:
> As you yourself have mentioned, the size of the D community as it stands today presents some impediment to the possible maintenance and stability of alternative libraries.  If something is in Phobos you know that you can depend on it, that bugs will get fixed, and that changes to the language won't stop the code from compiling (since it will be fixed).


On the other hand, Phobos is so tightly coupled to dmd that if you want a Phobos bugfix, you might also be stuck with a dmd regression and get nowhere.

Moreover, Phobos doesn't really get all that many bug fixes. It lags on the release cycle and a few modules are basically just abandoned. Third party alternatives have been available the whole time though.

> The bus factor is high for external libraries - people get sick, divorced, change interest, and so on.

But there's nothing stopping you from forking it yourself...

On Thursday, 28 January 2016 at 11:25:08 UTC, Laeeth Isharc wrote:
>
> I do like the building-block idea you suggest, but one must think about the deeper reasons for why things are owned by which people.  (I have found the Coase theorem and work on industrial organisation to be quite stimulating in thinking about this question).
>
> In theory it's completely irrelevant as to whether is something is in the standard library or can just be imported via dub or a git clone, but in practice that's not the case.
>

Sigh...

The Coase Theorem is about externalities. The whole point of the Coase theorem is that when one person is causing a nuisance or polluting it is costly to bargain so the efficient allocation may not result. The law/courts should assign property rights in such a way to maximize efficiency. This does not seem relevant.

Based on the second paragraph, I think you meant Ronald Coase's work in the paper The Nature of the Firm. This paper asks the question why things are made in a firm or contracted to other firms. This clearly parallels your discussion of why things are included in phobos or not.

However, the whole point of the Nature of the Firm is that transaction costs exist in the real world. Firms trade-off the transaction costs of contracting business outside the firm with the benefits of doing things in house. These pressures lead to constraints on the maximum size of a firm.

Coming back to phobos and the fact that there are transaction costs in the real world (discussions on the forum about the future of D are clearly transaction costs), this implies that "in theory" it is not irrelevant as to whether something is in the standard library or not. As discussed elsewhere, there are clearly benefits to putting some things in phobos (if only for providing a framework for others), and there are costs as it gets too large.

On Thursday, 28 January 2016 at 16:12:44 UTC, jmh530 wrote:
> the standard library or not. As discussed elsewhere, there are clearly benefits to putting some things in phobos (if only for providing a framework for others), and there are costs as it gets too large.

That's the maintenance costs, but there are other related costs:

1. It takes a lot of work to get it in, you have to negotiate with non-domain experts to get in improvements.

2. The presence of sub-optimal standard functionality discourage development of slightly better functionality as a third party solution. So you loose evolutionary advantages.

3. You cannot easily modify it as it is distributed with the compiler. A standard library is essentially an API with a reference implementation, but compilers can do whatever they want in terms of implementation. Changes can therefore lead to incompatibilities between compilers.

4. You cannot easily fix bugs, because applications depends on the old behaviour. So a bug fix is a breaking change. You have to deprecate and provide the same functionality under a new name instead.

External libraries can avoid a lot of these issues by versioning. Selecting between many different versions of submodules of a standard library is way too complicated.

On Monday, 25 January 2016 at 03:49:56 UTC, Rikki Cattermole wrote:
> That won't be happening anytime soon.
> Until we have image and windowing in Phobos (I'm working on both) there is no way a GUI toolkit is going in. And from what I know there will be a LOT of work to update it.

I've read this thread partially and I agree with you. In my opinion the key to the success is a good standard library with batteries included.

The opinion that this approach is outdated is very subjective.

I hope D GUI will be usable some day for me and other people not wanting to fight with tools (and external libraries).

If there is something from your project ready for test drive let me know.

Piotrek

On 29/01/16 6:40 AM, Piotrek wrote:
> On Monday, 25 January 2016 at 03:49:56 UTC, Rikki Cattermole wrote:
>> That won't be happening anytime soon.
>> Until we have image and windowing in Phobos (I'm working on both)
>> there is no way a GUI toolkit is going in. And from what I know there
>> will be a LOT of work to update it.
>
> I've read this thread partially and I agree with you. In my opinion the
> key to the success is a good standard library with batteries included.
>
> The opinion that this approach is outdated is very subjective.
>
> I hope D GUI will be usable some day for me and other people not wanting
> to fight with tools (and external libraries).
>
> If there is something from your project ready for test drive let me know.
>
> Piotrek

Right now, image library is more or less ready for next feedback.
Windowing is almost there, really just needs a bit of testing and its done.

So in other words, the hold up, is me.

On Thursday, 28 January 2016 at 12:40:56 UTC, Ola Fosheim Grøstad wrote:
> I think that Sonke received too much "negative motivation" for his contributions recently, if I had been in his shoes I'd probably found working on vibe.d more fun. IRRC Ruppe also have voiced that he want to work on libraries which he has more freedom with.

Yeah, I think getting in Phobos is a waste of time and likely a net negative on the library due to how much harder it is to change phobos vs changing your own file.

In my perfect world, quality third party apps - as determined just by usage stats or something - would be automatically downloadable and their documentation searchable as if it was standard.

When you do `import std.string;` you expect it to just work, and you find std.string's docs easily from dmd.

I'd love it if you could do `import thirdparty.independent;` and it magically works too - without even need for a configuration file or an install command. And the docs are right there and tutorials are written however the author feels like writing them.


Then the line between "standard library" and other library basically disappears.


While that isn't likely to happen, we could at least start promoting third party stuff more equally.

> An frankly, as APIs have to be vetted on large applications in maintenance mode a lot of the effort put into arguing the design "perfect Phobos libraries" most likely will be in vain.

This is a reason why I tend to only write libs that I actually use myself - at least then I know every function has one happy user.

On Monday, 25 January 2016 at 02:37:40 UTC, Andrei Alexandrescu wrote:
> Hot off the press! http://wiki.dlang.org/Vision/2016H1 -- Andrei

Just out of curiosity, is getting the different compilers in sync still a priority? Right now we have dmd at 2.070, ldc at 2.068. and gdc at 2.066.