On Thursday, 25 June 2020 at 16:23:11 UTC, Petar Kirov [ZombineDev] wrote:
> [snip]
>
> I agree, this would at least allow people pick an exact version. (Of course, given that Phobos doesn't follow SemVer, every minor release (2.xx.0) could introduce a breaking change - a symbol being removed after the 2 year deprecation period.)
>
> But still, the next major problem to solve is actually ensuring that each Phobos release works with several different compiler releases, which as far as I can see is not going to happen anytime soon.

File bugzilla issue?

On Thursday, 25 June 2020 at 15:33:18 UTC, Paolo Invernizzi wrote:
> [snip]
>
> I understand your point of view and what you are trying to say. In my opinion it's a chicken and egg problem, IMHO.

Yes, that's a fair assessment.

> My question is: why 90% of the code you need is not in Phobos,

I can go in more detail later, but let's just say that Phobos is a generic library aiming to solve general use-cases of most/all language users. In many domains, such as mine that's less than 5% of what I need, the rest is really domain-specific. And obviously there are more domain-specific libraries in other more popular languages.
Also, Phobos is very poor in terms of general application-level programming. Unlike e.g. Node.js or Flutter, it's neither sufficient for frontend (web/mobile/desktop) nor backend programming.

> and why is Phobos stagnating?

Several related reasons from the top of my head:
1. People are simply not interested in contributing (anymore).
A few hypothetical examples (I/me == random contributor):
1.a. As an individual developer I write code for fun, so I don't find value in going through the difficult (as it should be) process of proposing my code for std and passing through code reviews.
1.b. As an organization, we like to contribute to the community, and publishing on GitHub/GitLab/etc. + code.dlang.org is mutually beneficial: Other community members or companies can find value in what we do and can reports bugs and contribute improvements. However, contributing to Phobos takes just too long and also limits the potential number of contributors. Also we don't control the code anymore. So in general it's hard to justify from a business point of view.
1.c. (I guess that would be a fair approximation of the ndslice case): I'm a highly motivated individual who wants to contribute a foundational piece to the standard library, the positive effects of which would permeate throughout the ecosystem. Few problems:
- I want to support older/newer GDC/LDC versions (a few years ago that wasn't at all possible for upstream phobos)
- It's a nice start, but actually the reason I made this contribution is so I can advance in my domain (e.g. scientific computing, fintech, whatever), so now I need to to write domain specific libraries. Well actually, to support those libraries I need to make changes to the API. Well I can't do that easily anymore as I don't own the code and breaking changes are not well supported in Phobos (no SemVer). I also want betterC support, so I need to vendor (fork and rewrite) parts of druntime. The changes are good enough for me, but they can't be accepted upstream. And so my library now has non-Phobos dependencies, and so obviously it can't be part of Phobos.
1.d. I work on a security-sensitive library (e.g. crypto). I need to follow a responsible security vulnerability disclosure policy. Also when I have fixed the issue, and the changes are audited, I need to push a new release ASAP. A new compiler release (even point/bugfix one) typically has a latency of at least 1 month. It can contain also other unrelated breaking changes.

2. Failure to cultivate and *****retain***** community talent. By community talent I mean contributors who are *not* paid. E.g. Dicebot, JinShil, berni, Kenji, Joakim, and so on. Yeah, each case has its specifics, but if people with contributions of their caliber were to leave my company, I would be seriously worried and losing sleep (and selling stonks :D)
For example, not until recently (let's say 2017/2018), if a newcomer were to look at this page: https://github.com/dlang/dmd/graphs/contributors they would conclude that dmd was Kenji and Walter's project (and not Walter + contributors), as Kenji had substantially bigger number of commits than Walter at that point of time.


> In the just released beta of DMD, there are zero Phobos references in the changelog ...

Well, this varies with each release, but outside of bug fixes, refactoring or otherwise improvements to existing code and addition of fundamental vocabulary types (e.g. SumType), what exactly do you expect to see? Can you list exactly what kinds of contributions would you expect to see from non-paid volunteers? And I don't mean to say "non-paid" in a bad way regarding DLF. I mean that people who work for free are usually even more self-motivated then paid contributors. Why would anyone want to put new modules in Phobos, where they can develop them much faster in the open?

Also, contributing to the compiler and runtime has a much higher value than Phobos, especially given how easy and beneficial in many cases it is to replace Phobos with a code.dlang.org package.

Additionally, most of the code in dmd/druntime is internal and it's much easier to change than Phobos, which for the most part is comprised of public functions. In the compiler and the runtime you can make huge changes without breaking backwards compatibility at all.

>>> Anyway, if we can, we try to stick with Phobos, as long as we don't have a particular problem to solve that needs an external library: recent example, Martin std.io or SumType instead of std.variant ...
>>
>> See, some of the problems in Phobos already have a good solution. The problem that we need to address is the trust.
>> The more high-quality and well-maintained libraries there are, the less cautious one would need to be before reaching to code.dlang.org.
>
> It's not only a problem of trust, it's a problem of integration and maintenance also: for example, some days ago I tried to use Martin and Steven std.io with std.concurrency, and I faced a wall of problems related to non copiable struct, variants, reference counting implementation, and so on ...

By trust, I also meant compatibility and general dependability.
For example, in the frontend world, even though Redux and React are developed by different teams (there's overlap but still), I feel like an illuminati apocalypse would be a more likely event than Redux somehow breaking React.js compatibility or vice versa :D

In the case of Steven and Martin's libraries, they're both pretty cool guys, whom I've met and chatted with at DConf, so I trust them as people, but they're both pretty busy these days (just my casual observation from GitHub and the forums), so I can't "trust" these libraries for critical stuff, as they don't have a high enough number of contributors who could take the lead in case Martin or Steven are not there.

I really hope we can improve as a community in this regard.

> I know that there're advantages in flexibility, but there are costs also, for example, not everyone scouting for DLang as a candidate for a task has your (great!) experience in managing all the sort of problems that arises when trying to combine different packages, or has the experience (or the time?) to choose the "golden" one.
>
> Anyway, you are right, it's not a binary choice.
>
> /P

Yeah, flexibility almost always brings complexity (e.g. generic code can be easy to write, but sometimes it can be complex to debug). Really, what I mean is that we need to create a carefully curated collection of packages that work together and are maintained by a good number of qualified people. That's kind of the idea of dlang-community, but with a single large project like DCD in it (and its dependencies), it kind of twists the balance of contributions towards DCD. There's also the vibe-d and libmir organizations. However, we lack a broader collaboration and cooperation between them.

> managing all the sort of problems that arises when trying to combine different packages

P.S. That is actually full-time job in the npm world :D
But it actually makes a lot of sense from business point of view. Nowadays, no company, no matter how huge could build in-house all that already exists (and is of high-quality) in the open-source world. It's cheaper to spend one week choosing 1 library from 100+ then spending a few months to create another one that solves the same problem. Especially if your in-house developers need to move on a different project and then no one could maintain your in-house creation.

----

These are all really difficult problems to solve, but solving them can potentially create the biggest value of the community. Just like the Nodejs codebase represents <<<< 0.01% of the code of the whole npmjs ecosystem, the D language spec, dmd, druntime, phobos and the tools are absolutely foundational and critical, but would still represent a small fraction of the total value that our community can create.
We now need to focus on scaling the bigger part of the pie.

On Thursday, 25 June 2020 at 17:11:58 UTC, jmh530 wrote:
> On Thursday, 25 June 2020 at 16:23:11 UTC, Petar Kirov [ZombineDev] wrote:
>> [snip]
>>
>> I agree, this would at least allow people pick an exact version. (Of course, given that Phobos doesn't follow SemVer, every minor release (2.xx.0) could introduce a breaking change - a symbol being removed after the 2 year deprecation period.)
>>
>> But still, the next major problem to solve is actually ensuring that each Phobos release works with several different compiler releases, which as far as I can see is not going to happen anytime soon.
>
> File bugzilla issue?

https://github.com/dlang/projects would be more appropriate, so if other people see value in this, I hope they will post there (I'm running out of time to continue this long topic atm).

On Thursday, 25 June 2020 at 16:42:36 UTC, aberba wrote:
>
> Almost the exact reason why i use JavaScript (nodejs) for work and not D. It takes more than a nice language. When some here say they don't use Dub or they don't care about dub packages, I'm wondering what kind of software they are writing.
>
> In my line if work, it's highly dependant of availability of packages to get the job done. Encryption, networking, data manipulation, database connectors, socket, rest, cloud services, etc. Its more like glueing services and APIs together.
>
> Unless you're building compilers and softwares that don't connect to internet, then it makes sense to not be bothered by the lack of certain packages on dub.
>
> This will only happen when the community collaborates, which will only happen when there's a visible leadership push/backing. I meant D conf online has taken forever to happen and I'm not sure if community building is a thing prioritized at all.
>
>
> I've been policing and trying several packages on dub...some, talking to their maintainers. I want to make this happen. Its the only way.
>
> I rarely use core nodejs packages. Its like npm has got everything. Its what dub needs. Already I'm happy with quite a few dub packages. And there's huge!! room for improvement if only there's a visible official push. Its not always about money.
>
> I'm not sure why we don't even have regular online community meetups. Yeah, there's people who would rather code and feel more productive that way. That's also fine.
>
> Except there's another side about community building. DConf is a Joy to watch. Just the community interaction alone and how ideas are shared. It could happen more often. More often online.

I agree with everything you said.

Currently, *each* member of my dev team lives in a different country and in a different time zone, so all my day is spent in online collaboration (chat, audio and video calls) so I also can't understand why there hasn't been a stronger push for this kind of communication in D. https://dlang.slack.com/ has been a great improvement in this regard from the past couple of years, but it is insufficient.

On Thu, Jun 25, 2020 at 04:42:36PM +0000, aberba via Digitalmars-d wrote: [...]
> Almost the exact reason why i use JavaScript (nodejs) for work and not D. It takes more than a nice language. When some here say they don't use Dub or they don't care about dub packages, I'm wondering what kind of software they are writing.

Don't get me wrong, I'm not downplaying dub packages.  But for my projects, it's mainly local computation and data processing and transformation.  One project *is* related to publishing on a website, but it's almost entirely prerendered static pages, so virtually all of the processing is done in batch-oriented mode beforehand.


> In my line if work, it's highly dependant of availability of packages to get the job done. Encryption, networking, data manipulation, database connectors, socket, rest, cloud services, etc. Its more like glueing services and APIs together.

Actually, even in this area D shines, because of the ease of wrapping existing C/C++ libraries.  If there's a C API for some library that provides the functionality you need, it's easy to interface with it. I've interfaced with e.g. libfreetype, X11, sqlite3, and a bunch of other libraries, and it's pretty convenient.  Of course, the best is if there's an existing D binding, then it's just plug-in and go, but even where there isn't an existing binding, it's still pretty easy to declare the C prototypes on an as-needed basis (which is what I did with X11 -- translating the entire X11 headers is too much work, and most of the API I don't even need anyway, so I just declare what I need and add more as the need arises).


> Unless you're building compilers and softwares that don't connect to internet, then it makes sense to not be bothered by the lack of certain packages on dub.

It depends on what kind of software you're building, certainly.  If you're writing code for today that will be obsolete in 2-5 years' time, then having a dub dependency pull in 200 dependencies is not a problem, because by the time it becomes a problem you'll already have moved on anyway.

I write my code to last; so anything that increases the likelihood of future breakage is a big disincentive for me.  I really do not wish to be rewriting a module that was working perfectly fine before, but broke because some obscure dependency or sub-dependency decided to change things in a way that breaks the way my code uses it. I consider that time-wasting churn.  That doesn't mean I expect code to be static -- but any change has to make actual progress, as in fixing a bug or adding a new feature, or improving the quality of the code, not just change for change's sake because of external factors completely unrelated to what I'm doing.


> This will only happen when the community collaborates, which will only happen when there's a visible leadership push/backing.

I'm not following this logic.  Why can't the community collaborate in spite of perceived lack of visible leadership push/backing?  The two are not mutually exclusive.  In fact, this being a volunteer-driven project, even if the leadership *does* push for community collaboration, it doesn't guarantee it will actually happen.  Volunteers do whatever they want to do, not what they're told to do, after all.


[...]
> I'm not sure why we don't even have regular online community meetups.

Because so far, nobody has stepped up to take charge of it.  Since you feel so strongly about this, why not be the one to take charge?  Set up an online meeting, post an announcement here to recruit people, and get the ball rolling.  Nobody's going to stop you, and in fact, judging by what many people are saying, probably a lot of people will appreciate the effort and support you.  If your effort turns out to be successful, it could even convince the BFDL(s) to officially endorse it.


> Yeah, there's people who would rather code and feel more productive that way.  That's also fine.
[...]

Personally, I have very limited time as it is to work on my D projects, so whatever little time I have, I'd really rather devote to making progress with my code than to participate in yet another online meeting/forum/what-not (which I already have more than enough of). But that's just me, personally; I don't speak for anyone else here.  And I also have no opposition to such a thing, should you decide to take it up. I might even support you in the background.  And I'm sure a good number of people here will support you too, based on what people have been saying about this.

So how about it?  Since D is lacking in this area, here's your opportunity to fill the gap.  Show us how to run the community better than we have been.  We'll support you.  Show us how to make things take off.


T

-- 
It's bad luck to be superstitious. -- YHL

On Thursday, 25 June 2020 at 18:25:47 UTC, Petar Kirov [ZombineDev] wrote:
> On Thursday, 25 June 2020 at 17:11:58 UTC, jmh530 wrote:
>> [snip]
>>
>> File bugzilla issue?
>
> https://github.com/dlang/projects would be more appropriate, so if other people see value in this, I hope they will post there (I'm running out of time to continue this long topic atm).

I filed one here
https://issues.dlang.org/show_bug.cgi?id=20977
anyone is free to improve it and add to projects.

On Thursday, 25 June 2020 at 17:03:02 UTC, Andrei Alexandrescu wrote:
> On 6/25/20 12:42 PM, aberba wrote:
>> This will only happen when the community collaborates, which will only happen when there's a visible leadership push/backing.
>
> This is speculation. What is it based on? Boost was started by Beman Dawes with no backing. I know next to nothing about npm, but a look at their website www.npmjs.com suggests it's a traditional for-profit company with no ties to ECMA or other JavaScript formal leadership bodies.

First, you're making the whole point of it about npm (JavaScript) specifically, there's also Cargo (rust).

JavaScript is now very big so there's multiple movements. But let's take nodejs, there's a foundation. I've watched several meetings by the core team and you can find them on YouTube. Its open and live for everyone to see.

>
> What concrete forms of said leadership push/backing do you envision?
Again since I'm more into nodejs recently, I'll just show you their get involved page.

https://nodejs.org/en/get-involved/

I want to be fair because nodejs community has grown and there's many stakeholders now...but the initial work to attract interest has to be done first.

I don't have all the answers. I may not have articulated my opinions well...for sure. However the thing about community, I'm not the only person who has that view.

On 6/25/20 3:38 PM, aberba wrote:
> On Thursday, 25 June 2020 at 17:03:02 UTC, Andrei Alexandrescu wrote:
>> On 6/25/20 12:42 PM, aberba wrote:
>>> This will only happen when the community collaborates, which will only happen when there's a visible leadership push/backing.
>>
>> This is speculation. What is it based on? Boost was started by Beman Dawes with no backing. I know next to nothing about npm, but a look at their website www.npmjs.com suggests it's a traditional for-profit company with no ties to ECMA or other JavaScript formal leadership bodies.
> 
> First, you're making the whole point of it about npm (JavaScript) specifically, there's also Cargo (rust).

Yah, rust is an even better example. They have talented people who are dedicated to everything related to the rust community, and doing so improves their own career and pays their bills. How would you walk and talk if your formal job, and that of a team around you, would be to serve, manage, and improve the D community? Completely different ballgame.

At least with the resources we can muster for the time being, this is a war of attrition we can't sustain. It's simple. Our flamewars are similar to the poor couple fighting and giving each other the example of the Joneses and how easy everything seems to be for them.

On Thursday, 25 June 2020 at 16:25:01 UTC, Andrei Alexandrescu wrote:
> On 6/25/20 10:27 AM, Petar Kirov [ZombineDev] wrote:
>> The way I see things is that we as a community need to focus on a vetted, well shaped collection of libraries.
>> We need to have process where third-party libraries are able to gain broader support and join dlang-community. Also we need dlang-community to have a healthy number of people who actually maintain the code.
>> We need the leadership to realize that investing just in dmd, druntime and phobos is necessary, but very insufficient.
>> W&A need to stop pretending that Dub doesn't exist. I feel that unless we embrace using code.dlang.org as method of distribution of everything that's currently part of the compiler archive, Dub's limitations will never be addressed and from that the broader community will suffer.
>
> (By the way just to clarify "A" in "W&A" would stand for Atila. I'm commenting as just one in the community.)

Hi Andrei, well, I meant Walter and you during 2015-2019, though just to be clear, I neither wish, nor it was my intension to turn this discussion into a personal attack. My goal is to stress how important it is to focus on helping the community grow and scale in a healthy way. Focusing just on the language spec and implementation is missing the bigger picture, IMO. But since you decided to address specifically this point, I'll try to elaborate more:

I don't recall you or Walter getting involved with the development of dub or the dub-registry in any significant manner. I remember that you criticized the addition of SDLang as an alternative to JSON package description format, but that's it.

I feel there is a very big disconnect between you and the community. One on hand, I think that Dub and code.dlang.org are one of the biggest assets of the community. On the other hand, it seems to me that you're not interested in Dub at all. It's just another community project that "they"'re free to do whatever they want it.
For example, my impression (though I'd be happy to be wrong), is that you have never actually tried using Dub for any project of significant size, or such depending on a significant proportion of code that you don't maintain. On the other side, take as an example any company that needs to bring a D web project quickly to market. The first thing they'll likely reach for is dub + vibe-d + mysql-native + some js frontend framework - basically 100% third-party code for that team. Since (as far as I'm aware) you haven't invested much time in projects like this, I'd guess that it would be hard for you to fully appreciate the necessity, challenges, and benefits of using a package manager like Dub. Which is why, to me it seems that Dub is not very high on your radar of important things to improve for to D succeed.

Different people having different priorities is absolutely normal but on the other hand, if hypothetically most people using D type `dub` in their terminal say 99% of their time compared to 1% for `dmd`, perhaps it's not unreasonable for the community to expect the leadership to take a Dub's issues in a proportional manner. Or at least the community would find it strange that the keywords code.dlang.org or Dub were not mentioned even once in the vision documents, e.g. https://wiki.dlang.org/Vision/2018H1. Yeah, "tooling" is mentioned, but honestly, that too unspecific to be actionable. On a related topic, "mobile" and "wasm" also not mentioned, which areas where D in theory could do very well, and its direct competitors are rapidly gaining market share.

> I'm a bit confused by this argument. So there is the compiler and standard library, and then there's a community-supported library, which includes a copy of the standard library under an improved versioning schema. It's all legal and encouraged by the generous licensing of the compiler and standard library, both of which Walter and I fought hard in the past to obtain and maintain.

Okay... though I don't see why you need to point this out.
Yes, getting dmd fully Boost-licensed was a great achievement, and I and the rest of community are all grateful for that, though we're currently discussing about Phobos. I have never used a language which doesn't have an open-source and permissively-licensed standard-library.

> Dub is the perfect place where community leadership can set the tone and organized things as they need to without heavy-handed intervention from above. Several people have criticized "the leadership" for not doing things the way they wanted, so it seems perplexing that leadership intervention is now asked for. What's that nonsense with "pretending dub doesn't exist"? So now they need to take ownership of the community-driven dub as well? Isn't that a bit too much work for a handful of folks in the best of times?
>
> Is money that's been asked for? We don't have enough to do competitive hiring.

No, money is not what's been asked for. For me, the most straightforward way to look at things is as if the DLF doesn't exist and there's no money on the table in the first place. (Also funding can only reward future contritions, so in a sense, it's not fair to past contributions.) So what's left (in this way of looking at things) is just individuals motivated for whatever non-financial reasons (at least not direct ones) who are working to improve D. It follows that motivated, talented people who effectively collaborate and cooperate are the primary asset of the community. The job of the "community leadership" would then be to attract, cultivate, and **retain** community talent and endorse collaboration. If funding is not an asset, then the only remaining leverage is good community management.

> I don't think Bjarne or Herb have ever been accused of not contributing to Boost (which they didn't), or asked to invest in it. Boost has been a community-led effort which has had a great influence on C++ development - with the participation of volunteers, not the formal standards committee.

We can only compare D to C++ on technical merits. The size of their community is just in a very different scale to ours. For example, I work in a startup where everyone needs to wear many hats and handle a very diverse set of responsibilities. We have more projects than people. Previously, I've worked at Fortune Top 10 software company with 20k+ employees. Needless to say, the scale is very different.

Instead, we should compare D to language communities developed in the past 10-15 years. I suggest you look around and see how each one is doing and what were the different steps they took. I can assure you that probably all of their leaders at some points have been accused of not paying enough attention to this or that thing important to some part of the community. That's normal. Leading a small community or organization is very different than leading a big one.

>>> Anyway, if we can, we try to stick with Phobos, as long as we don't have a particular problem to solve that needs an external library: recent example, Martin std.io or SumType instead of std.variant ...
>> 
>> See, some of the problems in Phobos already have a good solution. The problem that we need to address is the trust.
>> The more high-quality and well-maintained libraries there are, the less cautious one would need to be before reaching to code.dlang.org.
>
> This is and will always be the case, and it's normal. A standard library can't implement all special purpose libraries, or even a significant fraction. A community-led library is a great thing.

I agree completely. I was not trying to argue the other way around. I think Phobos needs to be slimmer (e.g. throw away curl, odbc and other nonsense).
For example, the only languages that have good built-in networking/HTTP libraries are the ones where this has been an explicit goal of the maintainers. Right now we don't have sufficient Phobos maintainers with the experience and interest in building a GUI framework, so nothing like this should even be attempted to be added to Phobos.

A library should only try to do what its maintainers are good at. In the case of Phobos that would be generic code like algorithms, ranges, allocators, containers, typecons, meta programming, process, path and file manipulation and basic IO. Everything else should be removed. I'm glad that https://dlang.org/phobos/std_xml is finally going away, as probably only Jonathan has real experience of building an XML library (hence dxml). Though that should have happened much sooner.

On Thursday, 25 June 2020 at 19:49:21 UTC, Andrei Alexandrescu wrote:
> Yah, rust is an even better example. They have talented people who are dedicated to everything related to the rust community, and doing so improves their own career and pays their bills. How would you walk and talk if your formal job, and that of a team around you, would be to serve, manage, and improve the D community? Completely different ballgame.
>
> At least with the resources we can muster for the time being, this is a war of attrition we can't sustain. It's simple. Our flamewars are similar to the poor couple fighting and giving each other the example of the Joneses and how easy everything seems to be for them.

The only thing the community can do is voice their disdain with what they are unhappy with. The situation with DIP1028 is what it takes to change course of the SS Didanic. The whole community has to be riled up in protest for a change to actually happen. It's difficult to make even small changes for the better.