June 25, 2020
On Thursday, 25 June 2020 at 12:28:27 UTC, Guillaume Piolat wrote:
> On Thursday, 25 June 2020 at 06:52:05 UTC, Petar Kirov [ZombineDev] wrote:
>>
>> The Rust community has realized that a slim standard library + a solid package manager is the way to go.
>
> The practice seems to indicate that the theory is false.
>
> The very first Rust program I built with Rust was an _example_ program for the Amethyst game engine (granted, a big endeavour).
>
> It downloaded _146_ packages and then proceeded to build them at a glacial pace. This was a really "npm" moment.
>
> In D my _full_ products have 6 packages (24 when counting sub-packages).
>
> It seems slim stdlib has enormous community implications and require inordinate amount of coordination and agreements ; it changes the whole culture.

The only thing that I need to say is that our team uses npm (actually yarn) for our stack and we're doing very well :)

So 146 number is pretty laughable in comparison. Running `yarn list | wc -l` on a random project of our gives me 7368 :D

Whether this is a good thing is a different question altogether, but in practice, this is not an issue at all for us and for thousands of other JS/TS teams around the world. What I know is that npm and yarn handle this huge (in comparison) number very well, but I'm not so sure about Dub. For example, yarn.lock has the SHA-512 of each of my dependencies. dub.selections.json only includes the version numbers.

June 25, 2020
On Thursday, 25 June 2020 at 12:28:27 UTC, Guillaume Piolat wrote:
>
> It seems slim stdlib has enormous community implications and require inordinate amount of coordination and agreements ; it changes the whole culture.

I just want to add that this is a *very* important point. We definitely need more collaboration and coordination in our community. Some examples:
- dstep, dpp, Calypso
- install.sh, dvm, digger (much broader scope, but can also do what dvm does, AFAIR), setup-dlang (only usable for GitHub actions, but actually replicates the code of install.sh and dvm to some extent).
- various @nogc libraries (containers, threading, etc.)
- various networking libraries
- etc.

I guess, the most important thing that I see in other language communities that I miss in D is exactly the coordination and cooperation. Of course it's very far from perfect in all of them, but it still feels that D is falling behind.
And I think that this is not a money problem, but a community and leadership problem.
June 25, 2020
On Thursday, 25 June 2020 at 14:27:02 UTC, Petar Kirov [ZombineDev] wrote:
> Technically, phobos is actually already on Dub (https://code.dlang.org/packages/phobos), so in the future nothing could prevent you to continue using it, but you would just get it from there and not from the compiler distribution archive. It's the same code, made by the same people, just with a different distribution model.

Aside: it seems like Dub is not properly picking up the tagged releases, and only has '~master' and '~stable' available. Fixing this would make using Phobos via dub a much more attractive proposition.
June 25, 2020
On Thursday, 25 June 2020 at 14:41:49 UTC, Petar Kirov [ZombineDev] wrote:
>
> So 146 number is pretty laughable in comparison. Running `yarn list | wc -l` on a random project of our gives me 7368 :D
>

I'l assume your post isn't satire ^^

This doesn't rezally work for native since:
- there is the cost of compiling this ahead of time, which isn't negligible and with a cost structure unlike Javascript (in the best case, code has to be compiled in and then stripped by linker, very slow)
- native programs have many possibilities to go wrong (deadlocks, threading, memory allocations...) and that differs from project to project ; you cannot enforce any invariant with such a number of libraries. And it's a huge hurdle for "common off the shelf" libraries.
June 25, 2020
On Thursday, 25 June 2020 at 14:27:02 UTC, Petar Kirov [ZombineDev] wrote:
> On Thursday, 25 June 2020 at 12:13:21 UTC, Paolo Invernizzi wrote:
>>
>> I know that we are quite few on this side, but IMHO a well shaped standard library has advantages over a sparse ecosystem of independent library ...
>
> Who says that this is a binary choice? And also, the way you fraise it it's obvious that say Flutter is better choice than React as everything is very tightly integrated and you could build your whole app without reaching for third-party code. Also, it's obvious that in terms of trust, it's better to rely on the diligence of the upsteam development team, than random strangers from the internet.
>
> But what if 90% of the code my team needs has no place in Phobos? We already have to rely heavily on third-party packages. Yes there are a lot of risks and we try to be careful, but we have no other choice. We could spend years reiventing the wheel (high-quality code that already exists in other languages), or we could focus on delivering products to our users based on the already existing ecosystem in other languages. Well, a choice would exist if we had an unlimited budget, but given that we're not as lucky, we can only use other languages for our projects currently.
>
> Technically, phobos is actually already on Dub (https://code.dlang.org/packages/phobos), so in the future nothing could prevent you to continue using it, but you would just get it from there and not from the compiler distribution archive. It's the same code, made by the same people, just with a different distribution model.
>
> The way I see things is that we as a community need to focus on a vetted, well shaped collection of libraries.
> We need to have process where third-party libraries are able to gain broader support and join dlang-community. Also we need dlang-community to have a healthy number of people who actually maintain the code.
> We need the leadership to realize that investing just in dmd, druntime and phobos is necessary, but very insufficient.
> W&A need to stop pretending that Dub doesn't exist. I feel that unless we embrace using code.dlang.org as method of distribution of everything that's currently part of the compiler archive, Dub's limitations will never be addressed and from that the broader community will suffer.

I understand your point of view and what you are trying to say. In my opinion it's a chicken and egg problem, IMHO.

My question is: why 90% of the code you need is not in Phobos, and why is Phobos stagnating? In the just released beta of DMD, there are zero Phobos references in the changelog ...

>> Anyway, if we can, we try to stick with Phobos, as long as we don't have a particular problem to solve that needs an external library: recent example, Martin std.io or SumType instead of std.variant ...
>
> See, some of the problems in Phobos already have a good solution. The problem that we need to address is the trust.
> The more high-quality and well-maintained libraries there are, the less cautious one would need to be before reaching to code.dlang.org.

It's not only a problem of trust, it's a problem of integration and maintenance also: for example, some days ago I tried to use Martin and Steven std.io with std.concurrency, and I faced a wall of problems related to non copiable struct, variants, reference counting implementation, and so on ...

I know that there're advantages in flexibility, but there are costs also, for example, not everyone scouting for DLang as a candidate for a task has your (great!) experience in managing all the sort of problems that arises when trying to combine different packages, or has the experience (or the time?) to choose the "golden" one.

Anyway, you are right, it's not a binary choice.

/P



June 25, 2020
On Thursday, 25 June 2020 at 15:30:18 UTC, Guillaume Piolat wrote:
> On Thursday, 25 June 2020 at 14:41:49 UTC, Petar Kirov [ZombineDev] wrote:
>>
>> So 146 number is pretty laughable in comparison. Running `yarn list | wc -l` on a random project of our gives me 7368 :D
>>
>
> I'l assume your post isn't satire ^^

Yes, it's just a real, random data point, which is pretty typical for many frontend projects nowadays :/

> This doesn't rezally work for native since:
> - there is the cost of compiling this ahead of time, which isn't negligible and with a cost structure unlike Javascript (in the best case, code has to be compiled in and then stripped by linker, very slow)

I know, I know. We use TypeScript, so our code and many of our dependencies actually need to be compiled, before the program can be executed. For example, running a project after git clone takes between 1-10mins. (When it's more than 2-3 mins it's actually because of C/C++. Many nodejs packages are actually C/C++ code with JS bindings, which needs to be compiled on npm install, because in the npm world, typically they ship only source code and not binaries. And also because before Node didn't have a stable ABI, so if you wanted to support Node.js versions released in the past several years, you would have to ship 20 different binaries of the same library :D)
But that's just the first time, because TypeScript and React.js (through webpack) have incremental compilation, so each change takes less than several seconds (in many cases less than 1 sec). By change, I mean the time to typecheck and emit JS code. Of course I'm well aware that this is not a fair comparison to pure "native" languages.

> - native programs have many possibilities to go wrong (deadlocks, threading, memory allocations...) and that differs from project to project ; you cannot enforce any invariant with such a number of libraries. And it's a huge hurdle for "common off the shelf" libraries.

A fork bomb is possible with almost all languages in practice. Every npm package has the ability to run random js code on install, which runs with the privileges of the current user. "A security nightmare" - you say, but in practice it's not that bad, given they have invested much more than us in a process of managing security vulnerabilities: https://docs.npmjs.com/reporting-a-vulnerability-in-an-npm-package https://www.npmjs.com/policies/security
June 25, 2020
On Thursday, 25 June 2020 at 15:14:21 UTC, Paul Backus wrote:
> On Thursday, 25 June 2020 at 14:27:02 UTC, Petar Kirov [ZombineDev] wrote:
>> Technically, phobos is actually already on Dub (https://code.dlang.org/packages/phobos), so in the future nothing could prevent you to continue using it, but you would just get it from there and not from the compiler distribution archive. It's the same code, made by the same people, just with a different distribution model.
>
> Aside: it seems like Dub is not properly picking up the tagged releases, and only has '~master' and '~stable' available. Fixing this would make using Phobos via dub a much more attractive proposition.

I agree, this would at least allow people pick an exact version. (Of course, given that Phobos doesn't follow SemVer, every minor release (2.xx.0) could introduce a breaking change - a symbol being removed after the 2 year deprecation period.)

But still, the next major problem to solve is actually ensuring that each Phobos release works with several different compiler releases, which as far as I can see is not going to happen anytime soon.
June 25, 2020
On 6/25/20 10:27 AM, Petar Kirov [ZombineDev] wrote:
> The way I see things is that we as a community need to focus on a vetted, well shaped collection of libraries.
> We need to have process where third-party libraries are able to gain broader support and join dlang-community. Also we need dlang-community to have a healthy number of people who actually maintain the code.
> We need the leadership to realize that investing just in dmd, druntime and phobos is necessary, but very insufficient.
> W&A need to stop pretending that Dub doesn't exist. I feel that unless we embrace using code.dlang.org as method of distribution of everything that's currently part of the compiler archive, Dub's limitations will never be addressed and from that the broader community will suffer.

(By the way just to clarify "A" in "W&A" would stand for Atila. I'm commenting as just one in the community.)

I'm a bit confused by this argument. So there is the compiler and standard library, and then there's a community-supported library, which includes a copy of the standard library under an improved versioning schema. It's all legal and encouraged by the generous licensing of the compiler and standard library, both of which Walter and I fought hard in the past to obtain and maintain.

Dub is the perfect place where community leadership can set the tone and organized things as they need to without heavy-handed intervention from above. Several people have criticized "the leadership" for not doing things the way they wanted, so it seems perplexing that leadership intervention is now asked for. What's that nonsense with "pretending dub doesn't exist"? So now they need to take ownership of the community-driven dub as well? Isn't that a bit too much work for a handful of folks in the best of times?

Is money that's been asked for? We don't have enough to do competitive hiring.

I don't think Bjarne or Herb have ever been accused of not contributing to Boost (which they didn't), or asked to invest in it. Boost has been a community-led effort which has had a great influence on C++ development - with the participation of volunteers, not the formal standards committee.

>> Anyway, if we can, we try to stick with Phobos, as long as we don't have a particular problem to solve that needs an external library: recent example, Martin std.io or SumType instead of std.variant ...
> 
> See, some of the problems in Phobos already have a good solution. The problem that we need to address is the trust.
> The more high-quality and well-maintained libraries there are, the less cautious one would need to be before reaching to code.dlang.org.

This is and will always be the case, and it's normal. A standard library can't implement all special purpose libraries, or even a significant fraction. A community-led library is a great thing.
June 25, 2020
On Thursday, 25 June 2020 at 14:27:02 UTC, Petar Kirov [ZombineDev] wrote:
> On Thursday, 25 June 2020 at 12:13:21 UTC, Paolo Invernizzi wrote:
>>
>> I know that we are quite few on this side, but IMHO a well shaped standard library has advantages over a sparse ecosystem of independent library ...
>
> Who says that this is a binary choice? And also, the way you fraise it it's obvious that say Flutter is better choice than React as everything is very tightly integrated and you could build your whole app without reaching for third-party code. Also, it's obvious that in terms of trust, it's better to rely on the diligence of the upsteam development team, than random strangers from the internet.
>
> But what if 90% of the code my team needs has no place in Phobos? We already have to rely heavily on third-party packages. Yes there are a lot of risks and we try to be careful, but we have no other choice. We could spend years reiventing the wheel (high-quality code that already exists in other languages), or we could focus on delivering products to our users based on the already existing ecosystem in other languages. Well, a choice would exist if we had an unlimited budget, but given that we're not as lucky, we can only use other languages for our projects currently.
>
> Technically, phobos is actually already on Dub (https://code.dlang.org/packages/phobos), so in the future nothing could prevent you to continue using it, but you would just get it from there and not from the compiler distribution archive. It's the same code, made by the same people, just with a different distribution model.
>
> The way I see things is that we as a community need to focus on a vetted, well shaped collection of libraries.
> We need to have process where third-party libraries are able to gain broader support and join dlang-community. Also we need dlang-community to have a healthy number of people who actually maintain the code.
> We need the leadership to realize that investing just in dmd, druntime and phobos is necessary, but very insufficient.
> W&A need to stop pretending that Dub doesn't exist. I feel that unless we embrace using code.dlang.org as method of distribution of everything that's currently part of the compiler archive, Dub's limitations will never be addressed and from that the broader community will suffer.
>
>> Anyway, if we can, we try to stick with Phobos, as long as we don't have a particular problem to solve that needs an external library: recent example, Martin std.io or SumType instead of std.variant ...
>
> See, some of the problems in Phobos already have a good solution. The problem that we need to address is the trust.
> The more high-quality and well-maintained libraries there are, the less cautious one would need to be before reaching to code.dlang.org.


Almost the exact reason why i use JavaScript (nodejs) for work and not D. It takes more than a nice language. When some here say they don't use Dub or they don't care about dub packages, I'm wondering what kind of software they are writing.

In my line if work, it's highly dependant of availability of packages to get the job done. Encryption, networking, data manipulation, database connectors, socket, rest, cloud services, etc. Its more like glueing services and APIs together.

Unless you're building compilers and softwares that don't connect to internet, then it makes sense to not be bothered by the lack of certain packages on dub.

This will only happen when the community collaborates, which will only happen when there's a visible leadership push/backing. I meant D conf online has taken forever to happen and I'm not sure if community building is a thing prioritized at all.


I've been policing and trying several packages on dub...some, talking to their maintainers. I want to make this happen. Its the only way.

I rarely use core nodejs packages. Its like npm has got everything. Its what dub needs. Already I'm happy with quite a few dub packages. And there's huge!! room for improvement if only there's a visible official push. Its not always about money.

I'm not sure why we don't even have regular online community meetups. Yeah, there's people who would rather code and feel more productive that way. That's also fine.

Except there's another side about community building. DConf is a Joy to watch. Just the community interaction alone and how ideas are shared. It could happen more often. More often online.


June 25, 2020
On 6/25/20 12:42 PM, aberba wrote:
> This will only happen when the community collaborates, which will only happen when there's a visible leadership push/backing.

This is speculation. What is it based on? Boost was started by Beman Dawes with no backing. I know next to nothing about npm, but a look at their website www.npmjs.com suggests it's a traditional for-profit company with no ties to ECMA or other JavaScript formal leadership bodies.

What concrete forms of said leadership push/backing do you envision?