On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d wrote:
> On 10/25/17 23:57, Jacob Carlborg wrote:
> > I'm more concerned that I don't think we'll manage to implement a complete API and 100% bug free at the first try.
>
> Depends on how one defines first try. Phobos as a pretty solid process for making sure these things are reasonably bug free. And near as I can tell, Phobos is pretty good about accepting bug fixes.

The bigger problem is API bugs. The review process is rigorous enough to weed out a lot of stuff, but the end result is typically an API that we _think_ looks good but usually hasn't seen much real world use (much as its design may be based on real world experience). And if it turns out that the API has problems, it can be very difficult to fix that in a way that doesn't break code. Sometimes, we're able to reasonably do something about it and sometimes not. In theory, std.experimental would mitigate that, and that's where anything that was accepted would go first, but the process for getting new modules into Phobos is very much geared towards getting an API right up front rather than getting something that's close and iterating towards where it ultimately should be.

What would probably be better in general would be to be writing stuff that ends up on code.dlang.org first and gets some real world use there and then look at getting it into Phobos later rather than aiming directly for Phobos. Not only would that likely help lead towards better code being in Phobos, but we'd still get useful stuff even if it didn't make it into Phobos.

- Jonathan M Davis

On Monday, October 23, 2017 12:13:12 Laeeth Isharc via Digitalmars-d wrote:
> How can we generate a static binary ?  I asked about this before, and the response was that it's a bad idea because of security vulns and so on.  True if you are running on a conventional Linux host.  But on the other hand, it's not that great if when the system phobos is upgraded all the D binaries break.  You can write a script using rdmd or dub scripting feature, but that doesn't work for more complex programs.

I believe that it works just fine if you do the linking step yourself, but dmd doesn't properly handle it with -L-static:

https://issues.dlang.org/show_bug.cgi?id=6952

I have no idea how easy it would be to the linking manually with dub.

Alternatively, I believe that you can statically link against any 3rd party libraries that you're using by linking to the .a file rather than by the library name. That won't work for glibc, but much as I'd love to statically link that, it's my understanding that doing so is generally a bad idea. I don't recall the reasons though - and with the increased stability in linux's ABI, I would have thought that it would be far less of an issue than it would have been in the past. Certainly, the concept of being able to statically link a program and then just run it on most any recent linux distro without recompiling it sounds very appealing. Yes, you'll have to recompile it for any security updates to any of your dependencies, whereas that can be avoided with dynamic linking, but depending on what you're doing, that often isn't really an issue, and it can be managed if it is (heck, you're doing basically the same thing when maintaining something like a docker image).

- Jonathan M Davis

On 10/26/17 17:51, Jonathan M Davis wrote:
> On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d wrote:
>> On 10/25/17 23:57, Jacob Carlborg wrote:
>>> I'm more concerned that I don't think we'll manage to implement a
>>> complete API and 100% bug free at the first try.
>>
>> Depends on how one defines first try. Phobos as a pretty solid process
>> for making sure these things are reasonably bug free. And near as I can
>> tell, Phobos is pretty good about accepting bug fixes.
>
> The bigger problem is API bugs. The review process is rigorous enough to
> weed out a lot of stuff, but the end result is typically an API that we
> _think_ looks good but usually hasn't seen much real world use (much as its
> design may be based on real world experience). And if it turns out that the
> API has problems, it can be very difficult to fix that in a way that doesn't
> break code. Sometimes, we're able to reasonably do something about it and
> sometimes not. In theory, std.experimental would mitigate that, and that's
> where anything that was accepted would go first, but the process for getting
> new modules into Phobos is very much geared towards getting an API right up
> front rather than getting something that's close and iterating towards where
> it ultimately should be.
>
> What would probably be better in general would be to be writing stuff that
> ends up on code.dlang.org first and gets some real world use there and then
> look at getting it into Phobos later rather than aiming directly for Phobos.
> Not only would that likely help lead towards better code being in Phobos,
> but we'd still get useful stuff even if it didn't make it into Phobos.
>
> - Jonathan M Davis
>

I understand the concern, however, I can see potential mitigations. For one, steal an API concept from somewhere else. I've had reasonable success so far stealing ADO.NET and the refactoring it into something more idiomatic. Using that as a starting point gave me a pretty good understanding of what was needed and it gave me a prototype API that has been battle-tested.

Has anything from code.dlang.org been pulled into Phobos yet? I'm not aware of anything. code.dlang.org is where Phobos projects go to quietly die in obscurity.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On Thursday, October 26, 2017 19:19:57 Adam Wilson via Digitalmars-d wrote:
> On 10/26/17 17:51, Jonathan M Davis wrote:
> > On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d
wrote:
> >> On 10/25/17 23:57, Jacob Carlborg wrote:
> >>> I'm more concerned that I don't think we'll manage to implement a complete API and 100% bug free at the first try.
> >>
> >> Depends on how one defines first try. Phobos as a pretty solid process for making sure these things are reasonably bug free. And near as I can tell, Phobos is pretty good about accepting bug fixes.
> >
> > The bigger problem is API bugs. The review process is rigorous enough to weed out a lot of stuff, but the end result is typically an API that we _think_ looks good but usually hasn't seen much real world use (much as its design may be based on real world experience). And if it turns out that the API has problems, it can be very difficult to fix that in a way that doesn't break code. Sometimes, we're able to reasonably do something about it and sometimes not. In theory, std.experimental would mitigate that, and that's where anything that was accepted would go first, but the process for getting new modules into Phobos is very much geared towards getting an API right up front rather than getting something that's close and iterating towards where it ultimately should be.
> >
> > What would probably be better in general would be to be writing stuff that ends up on code.dlang.org first and gets some real world use there and then look at getting it into Phobos later rather than aiming directly for Phobos. Not only would that likely help lead towards better code being in Phobos, but we'd still get useful stuff even if it didn't make it into Phobos.
> >
> > - Jonathan M Davis
>
> I understand the concern, however, I can see potential mitigations. For one, steal an API concept from somewhere else. I've had reasonable success so far stealing ADO.NET and the refactoring it into something more idiomatic. Using that as a starting point gave me a pretty good understanding of what was needed and it gave me a prototype API that has been battle-tested.
>
> Has anything from code.dlang.org been pulled into Phobos yet? I'm not aware of anything. code.dlang.org is where Phobos projects go to quietly die in obscurity.

Nothing has been pulled into Phobos from anywhere in a while. The last thing was probably checkedint, and before that it was std.allocator - both of which came from Andrei. ndslice was in std.experimental for a little while but Ilya removed it, because he decided that it wasn't working to have it in Phobos, because he couldn't continue to improve on it there. Beyond that, I think that the last thing was the logger, which has just languished in std.experimental. It's my understanding that it needs some more work, but basically, once we added std.experimental, nothing has made it into Phobos proper. And over the last few years, there haven't been many attempts to get anything into Phobos, so not much has even made it into std.experimental.

There was the GSoC project for a new XML parser, but that project seems to have died after the student got too busy with school after GSoC, and Sonke's std.json replacement didn't finish making its way through the review process, and I think that Sonke has given up on getting it in (if I understand correctly, there was just too much disagreement over what the std.json replacement should look like). Overall, people have just stopped trying to get major stuff into Phobos. New functions get added to existing modules, but pretty much no one seems to want to go through the Phobos review process to get anything into Phobos.

code.dlang.org is where folks put anything that they're doing that they want to make available, and IIRC, the only item from there that's even attempted to make it into Phobos was Sonke's JSON parser. Much as some folks continue to talk about getting some piece of functionality into Phobos, no one is trying if it's anything major.

So, it's not like Phobos projects are going to code.dlang.org to die. In general, they simply aren't even being attempted, and any serious projects that do exist are on code.dlang.org. Some do sit there unfinished (e.g. std_experimental_xml), but largely, the authors just don't want to go to the effort of getting the code into Phobos.

And honestly, in general, at this point, I don't think that I'd want to bother either. It's quite a lot of work to get something through the Phobos review process, and once it's in Phobos, you lose control over it. If I work on something major, I can just put it up on code.dlang.org, and anyone who wants to use it can, and I have full control over what I do with the code base without having to get approval from Andrei or anyone else as to what I do with it. So, unless we're talking about something that practically needs to be in the standard library, I doubt that I'd bother trying even if I wrote the code and was maintaining it. And most stuff really doesn't need to be in the standard library, much as it might be nice.

But even if the goal is to get some of this stuff into Phobos, I still think that we're better off putting it up on code.dlang.org first and getting it battle-tested instead of pushing to get it put directly into Phobos.

- Jonathan M Davis

On 2017-10-26 12:25, Adam Wilson wrote:

> My apologies, something rather the other direction. Instead of forcing compat with vibe.d, going to vibe.d and say: "here is our standard event-loop, it has everything you need, you'll need to use it for all the other goodness to work". I know others can make good arguments about why the vibe event-loop is insufficient, and I'll let them make them. 

My concern is not about the event loop, it's about asynchronous IO. vibe.d needs to use asynchronous IO and I assume that's regardless what kind of event loop implementation is used. Does all the existing database drivers that you want to use support asynchronous IO?

-- 
/Jacob Carlborg

On 10/27/17 00:18, Jacob Carlborg wrote:
> On 2017-10-26 12:25, Adam Wilson wrote:
>
>> My apologies, something rather the other direction. Instead of forcing
>> compat with vibe.d, going to vibe.d and say: "here is our standard
>> event-loop, it has everything you need, you'll need to use it for all
>> the other goodness to work". I know others can make good arguments
>> about why the vibe event-loop is insufficient, and I'll let them make
>> them.
>
> My concern is not about the event loop, it's about asynchronous IO.
> vibe.d needs to use asynchronous IO and I assume that's regardless what
> kind of event loop implementation is used. Does all the existing
> database drivers that you want to use support asynchronous IO?
>

PgSQL/MySQL/MSSQL all do, I think that covers about 90% of usage. IIRC Oracle does as well.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
> I just read the following two week-old comment on the ldc issue tracker, when someone tried to run D on Alpine linux:
>
> "For now everything works(?) but I think the process could be improved.. Would be really cool to have LDC easily building alpine containers + static D binaries for microservice and tooling development. I'm pretty tired of reading Go code :)"
> https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
>
> It strikes me that microservices are a great way for new programming languages like D to get tried and gain some uptake, but that D might not be that easy to deploy to that scenario yet.
>
Its the future. Netflix, Google, Facebook, etc. all have open source tools around microservices. Its currently ruled by JavaScript > Go > Java. JavaScript being the leader.

They have these in common:
1. Easy to deploy their code in docker containers including alpine Linux.
2. They have APIs for major cloud services. Both official and third-party.
3. Good support for networking. HTTP, Websockets, IPC*, etc. Mostly HTTP.

That's it. The major advantages besides the hype.

Lets see about D based on these requirements.
1. Kind of. There are dmd-ldc-dub docker images on docker hub. One by sociomantic. None using Alpine Linux as base though. Most people prefer alpine cus its lightweight (not a requirement).
2. Nope. Official APIs comes when there's mass adoption for that language and has good ROI. No complete and production ready cloud services API that I know of. Seems D users in that area roll out something on their own for private use with features they need. But most cloud PaaS provide support for custom runtimes which D has one for Heroku. Major candidate is Google AppEngine.
3. Phobos doesn't have a D HTTP API. Its sad but we have "requests" at code.dlang.org which works. We have vibe.d for http servers (aka nodejs of D) and seems popular based on threads about it and downloads.

> So this is a question for those deploying microservices, as I'm not in that field, what can the D devs do to make it as easy as possible to get D microservices up and running, make some Docker and Alpine containers with ldc/dub/vibe.d preinstalled publicly available?  What else, what kinds of libraries do you normally use?

There several database APIs at cods.dlang.org. I don't know why some complain about no db APIs.

>
> This is a niche that D and all newer languages should target.  How do we do it?
Good question.

On Saturday, 28 October 2017 at 14:55:25 UTC, aberba wrote:
> On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
>> I just read the following two week-old comment on the ldc issue tracker, when someone tried to run D on Alpine linux:
>>
>> "For now everything works(?) but I think the process could be improved.. Would be really cool to have LDC easily building alpine containers + static D binaries for microservice and tooling development. I'm pretty tired of reading Go code :)"
>> https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
>>
>> It strikes me that microservices are a great way for new programming languages like D to get tried and gain some uptake, but that D might not be that easy to deploy to that scenario yet.
>>
> Its the future.

Highly doubt that. It really depend on the scale of your operations.

> Netflix, Google, Facebook, etc. all have open source tools around microservices. Its currently ruled by JavaScript > Go > Java. JavaScript being the leader.
>
> They have these in common:
> 1. Easy to deploy their code in docker containers including alpine Linux.

Interestingly while Docker may seem all the rage in startups I find its use limited to test environments in the real world.

Also Java fat jars were super easy to deploy ages before docker. They are also a great deal smaller.

> 2. They have APIs for major cloud services. Both official and third-party.
> 3. Good support for networking. HTTP, Websockets, IPC*, etc. Mostly HTTP.
>

Honestly APIs these days can be written in anything that is able to put together a few HTTP responses. Unless you are doing serious work like:
- DBs
- Search engines
- ML pipelines
- Real-time event processing systems
....

Any semimodern language/technology with a several hosts can manage to saturate 1Gbit link. Some take a certain amount of tuning others work out of the box. If you go for 40gbit/s it may be important to choose the right technology otherwise it’s all a matter of taste.

On Monday, 23 October 2017 at 18:18:28 UTC, Jacob Carlborg wrote:
> On 2017-10-23 17:35, Joakim wrote:
>
>> I'm sure someone could put these together if the above stuff worked.  The question is who's interested in volunteering to help put this all together?
>
> Yeah, exactly. My plate if already full and all this is lower down on the priority list.

Yuxuan Shui has ported druntime to Musl over the last couple months:

https://github.com/dlang/druntime/pulls?q=is%3Apr+author%3Ayshui+is%3Aclosed

With his changes, all druntime unit tests pass, now only a few asserts tripping in the additional tests for shared libraries.  I added a couple more tweaks to get all but one of the Phobos unit tests passing too.

With a last tiny patch to add a command-line flag to dmd to easily use this port- already added to the latest ldc 1.8 beta, though it doesn't have all the druntime support: https://github.com/ldc-developers/ldc/pull/2373 - anyone can use D with Alpine/Musl containers for microservices.

It would be nice if anyone will rewrite Musl to betterC :)