On 2017-10-23 14:17, Laeeth Isharc wrote:

> Can you elaborate on how the TLS implementation needs to be changed?

# echo 'void main() {}' > main.d && dmd -c main.d && gcc main.o -o main -m64 -static -L/root/.dvm/compilers/dmd-2.076.1/linux/bin/../lib64 -Xlinker -Bstatic -lphobos2  -lpthread -lm -lrt -ldl

/root/.dvm/compilers/dmd-2.076.1/linux/bin/../lib64/libphobos2.a(sections_elf_shared_782_420.o): In function `_D2rt19sections_elf_shared11getTLSRangeFNbNimmZAv':
src/rt/sections_elf_shared.d:(.text._D2rt19sections_elf_shared11getTLSRangeFNbNimmZAv[_D2rt19sections_elf_shared11getTLSRangeFNbNimmZAv]+0x38): undefined reference to `__tls_get_addr'
collect2: error: ld returned 1 exit status

I need to manually invoke GCC to link because DMD will pass "-Xlinker --export-dynamic -Xlinker -Bdynamic" and I need to add "-static".

> If someone wanted to work on rabbit bindings/wrapper, I might be open to sponsoring that.  I'm not in love with Rabbit (one node uses more than 40% of memory so the node goes down, taking the cluster with it.  really?), but we use it currently.
> 
> I made a start on bindings for librabbitmq here:
> https://github.com/kaleidicassociates/rabbitmq-d

Is that compatible with vibe.d?

-- 
/Jacob Carlborg

On 10/23/17 05:08, Jacob Carlborg wrote:
> * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
> compatible with vibe.d
> * Database driver abstraction on top of the above drivers, perhaps some
> lightweight ORM library

I've been looking pretty extensively at these two items recently.

If the database drivers are compatible with Vibe.d AND we wish to provide a common abstraction layer for them (presumably via Phobos) then order for the abstraction layer to aware of the whether the driver is making a blocking or non-blocking call we must include Vibe.D in the abstraction layer. Ergo, we must include at least the vibe-core package in Phobos, or more preferably, DRT.

I had heard noises about that a few months ago. Anything happening on that front?

An event loop is a key piece of a project (Async/Await) that I want to work on, and having it in DRuntime would make that project fantastically simpler. IMHO, the bulk of the time required is in getting an event loop into DRT, the rest is a LOT of relatively straightforward compiler lowering.

IMHO, DRT is in significant need of an event loop system. This would allow us to simplify a large number of problems (Async/Await, GUI's, IO, etc). As near as I can tell, the problem isn't so much doing the work, but getting the required sign-off's for inclusion into DRT.

Another problem that I've been made aware of is that vibe-core may not be ideal in certain situations. As this would be landed in DRT itself this would obviously need to be addressed.

What would the appetite be for working together to come up with a reasonably generic event loop for DRT that vibe and other systems could then leverage?

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On 23/10/2017 11:02 PM, Adam Wilson wrote:
> On 10/23/17 05:08, Jacob Carlborg wrote:
>> * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
>> compatible with vibe.d
>> * Database driver abstraction on top of the above drivers, perhaps some
>> lightweight ORM library
> 
> I've been looking pretty extensively at these two items recently.
> 
> If the database drivers are compatible with Vibe.d AND we wish to provide a common abstraction layer for them (presumably via Phobos) then order for the abstraction layer to aware of the whether the driver is making a blocking or non-blocking call we must include Vibe.D in the abstraction layer. Ergo, we must include at least the vibe-core package in Phobos, or more preferably, DRT.
> 
> I had heard noises about that a few months ago. Anything happening on that front?
> 
> An event loop is a key piece of a project (Async/Await) that I want to work on, and having it in DRuntime would make that project fantastically simpler. IMHO, the bulk of the time required is in getting an event loop into DRT, the rest is a LOT of relatively straightforward compiler lowering.
> 
> IMHO, DRT is in significant need of an event loop system. This would allow us to simplify a large number of problems (Async/Await, GUI's, IO, etc). As near as I can tell, the problem isn't so much doing the work, but getting the required sign-off's for inclusion into DRT.
> 
> Another problem that I've been made aware of is that vibe-core may not be ideal in certain situations. As this would be landed in DRT itself this would obviously need to be addressed.
> 
> What would the appetite be for working together to come up with a reasonably generic event loop for DRT that vibe and other systems could then leverage?
> 

*whispers* heyyy, heard about SPEW[0]?

[0] https://github.com/Devisualization/spew/blob/master/src/base/cf/spew/event_loop/defs.d

On 2017-10-24 00:02, Adam Wilson wrote:

> I've been looking pretty extensively at these two items recently.
> 
> If the database drivers are compatible with Vibe.d AND we wish to provide a common abstraction layer for them (presumably via Phobos) then order for the abstraction layer to aware of the whether the driver is making a blocking or non-blocking call we must include Vibe.D in the abstraction layer. Ergo, we must include at least the vibe-core package in Phobos, or more preferably, DRT.

It can be an optional dependency. Looking at ddb [1] it's not that much code that will be different if vibe.d is used or not. It's basically only reading and writing to the socket that is different [2]. Dub provides predefined version for different dependencies, i.e. Have_vibe_d_core.

> I had heard noises about that a few months ago. Anything happening on that front?

No, not as far as I know. Sönke seems really busy recently.

> What would the appetite be for working together to come up with a reasonably generic event loop for DRT that vibe and other systems could then leverage?

I would be really nice to database support directly in the standard library but it's not critical for me. It takes a lot of work with and massive overhead to get something into Phobos. It's also going to be really slow get in updates. I'm not sure if it's worth it.

[1] https://github.com/pszturmaj/ddb
[2] https://github.com/pszturmaj/ddb/blob/master/source/ddb/postgres.d#L189-L246

-- 
/Jacob Carlborg

On 10/23/17 18:51, rikki cattermole wrote:
> On 23/10/2017 11:02 PM, Adam Wilson wrote:
>> On 10/23/17 05:08, Jacob Carlborg wrote:
>>> * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
>>> compatible with vibe.d
>>> * Database driver abstraction on top of the above drivers, perhaps some
>>> lightweight ORM library
>>
>> I've been looking pretty extensively at these two items recently.
>>
>> If the database drivers are compatible with Vibe.d AND we wish to
>> provide a common abstraction layer for them (presumably via Phobos)
>> then order for the abstraction layer to aware of the whether the
>> driver is making a blocking or non-blocking call we must include
>> Vibe.D in the abstraction layer. Ergo, we must include at least the
>> vibe-core package in Phobos, or more preferably, DRT.
>>
>> I had heard noises about that a few months ago. Anything happening on
>> that front?
>>
>> An event loop is a key piece of a project (Async/Await) that I want to
>> work on, and having it in DRuntime would make that project
>> fantastically simpler. IMHO, the bulk of the time required is in
>> getting an event loop into DRT, the rest is a LOT of relatively
>> straightforward compiler lowering.
>>
>> IMHO, DRT is in significant need of an event loop system. This would
>> allow us to simplify a large number of problems (Async/Await, GUI's,
>> IO, etc). As near as I can tell, the problem isn't so much doing the
>> work, but getting the required sign-off's for inclusion into DRT.
>>
>> Another problem that I've been made aware of is that vibe-core may not
>> be ideal in certain situations. As this would be landed in DRT itself
>> this would obviously need to be addressed.
>>
>> What would the appetite be for working together to come up with a
>> reasonably generic event loop for DRT that vibe and other systems
>> could then leverage?
>>
>
> *whispers* heyyy, heard about SPEW[0]?
>
> [0]
> https://github.com/Devisualization/spew/blob/master/src/base/cf/spew/event_loop/defs.d
>

You've mentioned it a few times. ;-)

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On 10/23/17 23:29, Jacob Carlborg wrote:
> On 2017-10-24 00:02, Adam Wilson wrote:
>
>> I've been looking pretty extensively at these two items recently.
>>
>> If the database drivers are compatible with Vibe.d AND we wish to
>> provide a common abstraction layer for them (presumably via Phobos)
>> then order for the abstraction layer to aware of the whether the
>> driver is making a blocking or non-blocking call we must include
>> Vibe.D in the abstraction layer. Ergo, we must include at least the
>> vibe-core package in Phobos, or more preferably, DRT.
>
> It can be an optional dependency. Looking at ddb [1] it's not that much
> code that will be different if vibe.d is used or not. It's basically
> only reading and writing to the socket that is different [2]. Dub
> provides predefined version for different dependencies, i.e.
> Have_vibe_d_core.
>

This of course makes the assumption that we clean-room our own protocol implementations which I am entirely against. Better to use what already exists.

>> I had heard noises about that a few months ago. Anything happening on
>> that front?
>
> No, not as far as I know. Sönke seems really busy recently.
>

That is what I was afraid of.

>> What would the appetite be for working together to come up with a
>> reasonably generic event loop for DRT that vibe and other systems
>> could then leverage?
>
> I would be really nice to database support directly in the standard
> library but it's not critical for me. It takes a lot of work with and
> massive overhead to get something into Phobos. It's also going to be
> really slow get in updates. I'm not sure if it's worth it.
>
> [1] https://github.com/pszturmaj/ddb
> [2]
> https://github.com/pszturmaj/ddb/blob/master/source/ddb/postgres.d#L189-L246
>

I actually don't think the slow updates are huge problem as these DB interface libraries are pretty slow to change themselves. For example, ADO.NET didn't change significantly from it's 1.0 release until the arrival of Async/Await in .NET 4.5, which was 10 years later. The biggest addition prior to Async was TVP support and that was tiny comparatively and came in 2005.

The libpq5 interface has barely changed in years. I can't speak to MySQL but IIRC it hasn't changed much either, at least not in a way that would effect the abstraction layer.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On Monday, 23 October 2017 at 12:13:12 UTC, Laeeth Isharc wrote:
> ...
> And on the other hand, for deployment, it's much easier to copy over one binary.  (In a sense it's funny that the question was asked in the context of containers, because containers are kind of an alternative to having a single binary).  When you're properly set-up of course it doesn't matter, but when you're moving towards that, a single binary is much easier.
> ...

A bit OT but this part made me think of [1] of which you may appreciate the insight. Right now I kind of feel like with D we get the downside in size of fat binaries without the upside in portability (although solutions exist for the dedicated and patient ones). This might proove an important drawback in the future... we'll see.

[1]: http://www.smashcompany.com/technology/why-would-anyone-choose-docker-over-fat-binaries

On 2017-10-26 00:53, Adam Wilson wrote:

> This of course makes the assumption that we clean-room our own protocol implementations which I am entirely against. Better to use what already exists.

I'm entirely against anything that is not compatible with vibe.d ;)

> I actually don't think the slow updates are huge problem as these DB interface libraries are pretty slow to change themselves. For example, ADO.NET didn't change significantly from it's 1.0 release until the arrival of Async/Await in .NET 4.5, which was 10 years later. The biggest addition prior to Async was TVP support and that was tiny comparatively and came in 2005.
> 
> The libpq5 interface has barely changed in years. I can't speak to MySQL but IIRC it hasn't changed much either, at least not in a way that would effect the abstraction layer.

I'm more concerned that I don't think we'll manage to implement a complete API and 100% bug free at the first try.

-- 
/Jacob Carlborg

On 10/25/17 23:57, Jacob Carlborg wrote:
> On 2017-10-26 00:53, Adam Wilson wrote:
>
>> This of course makes the assumption that we clean-room our own
>> protocol implementations which I am entirely against. Better to use
>> what already exists.
>
> I'm entirely against anything that is not compatible with vibe.d ;)
>

My apologies, something rather the other direction. Instead of forcing compat with vibe.d, going to vibe.d and say: "here is our standard event-loop, it has everything you need, you'll need to use it for all the other goodness to work". I know others can make good arguments about why the vibe event-loop is insufficient, and I'll let them make them. (Something about not supporting GUI loops, paging Mr. Cattermole). If that is really the case I don't see how being entirely vibe.d compatible and meeting the universal standard requirements of Phobos is possible.

There would need to be a requirements gathering phase so that the community as a whole can bring their use-cases before we dove into code.

>> I actually don't think the slow updates are huge problem as these DB
>> interface libraries are pretty slow to change themselves. For example,
>> ADO.NET didn't change significantly from it's 1.0 release until the
>> arrival of Async/Await in .NET 4.5, which was 10 years later. The
>> biggest addition prior to Async was TVP support and that was tiny
>> comparatively and came in 2005.
>>
>> The libpq5 interface has barely changed in years. I can't speak to
>> MySQL but IIRC it hasn't changed much either, at least not in a way
>> that would effect the abstraction layer.
>
> I'm more concerned that I don't think we'll manage to implement a
> complete API and 100% bug free at the first try.
>

Depends on how one defines first try. Phobos as a pretty solid process for making sure these things are reasonably bug free. And near as I can tell, Phobos is pretty good about accepting bug fixes.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On 26/10/2017 11:25 AM, Adam Wilson wrote:
> On 10/25/17 23:57, Jacob Carlborg wrote:
>> On 2017-10-26 00:53, Adam Wilson wrote:
>>
>>> This of course makes the assumption that we clean-room our own
>>> protocol implementations which I am entirely against. Better to use
>>> what already exists.
>>
>> I'm entirely against anything that is not compatible with vibe.d ;)
>>
> 
> My apologies, something rather the other direction. Instead of forcing compat with vibe.d, going to vibe.d and say: "here is our standard event-loop, it has everything you need, you'll need to use it for all the other goodness to work". I know others can make good arguments about why the vibe event-loop is insufficient, and I'll let them make them. (Something about not supporting GUI loops, paging Mr. Cattermole). If that is really the case I don't see how being entirely vibe.d compatible and meeting the universal standard requirements of Phobos is possible.
> 
> There would need to be a requirements gathering phase so that the community as a whole can bring their use-cases before we dove into code.

The problem isn't the event loop design.
Its a fairly solved problem.

The way vibe.d's works is very specific to their use case (which isn't wrong if you only consider them). You can't 'hook' into it. Which makes it very undesirable for Phobos. Since it won't cover most use cases. Even if the source they are using is compatible with an external GUI event loop (which it should be for Windows from what I've read).

So I wouldn't be starting with vibe.d's event loop model. Quite to the contrary, kill it. Build something that will last throughout the ages for everyone and put this problem to rest.