Great work Etienne!

will libasync make it into phobos?

On Wednesday, 24 September 2014 at 13:13:34 UTC, Etienne wrote:
> It's finally here: https://github.com/etcimon/libasync
>
> We all know how event loops are the foundation of more popular libraries Qt and Nodejs.. we now have a natively compiling async library entirely written in D.
>
> This event library was tested on Win32, Linux x64, Mac OS x64, with DMD 2.066, offers the more low-level async objects: timers, file i/o, dns resolver, tcp, udp, listeners, signals (cross-thread), notifications (same thread), and more recently (and with great efforts for implementing with OS X / BSD) a directory watcher.
>
> e.g. You can run a timer with:
>
> 	import std.datetime; import std.stdio; import libasync.all;
> 	EventLoop evl = new EventLoop;
> 	auto timer = new AsyncTimer(evl);
> 	timer.duration(2.seconds).periodic().run({ writeln("Another 2 seconds have passed"); });
> 	while(evl.loop()) continue;
>
> The tests may be most revealing: https://github.com/etcimon/libasync/blob/master/source/libasync/test.d
>
> A (lightly tested) vibe.d driver using all those async objects is also available and currently ongoing a pull request:
>
> https://github.com/etcimon/vibe.d/tree/native-events
>
> The incentive was to make vibe.d compile in completely native D, I'm now moving onto a botan C++ => D wrapper for it, I plan on moving objects to D over the years until the TLS library can be completely native. I thank Walter for the efforts on extern(C++)
>
> Finally, I release this on the basis of an MIT license, looking forward to seeing our community flourishing with yet more native libraries. Code on

On 24/09/14 22:09, Etienne wrote:

> I've thought about it but it isn't compatible with other BSD platforms
> and has no docs about using it with kqueue, it ended up looking more
> complicated and unstable because I could read complaints everywhere I
> looked. It ended up putting me in a direction where I needed separate
> threads with their own event loops and communication through signals,
> which didn't fit in with what I was used to doing from what the other
> platforms offered.

The only complains I've seen with FSEvents is that it doesn't support notifying about file changes, only directory changes. But that has been fixed in OS X 10.7 so that complain is moot.

> I found the solution with kqueue's vnode, which acts like inotify. It
> doesn't have recursion nor file modification events when watching
> folders so individual files had to be watched and the directories
> checked against cache data every time an event came in.

That sounds like a quite big limitation.

-- 
/Jacob Carlborg

On 2014-09-25 2:51 AM, Jacob Carlborg wrote:
> The only complains I've seen with FSEvents is that it doesn't support
> notifying about file changes, only directory changes. But that has been
> fixed in OS X 10.7 so that complain is moot.

Good, and according to the stats I've seen, 95% of users are on the 2 most recent OS X versions!

>
>> I found the solution with kqueue's vnode, which acts like inotify. It
>> doesn't have recursion nor file modification events when watching
>> folders so individual files had to be watched and the directories
>> checked against cache data every time an event came in.
>
> That sounds like a quite big limitation.
>

It is, but I managed to get around it. Although I haven't yet implemented the renaming/move operation detection. I might as well plan on writing the FSEvent implementation for Mac OS X and state some limitations with FreeBSD in that case =) Thanks for the hint!

On 09/24/2014 06:30 PM, Etienne wrote:
>  > This is fantastic! We really need something like this in a Facebook
>  > project.
>
> That's pleasing to hear, although I'm pretty sure Facebook is far from
> being the only organization who will benefit from this ;)

Who doesn't need something like this?

>  > Would be appropriate for you to consider making a bid for adding it
> to Phobos?
>
> Thanks for the invitation, I'll start working on a phobos fork with this
> added into std.async and re-licensed to Boost. Would that be an
> appropriate namespace?

I still have to try out the library, but I'd like to see that.

One thing that always bothers me with async libraries is that now every IO class has it async cousin, so there is Socket and AsyncSocket, resolveDNS and asyncResolveDNS.
With Fibers and a Scheduler it's actually possible to present the same API to asynchronous and synchronous code. I'd really like to see this at some point, but a cross-platform event loop in phobos is a great first step.

On Thursday, 25 September 2014 at 03:29:03 UTC, Zhao Puming wrote:
> Great work Etienne!
>
> will libasync make it into phobos?

I certainly hope so.  We need async functionality if we're to ever have a decent socket package in Phobos.

On Thu, Sep 25, 2014 at 11:48:29PM +0000, Sean Kelly via Digitalmars-d wrote:
> On Thursday, 25 September 2014 at 03:29:03 UTC, Zhao Puming wrote:
> >Great work Etienne!
> >
> >will libasync make it into phobos?
> 
> I certainly hope so.  We need async functionality if we're to ever have a decent socket package in Phobos.

Adam has also written an event loop (arsd.eventloop), which also has timers and a signals-and-slots subsystem:

	https://github.com/adamdruppe/arsd/blob/master/eventloop.d

I'm not sure if it supports multithreading, but its event registration API is very cool. I think we should pick the best of both projects for inclusion in Phobos.


T

-- 
Let X be the set not defined by this sentence...

On 2014-09-25 19:34, Martin Nowak wrote:
> One thing that always bothers me with async libraries is that now every
> IO class has it async cousin, so there is Socket and AsyncSocket,
> resolveDNS and asyncResolveDNS.
> With Fibers and a Scheduler it's actually possible to present the same
> API to asynchronous and synchronous code. I'd really like to see this at
> some point, but a cross-platform event loop in phobos is a great first
> step.

Exactly, though I'm pretty sure this could be solved by moving vibe core to std.vibe, and then adding it as a dependency for std.socket and std.concurrency. It doesn't feel right having the future/promise aside of fibers with a scheduler, it needs to be done correctly from the start.

On 9/24/14, Etienne via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> It's finally here: https://github.com/etcimon/libasync

Small nitpick with the spelling: asynchroneous => asynchronous. I personally don't care about spelling but many people tend to (unfortunately) look the other way when they find typos.

Btw, have you had a look at http://wiki.dlang.org/Event_system ? I'm just wondering how much of that idea page / proposal is covered.

Anyway, this is a solid initiative. I look forward to seeing it in Phobos!

On 2014-09-26 5:29 AM, Andrej Mitrovic via Digitalmars-d wrote:
> Small nitpick with the spelling: asynchroneous => asynchronous. I
> personally don't care about spelling but many people tend to
> (unfortunately) look the other way when they find typos.

Hm, my french messed with my mind on that one :-P we say asynchrone or asynchroné

> Btw, have you had a look at http://wiki.dlang.org/Event_system ? I'm
> just wondering how much of that idea page / proposal is covered.
>
> Anyway, this is a solid initiative. I look forward to seeing it in Phobos!

I originally developed it with vibe.d's usage in mind which implicitely forced me to cover most of this list, but currently it's lacking AsyncObject which would allow to monitor file descriptors and cover a good chunk of this list. I'm sure it will be a very useful resource for upcoming decisions.

On Wed, 24 Sep 2014 06:13:31 -0700, Etienne <etcimon@gmail.com> wrote:

> It's finally here: https://github.com/etcimon/libasync
>
> We all know how event loops are the foundation of more popular libraries Qt and Nodejs.. we now have a natively compiling async library entirely written in D.
>
> This event library was tested on Win32, Linux x64, Mac OS x64, with DMD 2.066, offers the more low-level async objects: timers, file i/o, dns resolver, tcp, udp, listeners, signals (cross-thread), notifications (same thread), and more recently (and with great efforts for implementing with OS X / BSD) a directory watcher.
>
> e.g. You can run a timer with:
>
> 	import std.datetime; import std.stdio; import libasync.all;
> 	EventLoop evl = new EventLoop;
> 	auto timer = new AsyncTimer(evl);
> 	timer.duration(2.seconds).periodic().run({ writeln("Another 2 seconds have passed"); });
> 	while(evl.loop()) continue;
>
> The tests may be most revealing: https://github.com/etcimon/libasync/blob/master/source/libasync/test.d
>
> A (lightly tested) vibe.d driver using all those async objects is also available and currently ongoing a pull request:
>
> https://github.com/etcimon/vibe.d/tree/native-events
>
> The incentive was to make vibe.d compile in completely native D, I'm now moving onto a botan C++ => D wrapper for it, I plan on moving objects to D over the years until the TLS library can be completely native. I thank Walter for the efforts on extern(C++)
>
> Finally, I release this on the basis of an MIT license, looking forward to seeing our community flourishing with yet more native libraries. Code on

You mentioned Botan. I already have a C++ => D Wrapper project going over here: https://github.com/ellipticbit/titanium

I am working out a bug where the memory corrupts itself when passing data back to D but it works and most of the leg-work is done. And I am definitely open to pull-requests.

-- 
Adam Wilson
GitHub/IRC: LightBender
Aurora Project Coordinator