Right now we can nicely stream an URL as an input range. A great extension would be to fetch several URLs at once. When accessing r.front for that range, the user gets a pair of URL and data chunk.

Of course the point is to make all these fetches work simultaneously. Inside, the range would e.g. use asynchronous curl.


Andrei

On Wednesday, 16 December 2015 at 15:54:09 UTC, Andrei Alexandrescu wrote:
> Right now we can nicely stream an URL as an input range. A great extension would be to fetch several URLs at once. When accessing r.front for that range, the user gets a pair of URL and data chunk.
>
> Of course the point is to make all these fetches work simultaneously. Inside, the range would e.g. use asynchronous curl.
>
>
> Andrei

Whenever I want to do that I use vibe.d and my observe library (based on RX observables) and the code would look a bit like this:

urls
	.observe()
	.zip(localPaths)
	.fork(numParallel)
	.concatMap!((part)
	{
		return downloadFile(part[0],part[1]).retry(3)
	})
	.subscribe(
		(p){writeln("File %s downloaded to %s",p[0],p[1]);},
		(e){writeln("Error ",e);},
		(){writeln("Completed!");}
	);

Here observe() makes an observable out of any range; fork() uses vibe.d's Fibers to achieve concurrency; downloadFile() returns another observable that gets concatMap'ped; and retry does what you think it does. Also, errors get propagated without throws and catches.

I think what I am trying to say is that a lot of stuff is already available on code.dlang.org, just not in phobos. Which begs the question, should it be? And if it does, shouldn't you rather consider merging vibe.d with phobos? Or parts of it?

On 12/16/2015 02:12 PM, Sebastiaan Koppe wrote:
> I think what I am trying to say is that a lot of stuff is already
> available on code.dlang.org, just not in phobos. Which begs the
> question, should it be? And if it does, shouldn't you rather consider
> merging vibe.d with phobos? Or parts of it?

Generally phobos should be compelling in and of itself and grow in breadth and depth over time.

I think a facility for asynchronous HTTP fetching would be a great addition. Your code looks pretty rad and well integrated with the rest of Phobos. I encourage you to pursue merging it to Phobos.


Andrei

On Wednesday, 16 December 2015 at 19:32:36 UTC, Andrei Alexandrescu wrote:
> On 12/16/2015 02:12 PM, Sebastiaan Koppe wrote:
>> I think what I am trying to say is that a lot of stuff is already
>> available on code.dlang.org, just not in phobos. Which begs the
>> question, should it be? And if it does, shouldn't you rather consider
>> merging vibe.d with phobos? Or parts of it?
>
> Generally phobos should be compelling in and of itself and grow in breadth and depth over time.

Well it can never encompass all the library code people write, it having a way lower threshold. In that regard I suppose Phobos does well when it tries to support libraries by developing good low/mid-level code. Stuff like std.algorithms, collections and ndslice. But do you really want to put a ODBC client in there, as an example? Or a grayscale filter?

> I think a facility for asynchronous HTTP fetching would be a great addition. Your code looks pretty rad and well integrated with the rest of Phobos. I encourage you to pursue merging it to Phobos.

Thanks. It is more an async/observable library than anything else. And it depends on vibe.d. When I get rid of the vibe.d dependency, or make it optional, I might consider prepping it up for inclusion.

On Wednesday, 16 December 2015 at 20:09:23 UTC, Sebastiaan Koppe wrote:
> But do you really want to put a ODBC client in there, as an example?

Not sure if you're being serious making a joke, because that exists and Andrei was the one who made it http://dlang.org/phobos/etc_c_odbc_sql.html

On 12/16/2015 03:09 PM, Sebastiaan Koppe wrote:
> But do you really want to put a ODBC client in there, as an example? Or
> a grayscale filter?

ODBC maybe, grayscale filter probably not.

I should add I've argued for including some of the core vibe.d stuff in Phobos. Sadly nobody is championing such a project for the time being.


Andrei

On Wednesday, 16 December 2015 at 21:00:55 UTC, Andrei Alexandrescu wrote:
> I should add I've argued for including some of the core vibe.d stuff in Phobos. Sadly nobody is championing such a project for the time being.
>
>
> Andrei

Would that include its stream stuff? We've been needing a std.stream replacement for a long time.

On Wednesday, 16 December 2015 at 21:00:55 UTC, Andrei Alexandrescu wrote:
> On 12/16/2015 03:09 PM, Sebastiaan Koppe wrote:
>> But do you really want to put a ODBC client in there, as an example? Or
>> a grayscale filter?
>
> ODBC maybe, grayscale filter probably not.
>
> I should add I've argued for including some of the core vibe.d stuff in Phobos. Sadly nobody is championing such a project for the time being.
>
>
> Andrei

That is a good idea. What would a good approach for such a project be? Many small pr's that move core functions to phobos piece by piece.