I am trying to make non blocking web requests to a web service.

vibe.core.concurrency.Future!(UserData)[] futurelist;

// I will make http requests in for loop and push them to futureList
foreach( elem; elemList )
{
        // In makeWebRequest I make the httprequest, parse json and push to my UserData
        auto future = vibe.core.concurrency.async( &elem.makeWebRequest );
        futurelist ~= future;
}


// I want to do call ProcessResponceData for each future which is ready.
while ( futurelist.length > 0 )
{
        futurelist.filter!(a => a.ready()).each!(a=> ProcessResponceData(a.getResult()));

        //!!!!!  Here is my problem
        //!!!!!  I want to remove the futures which are already processed.
        futurelist = futurelist.filter!(a => !a.ready()).array;	
        sleep(10.msecs);

}

I am having troubles while trying to remove the future which I already processed.

futurelist = futurelist.filter!(a => !a.ready()).array;	 results with:

/usr/local/bin/../import/std/array.d(2716,20): Error: can't have array of nothrow @safe void()
/usr/local/bin/../import/std/array.d(2782,46): Error: can't have array of inout nothrow @safe void()
/usr/local/bin/../import/std/array.d(3158,1): Error: template instance std.array.Appender!(Future!(UserData)[]) error instantiating
/usr/local/bin/../import/std/array.d(133,32):        instantiated from here: appender!(Future!(UserData)[])

futurelist = futurelist.filter!(a => !a.ready());	 results with:

 Error: cannot implicitly convert expression (filter(futurelist)) of type FilterResult!(__lambda5, Future!(UserData)[]) to Future!(AnalyzeData)[]

Do you have any suggestion or workaround for my case?
What lesson I should learn from this case? Do you guys have any idea why I am getting those compile errors?


Thanks
Erdem

On Saturday, 28 October 2017 at 13:51:42 UTC, kerdemdemir wrote:
> I am trying to make non blocking web requests to a web service.
>
> vibe.core.concurrency.Future!(UserData)[] futurelist;
>
> // I will make http requests in for loop and push them to futureList
> foreach( elem; elemList )
> {
>         // In makeWebRequest I make the httprequest, parse json and push to my UserData
>         auto future = vibe.core.concurrency.async( &elem.makeWebRequest );
>         futurelist ~= future;
> }
>
>
> // I want to do call ProcessResponceData for each future which is ready.
> while ( futurelist.length > 0 )
> {
>         futurelist.filter!(a => a.ready()).each!(a=> ProcessResponceData(a.getResult()));
>
>         //!!!!!  Here is my problem
>         //!!!!!  I want to remove the futures which are already processed.
>         futurelist = futurelist.filter!(a => !a.ready()).array;	
>         sleep(10.msecs);
>
> }
>
> I am having troubles while trying to remove the future which I already processed.
>
> futurelist = futurelist.filter!(a => !a.ready()).array;	 results with:
>
> /usr/local/bin/../import/std/array.d(2716,20): Error: can't have array of nothrow @safe void()
> /usr/local/bin/../import/std/array.d(2782,46): Error: can't have array of inout nothrow @safe void()
> /usr/local/bin/../import/std/array.d(3158,1): Error: template instance std.array.Appender!(Future!(UserData)[]) error instantiating
> /usr/local/bin/../import/std/array.d(133,32):        instantiated from here: appender!(Future!(UserData)[])
>
> futurelist = futurelist.filter!(a => !a.ready());	 results with:
>
>  Error: cannot implicitly convert expression (filter(futurelist)) of type FilterResult!(__lambda5, Future!(UserData)[]) to Future!(AnalyzeData)[]
>
> Do you have any suggestion or workaround for my case?
> What lesson I should learn from this case? Do you guys have any idea why I am getting those compile errors?
>
>
> Thanks
> Erdem

I'd take a look at why the error message says
`Future!(UserData)[]) to Future!(AnalyzeData)[]`
is AnalyzeData  the type returned by ProcessResponceData?

Alternatively you could use a singly linked list and splice out elements that pass the filter predicate. I think you'd have to roll your own though.

On Sunday, 29 October 2017 at 01:44:37 UTC, Nicholas Wilson wrote:
> Alternatively you could use a singly linked list and splice out elements that pass the filter predicate. I think you'd have to roll your own though.

Something like https://github.com/dlang/phobos/pull/5821

> I'd take a look at why the error message says
> `Future!(UserData)[]) to Future!(AnalyzeData)[]`
> is AnalyzeData  the type returned by ProcessResponceData?
>
> Alternatively you could use a singly linked list and splice out elements that pass the filter predicate. I think you'd have to roll your own though.

I am sorry Wilson I normally in my code UserData is AnalyzeData = UserData but I replace the name to make it more understandable.

For now I solved my problem like

vibe.core.concurrency.Future!(UserData)[] futurelist;
foreach( elem; elemList )
{
    auto future = vibe.core.concurrency.async( &elem.makeWebRequest );
    futurelist ~= future;
}

int maximumSleepTime = 0;
while(futurelist.any!(a => !a.ready())) //--> Blocking until all of them ready
{
    maximumSleepTime++;
    sleep(10.msecs);
    if ( maximumSleepTime  > 2000 )
      return;
}

futurelist.each!(a=> Process(a.getResult()));

Unfortunately that is really bad when I make 250 web requests and only one of them is bad(I mean not ready). And 249 request have to wait for the bad one.

I will take a deeper look to data structure you suggested.

Thanks