Dear,

I try to use the async buffer describe into std.parallelism documentation but my test code core dump!

documentation: https://dlang.org/phobos/std_parallelism.html#.TaskPool.asyncBuf.2

snipptet: https://paste.fedoraproject.org/paste/K5W1O1dLzZ0sPV8NeGztUg

Trace:

$ gdb test
(gdb) r
Starting program: /env/export/v_home/q_unix/jmercier/Projets/wc/test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
nb octets: 10485760
nb iterations: 10240
Done
[New Thread 0x7ffff681f700 (LWP 34340)]
[New Thread 0x7ffff601e700 (LWP 34341)]
[New Thread 0x7ffff581d700 (LWP 34342)]
[New Thread 0x7ffff501c700 (LWP 34343)]
[New Thread 0x7ffff481b700 (LWP 34344)]
[New Thread 0x7fffdffff700 (LWP 34345)]
[New Thread 0x7fffdf7fe700 (LWP 34346)]
[New Thread 0x7fffdeffd700 (LWP 34347)]

Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7fffdeffd700 (LWP 34347)]
0x00007ffff72119f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 libgcc-4.8.5-39.el7.x86_64
(gdb) bt
#0  0x00007ffff72119f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ffff7f2cae0 in core.sync.condition.Condition.wait() ()
   from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libdruntime-ldc-shared.so.88
#2  0x00007ffff7addd06 in std.parallelism.TaskPool.pop() ()
   from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libphobos2-ldc-shared.so.88
#3  0x00007ffff7addbcf in std.parallelism.TaskPool.startWorkLoop() ()
   from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libphobos2-ldc-shared.so.88
#4  0x00007ffff7f2e8bc in thread_entryPoint ()
   from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libdruntime-ldc-shared.so.88
#5  0x00007ffff720de65 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff6a1e88d in clone () from /lib64/libc.so.6


Thanks for your help

Best regards

On 11/07/2019 07:07 AM, bioinfornatics wrote:
> Dear,
>
> I try to use the async buffer describe into std.parallelism
> documentation but my test code core dump!

I admit I don't fully understand the lifetime issues but removing the "code smell" of the modul-level File object solved the issue for me, which requires four changes:

// (1) Comment out the module-level variable
// File file;

// ...

@system
void next(File file, ref ubyte[] buf)
{
  // (2.a) Use the parameter 'file'
  // (2.b) Adjust the length of the buffer
  //       (rawRead may read less than the requested size)
    buf = file.rawRead(buf);
}

// ...

    // (3) Define a local variable
    auto file = File(filePath, "rb");

// ...

    // (4) Use "callables" that use the local 'file':
    auto asyncReader = taskPool.asyncBuf((ref ubyte[] buf) => next(file, buf),
                                         () => file.eof,
                                         bufferSize);

Ali

On Friday, 8 November 2019 at 01:12:37 UTC, Ali Çehreli wrote:
> On 11/07/2019 07:07 AM, bioinfornatics wrote:
> > Dear,
> >
> > I try to use the async buffer describe into std.parallelism
> > documentation but my test code core dump!
>
> I admit I don't fully understand the lifetime issues but removing the "code smell" of the modul-level File object solved the issue for me, which requires four changes:
>
> // (1) Comment out the module-level variable
> // File file;
>
> // ...
>
> @system
> void next(File file, ref ubyte[] buf)
> {
>   // (2.a) Use the parameter 'file'
>   // (2.b) Adjust the length of the buffer
>   //       (rawRead may read less than the requested size)
>     buf = file.rawRead(buf);
> }
>
> // ...
>
>     // (3) Define a local variable
>     auto file = File(filePath, "rb");
>
> // ...
>
>     // (4) Use "callables" that use the local 'file':
>     auto asyncReader = taskPool.asyncBuf((ref ubyte[] buf) => next(file, buf),
>                                          () => file.eof,
>                                          bufferSize);
>
> Ali

Thanks a lot Ali, the error message was understandable to me, and the way you fixed differ from the documentation. Does that means they are a bug ?

Moreover, this snippet is used to evaluate how to parse efficiently versus the wc -l command.

fixed snippet: https://paste.fedoraproject.org/paste/B~PjobIlVXIaJPfMjCcWSA

And I have 2 problems
1/ the executable is at least twice a time slower than wc -l
  I try to increase
    a) the buffer with a multiple of page size (parameter -n)
    b) the number of thread (parameter -t)
2/ the result is not exactly the same as wc -l give wich imply a bug while counting \n inside the buffer.

$ time ./build/wc_File_by_chunks_async_buffer -n 4 -t 8 -i test_100M
1638656 test_100M

real    0m0.106s
user    0m0.101s
sys     0m0.051s

$ time wc -l test_100M
1638400 test_100M

real    0m0.067s
user    0m0.030s
sys     0m0.037s


Thanks again Ali without you It was impossible to me

> the error message was understandable to me,

... the error message was not understandable to me ...

On Friday, 8 November 2019 at 08:58:36 UTC, bioinfornatics wrote:
>> the error message was understandable to me,
>
> ... the error message was not understandable to me ...

I do not have found yet why the line counter is false.
I can tell if the amount to read imply that the last read is not not strictly equal to the buffer then the result is false as if it left less thing to read the result is not shorter than requested buffer size as it is told into the documentation:
https://dlang.org/phobos/std_stdio.html#.File.rawRead

as example I change the code in order to shuffle line content to get the ability to locate the bug

Real last lines are:
1
190121746114132251381321230342516302196252336238211523943272873744285119323293314107
316322221221132661353262123081115418570291330356278322215013742329426
714213310231111593822146521912312869120169289362332157427352432313112226373403123825
6812101511112462691294263101232
90182312212511430133514352114282271133753782360462351124233948222161956731321
11822481725231121323330910521376234322119392811262411335432102108273
463633112212153255811679207


while the code give:
1
190121746114132251381321230342516302196252336238211523943272873744285119323293314107
316322221221132661353262123081115418570291330356278322215013742329426
714213310231111593822146521912312869120169289362332157427352432313112226373403123825
6812101511112462691294263101232
90182312212511430133514352114282271133753782360462351124233948222161956731321
11822481725231121323330910521376234322119392811262411335432102108273
4636331122121532558116792071512
1553203330299234738282167126033321272154232912111312416461868182323242
111932223509162312223104310231321116573254736811479599513441112318312221230321154
11363210249282132717260536372386522748224512596323581311
121932131303243861212327470532121636029110222323531121763
2499315312114672213683218207122451351984311032612832096363463812
....

I think we see the buffer content which is reused. so A big part of last read come from the previous chunk

On Friday, 8 November 2019 at 14:32:25 UTC, bioinfornatics wrote:
> On Friday, 8 November 2019 at 08:58:36 UTC, bioinfornatics wrote:
>> [...]
>
> I do not have found yet why the line counter is false.
> I can tell if the amount to read imply that the last read is not not strictly equal to the buffer then the result is false as if it left less thing to read the result is not shorter than requested buffer size as it is told into the documentation:
> https://dlang.org/phobos/std_stdio.html#.File.rawRead
>
> [...]


Updating the code with: (ref ubyte[] buffer){ buffer = file.rawRead(buffer); }

fix the problem.