Thread overview | ||||||||
---|---|---|---|---|---|---|---|---|
|
November 07, 2019 Using an async buffer | ||||
---|---|---|---|---|
| ||||
Dear, I try to use the async buffer describe into std.parallelism documentation but my test code core dump! documentation: https://dlang.org/phobos/std_parallelism.html#.TaskPool.asyncBuf.2 snipptet: https://paste.fedoraproject.org/paste/K5W1O1dLzZ0sPV8NeGztUg Trace: $ gdb test (gdb) r Starting program: /env/export/v_home/q_unix/jmercier/Projets/wc/test [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". nb octets: 10485760 nb iterations: 10240 Done [New Thread 0x7ffff681f700 (LWP 34340)] [New Thread 0x7ffff601e700 (LWP 34341)] [New Thread 0x7ffff581d700 (LWP 34342)] [New Thread 0x7ffff501c700 (LWP 34343)] [New Thread 0x7ffff481b700 (LWP 34344)] [New Thread 0x7fffdffff700 (LWP 34345)] [New Thread 0x7fffdf7fe700 (LWP 34346)] [New Thread 0x7fffdeffd700 (LWP 34347)] Program received signal SIGUSR1, User defined signal 1. [Switching to Thread 0x7fffdeffd700 (LWP 34347)] 0x00007ffff72119f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 (gdb) bt #0 0x00007ffff72119f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffff7f2cae0 in core.sync.condition.Condition.wait() () from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libdruntime-ldc-shared.so.88 #2 0x00007ffff7addd06 in std.parallelism.TaskPool.pop() () from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libphobos2-ldc-shared.so.88 #3 0x00007ffff7addbcf in std.parallelism.TaskPool.startWorkLoop() () from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libphobos2-ldc-shared.so.88 #4 0x00007ffff7f2e8bc in thread_entryPoint () from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libdruntime-ldc-shared.so.88 #5 0x00007ffff720de65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffff6a1e88d in clone () from /lib64/libc.so.6 Thanks for your help Best regards |
November 07, 2019 Re: Using an async buffer | ||||
---|---|---|---|---|
| ||||
Posted in reply to bioinfornatics | On 11/07/2019 07:07 AM, bioinfornatics wrote: > Dear, > > I try to use the async buffer describe into std.parallelism > documentation but my test code core dump! I admit I don't fully understand the lifetime issues but removing the "code smell" of the modul-level File object solved the issue for me, which requires four changes: // (1) Comment out the module-level variable // File file; // ... @system void next(File file, ref ubyte[] buf) { // (2.a) Use the parameter 'file' // (2.b) Adjust the length of the buffer // (rawRead may read less than the requested size) buf = file.rawRead(buf); } // ... // (3) Define a local variable auto file = File(filePath, "rb"); // ... // (4) Use "callables" that use the local 'file': auto asyncReader = taskPool.asyncBuf((ref ubyte[] buf) => next(file, buf), () => file.eof, bufferSize); Ali |
November 08, 2019 Re: Using an async buffer | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ali Çehreli | On Friday, 8 November 2019 at 01:12:37 UTC, Ali Çehreli wrote: > On 11/07/2019 07:07 AM, bioinfornatics wrote: > > Dear, > > > > I try to use the async buffer describe into std.parallelism > > documentation but my test code core dump! > > I admit I don't fully understand the lifetime issues but removing the "code smell" of the modul-level File object solved the issue for me, which requires four changes: > > // (1) Comment out the module-level variable > // File file; > > // ... > > @system > void next(File file, ref ubyte[] buf) > { > // (2.a) Use the parameter 'file' > // (2.b) Adjust the length of the buffer > // (rawRead may read less than the requested size) > buf = file.rawRead(buf); > } > > // ... > > // (3) Define a local variable > auto file = File(filePath, "rb"); > > // ... > > // (4) Use "callables" that use the local 'file': > auto asyncReader = taskPool.asyncBuf((ref ubyte[] buf) => next(file, buf), > () => file.eof, > bufferSize); > > Ali Thanks a lot Ali, the error message was understandable to me, and the way you fixed differ from the documentation. Does that means they are a bug ? Moreover, this snippet is used to evaluate how to parse efficiently versus the wc -l command. fixed snippet: https://paste.fedoraproject.org/paste/B~PjobIlVXIaJPfMjCcWSA And I have 2 problems 1/ the executable is at least twice a time slower than wc -l I try to increase a) the buffer with a multiple of page size (parameter -n) b) the number of thread (parameter -t) 2/ the result is not exactly the same as wc -l give wich imply a bug while counting \n inside the buffer. $ time ./build/wc_File_by_chunks_async_buffer -n 4 -t 8 -i test_100M 1638656 test_100M real 0m0.106s user 0m0.101s sys 0m0.051s $ time wc -l test_100M 1638400 test_100M real 0m0.067s user 0m0.030s sys 0m0.037s Thanks again Ali without you It was impossible to me |
November 08, 2019 Re: Using an async buffer | ||||
---|---|---|---|---|
| ||||
Posted in reply to bioinfornatics | > the error message was understandable to me,
... the error message was not understandable to me ...
|
November 08, 2019 Re: Using an async buffer | ||||
---|---|---|---|---|
| ||||
Posted in reply to bioinfornatics | On Friday, 8 November 2019 at 08:58:36 UTC, bioinfornatics wrote: >> the error message was understandable to me, > > ... the error message was not understandable to me ... I do not have found yet why the line counter is false. I can tell if the amount to read imply that the last read is not not strictly equal to the buffer then the result is false as if it left less thing to read the result is not shorter than requested buffer size as it is told into the documentation: https://dlang.org/phobos/std_stdio.html#.File.rawRead as example I change the code in order to shuffle line content to get the ability to locate the bug Real last lines are: 1 190121746114132251381321230342516302196252336238211523943272873744285119323293314107 316322221221132661353262123081115418570291330356278322215013742329426 714213310231111593822146521912312869120169289362332157427352432313112226373403123825 6812101511112462691294263101232 90182312212511430133514352114282271133753782360462351124233948222161956731321 11822481725231121323330910521376234322119392811262411335432102108273 463633112212153255811679207 while the code give: 1 190121746114132251381321230342516302196252336238211523943272873744285119323293314107 316322221221132661353262123081115418570291330356278322215013742329426 714213310231111593822146521912312869120169289362332157427352432313112226373403123825 6812101511112462691294263101232 90182312212511430133514352114282271133753782360462351124233948222161956731321 11822481725231121323330910521376234322119392811262411335432102108273 4636331122121532558116792071512 1553203330299234738282167126033321272154232912111312416461868182323242 111932223509162312223104310231321116573254736811479599513441112318312221230321154 11363210249282132717260536372386522748224512596323581311 121932131303243861212327470532121636029110222323531121763 2499315312114672213683218207122451351984311032612832096363463812 .... I think we see the buffer content which is reused. so A big part of last read come from the previous chunk |
November 08, 2019 Re: Using an async buffer | ||||
---|---|---|---|---|
| ||||
Posted in reply to bioinfornatics | On Friday, 8 November 2019 at 14:32:25 UTC, bioinfornatics wrote:
> On Friday, 8 November 2019 at 08:58:36 UTC, bioinfornatics wrote:
>> [...]
>
> I do not have found yet why the line counter is false.
> I can tell if the amount to read imply that the last read is not not strictly equal to the buffer then the result is false as if it left less thing to read the result is not shorter than requested buffer size as it is told into the documentation:
> https://dlang.org/phobos/std_stdio.html#.File.rawRead
>
> [...]
Updating the code with: (ref ubyte[] buffer){ buffer = file.rawRead(buffer); }
fix the problem.
|
Copyright © 1999-2021 by the D Language Foundation