On Monday, 5 March 2012 at 12:48:54 UTC, Andrei Alexandrescu wrote:
>
> Sounds great! Next step, should you be interested, is to create a pull request for phobos so we can integrate your code within.
>
> Andrei

I considered that.  I suppose the wildArgv code could go in std.path, and the file operations into std.file. and the pull requests against those files.

I haven't followed the discussions closely enough to know what are the rules/politics about adding another std library import into those.  It would require adding import of std.parallelism into std.file.

Am 14.02.2012 05:58, schrieb Jay Norwood:
> Attached is the source for a small parallel app that copies a source folder to a destination.  It creates the directory structure first using the breadth ordering, then uses a parallel foreach loop with the taskPool to copy all the regular files in parallel.  On my corei7, this copied a 1.5GB folder with around 36K entries to a destination in about 11.5 secs (src and dest on the same ssd drive).  This was about a second better than robocopy, which is the fastest alternative I could find.   The regular win7-64 copy takes 41 secs for the same folder.
>
> I'd like to add wildcard processing for the sources, but haven't found a good example.

do you compare single-threaded robocopy with your implementation or multithreaded?

you can command robocopy to use multiple threads with /MT[:n]

On Monday, 5 March 2012 at 16:35:09 UTC, dennis luehring wrote:
>
> do you compare single-threaded robocopy with your implementation or multithreaded?
>
> you can command robocopy to use multiple threads with /MT[:n]

yes, I tested vs multithread robocopy.  As someone pointed out, robocopy has lots of nice options, which I didn't try to duplicate, and is only about 10% slower on my test.

I was happy to see the D app in the same ballpark as robocopy, which means to me that the very simple and clean std.parallism taskpool foreach loop can produce very good multi-core results in a very concise and readable piece of code.  I've done some projects previously using omp pragmas in C++ and it is just so ugly.

So here is the output of a batch file I just ran on the ssd drive for the 1.5GB copy.  Robocopy displays that it took around 14 secs, while the release build of the D commandline cpd utility took around 12 secs.  That's a pretty consistent result on the ssd drive, which are more sensitive to cpu pr.

06:12 PM

H:\xx8>robocopy /E /NDL /NFL /NC /NS /MT:8 xx8c xx8ca

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Mon Mar 05 18:12:33 2012

   Source : H:\xx8\xx8c\
     Dest : H:\xx8\xx8ca\

    Files : *.*

  Options : *.* /NS /NC /NDL /NFL /S /E /COPY:DAT /MT:8 /R:1000000 /W:30

------------------------------------------------------------------------------
100%

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      2627      2626         1         0         0         0
   Files :     36969     36969         0         0         0         0
   Bytes :   1.502 g   1.502 g         0         0         0         0
   Times :   0:02:05   0:00:12                       0:00:00   0:00:01

   Ended : Mon Mar 05 18:12:47 2012

H:\xx8>time /T
06:12 PM

H:\xx8>rmd xx8ca\*
removing: xx8ca\Cross_Tools
removing: xx8ca\eclipse
removing: xx8ca\gnu
removing: xx8ca\PA
finished! time:17889 ms

H:\xx8>time /T
06:13 PM

H:\xx8>cpd xx8c\* xx8ca
copying: xx8c\Cross_Tools
copying: xx8c\eclipse
copying: xx8c\gnu
copying: xx8c\PA
finished! time: 11681 ms

H:\xx8>time /T
06:13 PM

btw, I just ran robocopy with /mt:1, and it took around 42 seconds on the same drive, which is about what I see with the standard windows copy, including the gui copy.  So, at least for these ssd drives the parallel processing results in worthwhile speed-ups.

Started : Mon Mar 05 18:24:31 2012
Ended : Mon Mar 05 18:25:13 2012

On Tuesday, 6 March 2012 at 00:29:01 UTC, Jay Norwood wrote:
> So here is the output of a batch file I just ran on the ssd drive for the 1.5GB copy.  Robocopy displays that it took around 14 secs, while the release build of the D commandline cpd utility took around 12 secs.  That's a pretty consistent result on the ssd drive, which are more sensitive to cpu pr.
>
> 06:12 PM
>
> H:\xx8>robocopy /E /NDL /NFL /NC /NS /MT:8 xx8c xx8ca
>
> -------------------------------------------------------------------------------
>    ROBOCOPY     ::     Robust File Copy for Windows
> -------------------------------------------------------------------------------
>
>   Started : Mon Mar 05 18:12:33 2012
>
>    Source : H:\xx8\xx8c\
>      Dest : H:\xx8\xx8ca\
>
>     Files : *.*
>
>   Options : *.* /NS /NC /NDL /NFL /S /E /COPY:DAT /MT:8 /R:1000000 /W:30
>
> ------------------------------------------------------------------------------
> 100%
>
> ------------------------------------------------------------------------------
>
>                Total    Copied   Skipped  Mismatch    FAILED    Extras
>     Dirs :      2627      2626         1         0         0
>    0
>    Files :     36969     36969         0         0         0
>    0
>    Bytes :   1.502 g   1.502 g         0         0         0
>    0
>    Times :   0:02:05   0:00:12                       0:00:00   0:00:01
>
>    Ended : Mon Mar 05 18:12:47 2012
>
> H:\xx8>time /T
> 06:12 PM
>
> H:\xx8>rmd xx8ca\*
> removing: xx8ca\Cross_Tools
> removing: xx8ca\eclipse
> removing: xx8ca\gnu
> removing: xx8ca\PA
> finished! time:17889 ms
>
> H:\xx8>time /T
> 06:13 PM
>
> H:\xx8>cpd xx8c\* xx8ca
> copying: xx8c\Cross_Tools
> copying: xx8c\eclipse
> copying: xx8c\gnu
> copying: xx8c\PA
> finished! time: 11681 ms
>
> H:\xx8>time /T
> 06:13 PM
>
> btw, I just ran robocopy with /mt:1, and it took around 42 seconds on the same drive, which is about what I see with the standard windows copy, including the gui copy.  So, at least for these ssd drives the parallel processing results in worthwhile speed-ups.
>
> Started : Mon Mar 05 18:24:31 2012
> Ended : Mon Mar 05 18:25:13 2012

The rule of thumb is use double number of threads of the logical cores..use GS rich copy 360 enterprise..supports upto 256 threads at once..not sure about robocopy.

On Tuesday, 6 March 2012 at 00:29:01 UTC, Jay Norwood wrote:
>
> H:\xx8>robocopy /E /NDL /NFL /NC /NS /MT:8 xx8c xx8ca
>

For a fair comparison, I think that's the command line:

robocopy /E /NDL /NFL /NC /NS /MT:8 /COPY:D /NODCOPY /XJ /R:0 /W:0 /NP /NJH /NJS /256 xx8c xx8ca