March 05, 2012 Re: parallel copy directory, faster than robocopy | ||||
---|---|---|---|---|
| ||||
Posted in reply to Andrei Alexandrescu | On Monday, 5 March 2012 at 12:48:54 UTC, Andrei Alexandrescu wrote:
>
> Sounds great! Next step, should you be interested, is to create a pull request for phobos so we can integrate your code within.
>
> Andrei
I considered that. I suppose the wildArgv code could go in std.path, and the file operations into std.file. and the pull requests against those files.
I haven't followed the discussions closely enough to know what are the rules/politics about adding another std library import into those. It would require adding import of std.parallelism into std.file.
|
March 05, 2012 Re: parallel copy directory, faster than robocopy | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jay Norwood | Am 14.02.2012 05:58, schrieb Jay Norwood:
> Attached is the source for a small parallel app that copies a source folder to a destination. It creates the directory structure first using the breadth ordering, then uses a parallel foreach loop with the taskPool to copy all the regular files in parallel. On my corei7, this copied a 1.5GB folder with around 36K entries to a destination in about 11.5 secs (src and dest on the same ssd drive). This was about a second better than robocopy, which is the fastest alternative I could find. The regular win7-64 copy takes 41 secs for the same folder.
>
> I'd like to add wildcard processing for the sources, but haven't found a good example.
do you compare single-threaded robocopy with your implementation or multithreaded?
you can command robocopy to use multiple threads with /MT[:n]
|
March 05, 2012 Re: parallel copy directory, faster than robocopy | ||||
---|---|---|---|---|
| ||||
Posted in reply to dennis luehring | On Monday, 5 March 2012 at 16:35:09 UTC, dennis luehring wrote:
>
> do you compare single-threaded robocopy with your implementation or multithreaded?
>
> you can command robocopy to use multiple threads with /MT[:n]
yes, I tested vs multithread robocopy. As someone pointed out, robocopy has lots of nice options, which I didn't try to duplicate, and is only about 10% slower on my test.
I was happy to see the D app in the same ballpark as robocopy, which means to me that the very simple and clean std.parallism taskpool foreach loop can produce very good multi-core results in a very concise and readable piece of code. I've done some projects previously using omp pragmas in C++ and it is just so ugly.
|
March 06, 2012 Re: parallel copy directory, faster than robocopy | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jay Norwood | So here is the output of a batch file I just ran on the ssd drive for the 1.5GB copy. Robocopy displays that it took around 14 secs, while the release build of the D commandline cpd utility took around 12 secs. That's a pretty consistent result on the ssd drive, which are more sensitive to cpu pr. 06:12 PM H:\xx8>robocopy /E /NDL /NFL /NC /NS /MT:8 xx8c xx8ca ------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows ------------------------------------------------------------------------------- Started : Mon Mar 05 18:12:33 2012 Source : H:\xx8\xx8c\ Dest : H:\xx8\xx8ca\ Files : *.* Options : *.* /NS /NC /NDL /NFL /S /E /COPY:DAT /MT:8 /R:1000000 /W:30 ------------------------------------------------------------------------------ 100% ------------------------------------------------------------------------------ Total Copied Skipped Mismatch FAILED Extras Dirs : 2627 2626 1 0 0 0 Files : 36969 36969 0 0 0 0 Bytes : 1.502 g 1.502 g 0 0 0 0 Times : 0:02:05 0:00:12 0:00:00 0:00:01 Ended : Mon Mar 05 18:12:47 2012 H:\xx8>time /T 06:12 PM H:\xx8>rmd xx8ca\* removing: xx8ca\Cross_Tools removing: xx8ca\eclipse removing: xx8ca\gnu removing: xx8ca\PA finished! time:17889 ms H:\xx8>time /T 06:13 PM H:\xx8>cpd xx8c\* xx8ca copying: xx8c\Cross_Tools copying: xx8c\eclipse copying: xx8c\gnu copying: xx8c\PA finished! time: 11681 ms H:\xx8>time /T 06:13 PM btw, I just ran robocopy with /mt:1, and it took around 42 seconds on the same drive, which is about what I see with the standard windows copy, including the gui copy. So, at least for these ssd drives the parallel processing results in worthwhile speed-ups. Started : Mon Mar 05 18:24:31 2012 Ended : Mon Mar 05 18:25:13 2012 |
December 11, 2017 Re: parallel copy directory, faster than robocopy | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jay Norwood | On Tuesday, 6 March 2012 at 00:29:01 UTC, Jay Norwood wrote:
> So here is the output of a batch file I just ran on the ssd drive for the 1.5GB copy. Robocopy displays that it took around 14 secs, while the release build of the D commandline cpd utility took around 12 secs. That's a pretty consistent result on the ssd drive, which are more sensitive to cpu pr.
>
> 06:12 PM
>
> H:\xx8>robocopy /E /NDL /NFL /NC /NS /MT:8 xx8c xx8ca
>
> -------------------------------------------------------------------------------
> ROBOCOPY :: Robust File Copy for Windows
> -------------------------------------------------------------------------------
>
> Started : Mon Mar 05 18:12:33 2012
>
> Source : H:\xx8\xx8c\
> Dest : H:\xx8\xx8ca\
>
> Files : *.*
>
> Options : *.* /NS /NC /NDL /NFL /S /E /COPY:DAT /MT:8 /R:1000000 /W:30
>
> ------------------------------------------------------------------------------
> 100%
>
> ------------------------------------------------------------------------------
>
> Total Copied Skipped Mismatch FAILED Extras
> Dirs : 2627 2626 1 0 0
> 0
> Files : 36969 36969 0 0 0
> 0
> Bytes : 1.502 g 1.502 g 0 0 0
> 0
> Times : 0:02:05 0:00:12 0:00:00 0:00:01
>
> Ended : Mon Mar 05 18:12:47 2012
>
> H:\xx8>time /T
> 06:12 PM
>
> H:\xx8>rmd xx8ca\*
> removing: xx8ca\Cross_Tools
> removing: xx8ca\eclipse
> removing: xx8ca\gnu
> removing: xx8ca\PA
> finished! time:17889 ms
>
> H:\xx8>time /T
> 06:13 PM
>
> H:\xx8>cpd xx8c\* xx8ca
> copying: xx8c\Cross_Tools
> copying: xx8c\eclipse
> copying: xx8c\gnu
> copying: xx8c\PA
> finished! time: 11681 ms
>
> H:\xx8>time /T
> 06:13 PM
>
> btw, I just ran robocopy with /mt:1, and it took around 42 seconds on the same drive, which is about what I see with the standard windows copy, including the gui copy. So, at least for these ssd drives the parallel processing results in worthwhile speed-ups.
>
> Started : Mon Mar 05 18:24:31 2012
> Ended : Mon Mar 05 18:25:13 2012
|
February 06, 2018 Re: parallel copy directory, faster than robocopy | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jay Norwood | The rule of thumb is use double number of threads of the logical cores..use GS rich copy 360 enterprise..supports upto 256 threads at once..not sure about robocopy. |
February 07, 2018 Re: parallel copy directory, faster than robocopy | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jay Norwood | On Tuesday, 6 March 2012 at 00:29:01 UTC, Jay Norwood wrote:
>
> H:\xx8>robocopy /E /NDL /NFL /NC /NS /MT:8 xx8c xx8ca
>
For a fair comparison, I think that's the command line:
robocopy /E /NDL /NFL /NC /NS /MT:8 /COPY:D /NODCOPY /XJ /R:0 /W:0 /NP /NJH /NJS /256 xx8c xx8ca
|
Copyright © 1999-2021 by the D Language Foundation