How does one copy folders (with files in them) including sub folders, and creating any needed folders?

Like:
land\house\cat.d
land\house\rat.exe
land\house\bedroom\ants.txt

to
root\island\house\cat.d
root\island\house\rat.exe
root\island\house\bedroom\ants.txt

One work around is to use 'system' (under std.process).

-joelcnz

On 2013-01-24 07:28, Joel wrote:
> How does one copy folders (with files in them) including sub folders,
> and creating any needed folders?
>
> Like:
> land\house\cat.d
> land\house\rat.exe
> land\house\bedroom\ants.txt
>
> to
> root\island\house\cat.d
> root\island\house\rat.exe
> root\island\house\bedroom\ants.txt
>
> One work around is to use 'system' (under std.process).

I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove.

-- 
/Jacob Carlborg

On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote:
> On 2013-01-24 07:28, Joel wrote:
>> How does one copy folders (with files in them) including sub folders,
>> and creating any needed folders?
>>
>> Like:
>> land\house\cat.d
>> land\house\rat.exe
>> land\house\bedroom\ants.txt
>>
>> to
>> root\island\house\cat.d
>> root\island\house\rat.exe
>> root\island\house\bedroom\ants.txt
>>
>> One work around is to use 'system' (under std.process).
>
> I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove.

Mustn't be very hard to manually write "copyDir".

The problem with "copyDir" is its transactional behavior: What should it do in case of a failure mid copy? Bail? Cleanup? Continue?

Anyways, I just threw this together. The first version bails on first error.
The second version keeps going as much as it can, and returns true on success.

The caller is then free to (or not to) call rmdirRecurse in case of failure.

//Throws exception on first error.
void copyDir(string inDir, string outDir)
{
    if (!exists(outDir))
        mkdir(outDir);
    else
        if (!isDir(outDir))
            throw new FileException(format("Destination path %s is not a folder.", outDir));

    foreach (entry; dirEntries(inDir.idup, SpanMode.shallow))
    {
        auto fileName = baseName(entry.name);
        auto destName = buildPath(outDir, fileName);
        if (entry.isDir())
            copyDir(entry.name, destName);
        else
            copy(entry.name, destName);
    }
}

//Silently keeps going as much as it can, then returns true on success,
//or false if an error occured.
bool copyDirSilent(string inDir, string outDir)
{
    if (!exists(outDir))
    {
        auto e = collectException(mkdir(outDir));
        if (e !is null)
            return false;
    }
    else
        if (!isDir(outDir))
            return false;

    foreach (entry; dirEntries(inDir, SpanMode.shallow))
    {
        auto fileName = baseName(entry.name);
        auto destName = buildPath(outDir, fileName);
        if (entry.isDir())
        {
            bool b = copyDirSilent(entry.name, destName);
            if (b == false)
                return false;
        }
        else
        {
            auto e = collectException(mkdir(outDir));
            if (e !is null)
                return false;
        }
    }
    return true;
}

Brilliant! Thanks.

I think the silent one has errors. But I've used the first function in my program now. With mine, strait after doing the copying is was supposed then update a file, but didn't. Your one seems to work though.

On Thursday, 24 January 2013 at 08:58:06 UTC, monarch_dodra wrote:
> On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote:
>> On 2013-01-24 07:28, Joel wrote:
>>> How does one copy folders (with files in them) including sub folders,
>>> and creating any needed folders?
>>>
>>> Like:
>>> land\house\cat.d
>>> land\house\rat.exe
>>> land\house\bedroom\ants.txt
>>>
>>> to
>>> root\island\house\cat.d
>>> root\island\house\rat.exe
>>> root\island\house\bedroom\ants.txt
>>>
>>> One work around is to use 'system' (under std.process).
>>
>> I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove.
>
> Mustn't be very hard to manually write "copyDir".
>
> The problem with "copyDir" is its transactional behavior: What should it do in case of a failure mid copy? Bail? Cleanup? Continue?
>
> Anyways, I just threw this together. The first version bails on first error.
> The second version keeps going as much as it can, and returns true on success.
>
> The caller is then free to (or not to) call rmdirRecurse in case of failure.
>
> //Throws exception on first error.
> void copyDir(string inDir, string outDir)
> {
>     if (!exists(outDir))
>         mkdir(outDir);
>     else
>         if (!isDir(outDir))
>             throw new FileException(format("Destination path %s is not a folder.", outDir));
>
>     foreach (entry; dirEntries(inDir.idup, SpanMode.shallow))
>     {
>         auto fileName = baseName(entry.name);
>         auto destName = buildPath(outDir, fileName);
>         if (entry.isDir())
>             copyDir(entry.name, destName);
>         else
>             copy(entry.name, destName);
>     }
> }
>
> //Silently keeps going as much as it can, then returns true on success,
> //or false if an error occured.
> bool copyDirSilent(string inDir, string outDir)
> {
>     if (!exists(outDir))
>     {
>         auto e = collectException(mkdir(outDir));
>         if (e !is null)
>             return false;
>     }
>     else
>         if (!isDir(outDir))
>             return false;
>
>     foreach (entry; dirEntries(inDir, SpanMode.shallow))
>     {
>         auto fileName = baseName(entry.name);
>         auto destName = buildPath(outDir, fileName);
>         if (entry.isDir())
>         {
>             bool b = copyDirSilent(entry.name, destName);
>             if (b == false)
>                 return false;
>         }
>         else
>         {
>             auto e = collectException(mkdir(outDir));
>             if (e !is null)
>                 return false;
>         }
>     }
>     return true;
> }

On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote:

> Someone posted code in these newsgroups of a parallel implementation of copy and remove.

I posted a parallel implementation a while back, and also put it on github.

The parallel trick is to create the folder structure first, then populate the folders with the files.  Best for ssd drives.

I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it.  This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd.

https://github.com/jnorwood

Sounds cool. I've bookmarked your github hub link.

Just wondering with this:

// parallel foreach for regular files
foreach(fn ; taskPool.parallel(files,100)) {
    string dfn = destRoot ~ fn[srcLen..$];
    copy(fn,dfn);
}

What is the 100 number for?

On Friday, 25 January 2013 at 05:30:11 UTC, Jay Norwood wrote:
> On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg wrote:
>
>> Someone posted code in these newsgroups of a parallel implementation of copy and remove.
>
> I posted a parallel implementation a while back, and also put it on github.
>
> The parallel trick is to create the folder structure first, then populate the folders with the files.  Best for ssd drives.
>
> I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it.  This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd.
>
> https://github.com/jnorwood

On 2013-01-25 06:30, Jay Norwood wrote:

> I posted a parallel implementation a while back, and also put it on github.
>
> The parallel trick is to create the folder structure first, then
> populate the folders with the files.  Best for ssd drives.
>
> I also wrote a copy version that orders file sequence on disk
> efficiently, using write through, and posted it.  This speeds up any
> subsequent file system operations done in the directory order as if you
> have done a defrag. Great for hard drives, but not needed for ssd.
>
> https://github.com/jnorwood

Any change you could turn this into a pull request and submit to Phobos?

-- 
/Jacob Carlborg

On 01/24/2013 10:23 PM, Joel wrote:

> // parallel foreach for regular files
> foreach(fn ; taskPool.parallel(files,100)) {

> What is the 100 number for?

It is the work unit size:

  http://dlang.org/phobos/std_parallelism.html#.TaskPool

Quoting:

The number of elements processed per work unit is controlled by the workUnitSize parameter. Smaller work units provide better load balancing, but larger work units avoid the overhead of communicating with other threads frequently to fetch the next work unit. Large work units also avoid false sharing in cases where the range is being modified. The less time a single iteration of the loop takes, the larger workUnitSize should be. For very expensive loop bodies, workUnitSize should be 1. An overload that chooses a default work unit size is also available.

Ali

>> I also wrote a copy version that orders file sequence on disk
>> efficiently, using write through, and posted it.  This speeds up any
>> subsequent file system operations done in the directory order as if you
>> have done a defrag. Great for hard drives, but not needed for ssd.
>>
>> https://github.com/jnorwood
>
> Any change you could turn this into a pull request and submit to Phobos?

Looks like I never promoted the write through code on github.  I
just posted it on the forum.dlang.org, along with some timings. I
was only experimenting on ntfs, and so the extension I posted
doesn't force write-through on linux.

http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj@forum.dlang.org

  After re-reading my post, I don't think I experimented with
write through in copydir.   I show the timings in the post for
ntfs operations on a 2GB layout after unzipping with a write
through version of unzip. I suspect you'd see similar improvement
following a copydir that uses write through.  The problem I saw
with ntfs was that the target directory was very fragmented when
allowing ntfs to do its lazy flushes.  I tried several
experiments, and using the write through was the only solution
that resulted in a reasonably defragged target immediately as the
result of the unzip.