C:\a\b\b\b.dustmite\..\..\..\..\x\y\pegged\dynamic\grammar.d: The process cannot access the file because it is being used by another process

When I run dustmite on a project.

The x is a junction. Nothing is accessing grammar.d except maybe visual studio internally... but even if it were it still should be accessible for copying. (after all I can open the file).

Seems visual D craps out because it is a junction?

Could this be remedied?

I use junctions often because it helps with organization(rather than duplicating).

It may not be from it being a junction, but if it is not I can't imagine why it is having trouble with it(assuming the error message is correct).

On Sunday, 4 November 2018 at 17:16:55 UTC, Michelle Long wrote:
> C:\a\b\b\b.dustmite\..\..\..\..\x\y\pegged\dynamic\grammar.d: The process cannot access the file because it is being used by another process
>
> When I run dustmite on a project.

As the error message says, the problem occurs because another file is accessing it.

Some common culprits are:

- Anti-virus / security software, such as Windows Defender. Adding the directory containing the D code being reduced as an exception usually helps.
- File synchronization software, such as OneDrive / SyncThing.
- Editors/IDEs. Make sure the directory being reduced is not part of any project/solution open in Visual Studio etc.

I think I made Dustmite retry after a second when it gets that error. Is that not the case? Post the full output, if so.

On Sunday, 4 November 2018 at 17:20:41 UTC, Vladimir Panteleev wrote:
> On Sunday, 4 November 2018 at 17:16:55 UTC, Michelle Long wrote:
>> C:\a\b\b\b.dustmite\..\..\..\..\x\y\pegged\dynamic\grammar.d: The process cannot access the file because it is being used by another process
>>
>> When I run dustmite on a project.
>
> As the error message says, the problem occurs because another file is accessing it.
>

No, have you not learned in your life time that just because something or someeone says something doesn't make it true?

> Some common culprits are:
>
> - Anti-virus / security software, such as Windows Defender. Adding the directory containing the D code being reduced as an exception usually helps.
> - File synchronization software, such as OneDrive / SyncThing.
> - Editors/IDEs. Make sure the directory being reduced is not part of any project/solution open in Visual Studio etc.
>

Nope, none of this because I run none. Please trust people a bit more when they say what they say. Not everyone is an imbecile.

> I think I made Dustmite retry after a second when it gets that error. Is that not the case? Post the full output, if so.


This has nothing to do with dustmite as far as I know. That is the full output.

Visual D creates a temp project and when it tries to copy the files this error occurs. It is almost surely due to a junction issue as I mentioned. (believe me when I tell you I'm not a moron so when I say something it generally has some truth in it)

If this is dustmite and not visual D then it most likely is due to the junction.


I have used dustmite before and it has worked in visual D, but either a regression occurred or it is due to using the junction(which is the first project I have created that uses a junction and dustmited).


Let me enlighten you a little: The reason why this is most likely a junction is:

1. There is little else it could be except visual studio itself locking the file, which it doesn't seem to be and has worked before.

2. Junctions are known to cause problems with some copy routines because one can't simply copy a junction as normal. The reason being is the junction suppose to be duplicated or all the data?



e.g., this happens to others:

https://answers.microsoft.com/en-us/windows/forum/windows8_1-files/bug-file-copies-freeze-when-symbolic-links-or/10e0c4f3-ca9e-403e-ba53-58be6bddce36?auth=1

On Sunday, 4 November 2018 at 21:11:27 UTC, Michelle Long wrote:
> No, have you not learned in your life time that just because something or someeone says something doesn't make it true?

In all my years of using Windows, I have never seen or heard for this error message to lie (i.e. occur due to a reason different than what it states). I've also used junctions and symbolic links a lot.

It's possible the junction is confusing some software into not closing a file, though.

> Nope, none of this because I run none. Please trust people a bit more when they say what they say. Not everyone is an imbecile.

Sometimes, the reason can be really unobvious. We've ran into a similar problem a while ago on CI machines, which turned out to be the security software included with the then-new version of Windows. It doesn't hurt to check.

> This has nothing to do with dustmite as far as I know. That is the full output.

Probably trying to invoke Dustmite manually would be progress then, with the Visual D bug report aside.

> If this is dustmite and not visual D then it most likely is due to the junction.

1. DustMite will not touch anything outside the parent directory of the path you give it (though the test command might).
2. DustMite doesn't care about junctions and treats them as directories. When creating test copies of the source, it will recreate the directory structure with junctions replaced with directories.

> 1. There is little else it could be except visual studio itself locking the file, which it doesn't seem to be and has worked before.

FWIW, you can use Process Explorer (by Microsoft) or Process Hacker (FOSS) to find which processes hold handles to a file. That might provide a clue.

> 2. Junctions are known to cause problems with some copy routines because one can't simply copy a junction as normal. The reason being is the junction suppose to be duplicated or all the data?

Yes, well known gotcha. std.file.dirEntries has a followSymlink parameter. DustMite uses the default (true).

On Sunday, 4 November 2018 at 22:12:16 UTC, Vladimir Panteleev wrote:
> On Sunday, 4 November 2018 at 21:11:27 UTC, Michelle Long wrote:
>> No, have you not learned in your life time that just because something or someeone says something doesn't make it true?
>
> In all my years of using Windows, I have never seen or heard for this error message to lie (i.e. occur due to a reason different than what it states). I've also used junctions and symbolic links a lot.
>
> It's possible the junction is confusing some software into not closing a file, though.
>
>> Nope, none of this because I run none. Please trust people a bit more when they say what they say. Not everyone is an imbecile.
>
> Sometimes, the reason can be really unobvious. We've ran into a similar problem a while ago on CI machines, which turned out to be the security software included with the then-new version of Windows. It doesn't hurt to check.
>
>> This has nothing to do with dustmite as far as I know. That is the full output.
>
> Probably trying to invoke Dustmite manually would be progress then, with the Visual D bug report aside.
>
>> If this is dustmite and not visual D then it most likely is due to the junction.
>
> 1. DustMite will not touch anything outside the parent directory of the path you give it (though the test command might).
> 2. DustMite doesn't care about junctions and treats them as directories. When creating test copies of the source, it will recreate the directory structure with junctions replaced with directories.
>
>> 1. There is little else it could be except visual studio itself locking the file, which it doesn't seem to be and has worked before.
>
> FWIW, you can use Process Explorer (by Microsoft) or Process Hacker (FOSS) to find which processes hold handles to a file. That might provide a clue.
>
>> 2. Junctions are known to cause problems with some copy routines because one can't simply copy a junction as normal. The reason being is the junction suppose to be duplicated or all the data?
>
> Yes, well known gotcha. std.file.dirEntries has a followSymlink parameter. DustMite uses the default (true).


I am pretty sure it is a bug in Visual D not handling the junction right. It was a fresh boot with no other programs ran before and given other experiences it seems almost surely(95%+) that it is a junction problem with visual D. Since it is such a high liklihood it should be the first thing tried.

While I could run dustmite from the command line, the point of using visual D is because it handles it all automatically, which is nice, when it works.

Now, I can't be sure if it's Visual D or Dustmite because that is the only error I get, but visual D says it needs to copy the project to a temp folder and then the error appears immediately. I bet if I replaced the junction with a normal directory it would work but then that defeats the purpose of using a junction.

It would take Rainer just a few minutes max to determine this by simply created a project or opening a pre-existing one and adding a junction the dragging it to the project and adding a d file in it then trying the dustmite command.

Again, I do not think this is a dustmite problem, it occurs when trying to run dustmite using the Visual D dustmite command(which, again, has worked in the past without issue and all it does is copy the project then run dustmite... and the error is about accessing a file).

Because I'm sure this is related to Visual D and junctions(whatever copying method it uses) I'm willing not to try random stuff to try and fix it... expecting that Rainer can reproduce this problem fairly quickly when he gets time. If he adds a junction and it doesn't have an error then I will look in to it more on my side. No reason for me to spend hours trying to figure out possible edge cases until I know it is not the most probable cause. It will take him a few minutes to investigate and chances are it will be easily solved.

If it turns out something is locking the file, then that is still a problem because nothing I have done locks the file it complains on(so it's still a bug). I did not ever touch that file in any way. I simply loaded the project and ran the command(the file is not even loaded in the editor or being accessed by anything external(unless windows is doing funky shit, which I doubt since I disabled most of all the crap it does)).

On 05/11/2018 00:08, Michelle Long wrote:
> It would take Rainer just a few minutes max to determine this by simply created a project or opening a pre-existing one and adding a junction the dragging it to the project and adding a d file in it then trying the dustmite command.

I just tried that and it worked fine for my tiny example. Copying the project is just a series of

	std.file.mkdirRecurse(dirName(destname));
	std.file.copy(srcname, destname);

on the files in the project.

Do you get the message in the Visual D output pane or the console window?

On Monday, 5 November 2018 at 07:24:36 UTC, Rainer Schuetze wrote:
>
>
> On 05/11/2018 00:08, Michelle Long wrote:
>> It would take Rainer just a few minutes max to determine this by simply created a project or opening a pre-existing one and adding a junction the dragging it to the project and adding a d file in it then trying the dustmite command.
>
> I just tried that and it worked fine for my tiny example. Copying the project is just a series of
>
> 	std.file.mkdirRecurse(dirName(destname));
> 	std.file.copy(srcname, destname);
>
> on the files in the project.
>
> Do you get the message in the Visual D output pane or the console window?

It is a visual D output problem.

I tried several things and had strange results.

I created a new project in D and ran dustmite:

created clean copy of the project in c:\users\main\documents\visual studio 2017\Projects\WindowsApp4\WindowsApp4.dustmite
Failed to make stdin stream inheritable by child process (Access is denied.)


I tried removing the junction and it still seems to fail on grammar.d.

I ran a chkdsk and no errors.


I used lockhunter and no locks on the file.

I tried copying the project over manually and no files where copied from the junction.

I tried copying the junction directory and everything was copied but pegged.

the dustmite directory shows that it is copying(some files)

So this might not be a Visual D issue but something in the OS itself.


The error I get is

D:\.......\Libraries\pegged\dynamic\grammar.d: The process cannot access the file because it is being used by another process.

In visual D's error pane.

Actually, the error is:

D:\Projects\Game\Game\Game.dustmite\..\..\..\..\D\Libraries\pegged\dynamic\grammar.d: The process cannot access the file because it is being used by another process.

When I click on the error it opened the file in the editor and it was correct.


I really have no idea what is going on.. maybe the junction is "bad"(it works in xyplorer but maybe something got corrupted?).

The fact that copying the project dir manually using xyplorer or windows file explorer both fail to copy the junction suggest it is not a Visual D problem.

When I copy the files over from the library directly(bypassing the junction issue) the same problem occurs.

So, it would suggest a problem with the grammar.d file, yet I can read it just fine.

Maybe it is just a problem with that file in relation to Visual D?

Maybe if you could allow the process to continue if we ignore "Remove dustmite dir" then I could just copy the files manually(which works).

It probably won't fix the stdin issue but it would get around the copying issue.

Other than that, I'm at a loss.

I moved some stuff around trying to see if it was anything with the project and all it it was change the copy error:

D:\Projects\Game\Game\Game.dustmite\..\..\..\Libraries\arsd\png.d: The process cannot access the file because it is being used by another process.

It did change the path but probably because I pointed the folder in visual studio to the junction(I think I was actually using the library path before).

In any case it seems maybe something is causing the file to be temporarily locked inside visual study when dustmite command is issued.

Could this be with the Visual D copying method?

If you are copying each file using recursion maybe it is accessing the file to copy before it is released in "looking" up?

You could try an alternative copy such as fastcopy(which can handle junctions well)?

If I click on the file it says can't open(such as png.d) then it opens it in the editor, so the path is correct and the file is not locked then.

Maybe retry the copy with a sleep a few times if it fails at first?

On Sunday, 4 November 2018 at 17:16:55 UTC, Michelle Long wrote:
> C:\a\b\b\b.dustmite\..\..\..\..\x\y\pegged\dynamic\grammar.d: The process cannot access the file because it is being used by another process
>
> When I run dustmite on a project.
>
> The x is a junction. Nothing is accessing grammar.d except maybe visual studio internally... but even if it were it still should be accessible for copying. (after all I can open the file).
>
> Seems visual D craps out because it is a junction?
>
> Could this be remedied?
>
> I use junctions often because it helps with organization(rather than duplicating).
>
> It may not be from it being a junction, but if it is not I can't imagine why it is having trouble with it(assuming the error message is correct).

Suggestion, when getting lock error, use Process Explorer to inspect which process has an open handle on it and work from there.  You can google Process Explorer and down load it from Microsoft website
Cheer

On 06/11/2018 04:09, Michelle Long wrote:
> D:\Projects\Game\Game\Game.dustmite\..\..\..\..\D\Libraries\pegged\dynamic\grammar.d:
> The process cannot access the file because it is being used by another
> process.
> 

I can reproduce it now, but it has nothing to do with links/junctions.

The problem is that the shown file is outside of the project folder hierarchy. Visual D tries to recreate the projects folder structure, so it copies

D:\Projects\Game\Game\Game\..\..\..\..\D\Libraries\pegged\dynamic\grammar.d

to

D:\Projects\Game\Game\Game.dustmite\..\..\..\..\D\Libraries\pegged\dynamic\grammar.d

which is the same file!

A simple workaround is to use a junction in the project folder to access pegged.

I guess Visual D should not copy files outside of the project folder, but they are also not subject to dustmite in that case.

Alternatively, the dustmite directory could be created in a common parent folder for all files, but that doesn't work for files distributed over different drives.

On Thursday, 8 November 2018 at 07:16:19 UTC, Rainer Schuetze wrote:
>
>
> On 06/11/2018 04:09, Michelle Long wrote:
>> D:\Projects\Game\Game\Game.dustmite\..\..\..\..\D\Libraries\pegged\dynamic\grammar.d:
>> The process cannot access the file because it is being used by another
>> process.
>> 
>
> I can reproduce it now, but it has nothing to do with links/junctions.
>
> The problem is that the shown file is outside of the project folder hierarchy. Visual D tries to recreate the projects folder structure, so it copies
>
> D:\Projects\Game\Game\Game\..\..\..\..\D\Libraries\pegged\dynamic\grammar.d
>
> to
>
> D:\Projects\Game\Game\Game.dustmite\..\..\..\..\D\Libraries\pegged\dynamic\grammar.d
>
> which is the same file!
>
> A simple workaround is to use a junction in the project folder to access pegged.
>
> I guess Visual D should not copy files outside of the project folder, but they are also not subject to dustmite in that case.
>
> Alternatively, the dustmite directory could be created in a common parent folder for all files, but that doesn't work for files distributed over different drives.

whoo! ;) That dogged a bullet! Glad you were able to figure it out.

I am using a junction!

I have my libs in D\Libraries.

In my project I have a junction that points to it.

I dragged that junction to VS to add the files.

So they don't exist at the same place.

Visual D seems to copy some of the files.

Maybe the better solution would just be to delay and repeat.

If somehow it is copying two files simultaneously then the delay and retry might a few times will just copy it twice(it would have to override it).

Or maybe just check if the file already exists and avoid copying it? (although the grammar.d file that it complained about was never copied, so this probably won't work)

Maybe you could simply record any failures and try them at the end.

Here I'm assuming the locking is temporary and will disappear quickly enough given that I can never see the file being locked.