For very long file names it is broke and every command fails. These paths are not all that long but over 256 limit. (For windows)

The problem this causes can be disastrous. If some check fails and runs code that isn't mean to run if the file exists, it could destroy data.


I replaced many of the std.file functions with executeShell(...) and using command line functions and they work. But then my code was still failing and it was because exist was returning false even though the file exists.

I'm sure this is not a big deal to some...

On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:
> For very long file names it is broke and every command fails. These paths are not all that long but over 256 limit. (For windows)

Please file a bug report with reproducible examples if you believe it's a bug.

On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:
> For very long file names it is broke and every command fails. These paths are not all that long but over 256 limit. (For windows)
>
> The problem this causes can be disastrous. If some check fails and runs code that isn't mean to run if the file exists, it could destroy data.
>
>
> I replaced many of the std.file functions with executeShell(...) and using command line functions and they work. But then my code was still failing and it was because exist was returning false even though the file exists.
>
> I'm sure this is not a big deal to some...

Without knowing the issue in detail, is this maybe related to https://www.google.de/amp/s/www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/amp/ ?

Kind regards
Andre

On Friday, 14 September 2018 at 19:17:58 UTC, bachmeier wrote:
> On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:
>> For very long file names it is broke and every command fails. These paths are not all that long but over 256 limit. (For windows)
>
> Please file a bug report with reproducible examples if you believe it's a bug.

It's a bug, but how the hell can I reproduce examples when it depends on the file system? Do you want me to make a VM image and upload that too?


dirEntries(elem, SpanMode.shallow))

also fails. I've changed all my code over to using executeShell and everything works except that now I can't even iterate over directories that are too long.


This is ridiculous!


"It doesn't matter. When I compile a program or DLL in C/C++ and many other languages, I use the Windows headers. These headers define MAX_PATH to 260.
So the program will have the limit set by compile time, no matter what you do in the OS later on. There is no advantage with this setting for now, except that you now *may* pass long paths to API functions w/o the infamous \\?\ prefix, but you have no guarantee for it.

So to make this have the advantage that some people think it will provide, we'd need:

    wait until the Windows headers (in Visual Studio, or the DDK, or the Platform SDK or whatever) is properly updated, i.e. the runtime libs may take advantage of it
    wait until MS forces this setting to be enabled all the time, otherwise it's useless when it stays optional
    not checking any more for the target OS in the application, assuming Win 10 with the mentioned update

The latter will only be the case when an app can't target an Windows version below 10. So all in all it will take years until we get the advantage for standalone applications, and for TC these things don't matter for now anyway.
"

enum size_t MAX_PATH = 260;

!!


Seems this is the problem! What a great failure scenario! How bout at least check if the filename is larger than max path?

Seesh, the more I use D the more I regret it.

https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maxpath

But because MAX_PATH is an enum, it can't be redefined without recompiling phobos, which means it will break later on...

It does say that this is not a problem with unicode... but...

On Friday, 14 September 2018 at 19:42:39 UTC, Josphe Brigmo wrote:
> It's a bug, but how the hell can I reproduce examples when it depends on the file system?

Something like this (though I don't know much about Windows, so this might be wrong):

auto path = getcwd;
auto dir = `0123456789abcdef`;
foreach (i; 0..20)
{
  path = buildPath(path, dir);
}
system("mkdir " ~ path);
dirEntries(path, SpanMode.shallow);

And describe what filesystem you're using (presumably NTFS), what OS and version, 64-bit vs 32-bit, etc.

On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:
> For very long file names it is broke and every command fails. These paths are not all that long but over 256 limit. (For windows)
>
> The problem this causes can be disastrous. If some check fails and runs code that isn't mean to run if the file exists, it could destroy data.
>
>
> I replaced many of the std.file functions with executeShell(...) and using command line functions and they work. But then my code was still failing and it was because exist was returning false even though the file exists.
>
> I'm sure this is not a big deal to some...

this is an issue with the Win32 API having that 260 character limit. To work around it, you can use the special path syntax Microsoft allows you to do, which will pass the string you provide directly to the Filesystem instead of parsing it, effectively raising your limit to whatever the Filesystem limits to.

See also their official site on this: https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation

C:/Some/Very/Long/Path.txt

should then become

\\?\C:\Some\Very\Long\Path.txt

converting / to \ is important because according to the docs windows doesn't do it with the \\?\ prefix. You also have to normalize the path beforehand (i.e. remove ..\ and .\ because they would be treated as actual folder names)


To change a server path such as

\\share\public

you change it to

\\?\UNC\share\public



There are also some other useful things in there you might want to look at too like the special files such as COM1

On Saturday, 15 September 2018 at 09:47:25 UTC, WebFreak001 wrote:
> On Friday, 14 September 2018 at 19:06:14 UTC, Josphe Brigmo wrote:
>> For very long file names it is broke and every command fails. These paths are not all that long but over 256 limit. (For windows)
>>
>> The problem this causes can be disastrous. If some check fails and runs code that isn't mean to run if the file exists, it could destroy data.
>>
>>
>> I replaced many of the std.file functions with executeShell(...) and using command line functions and they work. But then my code was still failing and it was because exist was returning false even though the file exists.
>>
>> I'm sure this is not a big deal to some...
>
> this is an issue with the Win32 API having that 260 character limit. To work around it, you can use the special path syntax Microsoft allows you to do, which will pass the string you provide directly to the Filesystem instead of parsing it, effectively raising your limit to whatever the Filesystem limits to.
>
> See also their official site on this: https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation
>
> C:/Some/Very/Long/Path.txt
>
> should then become
>
> \\?\C:\Some\Very\Long\Path.txt
>
> converting / to \ is important because according to the docs windows doesn't do it with the \\?\ prefix. You also have to normalize the path beforehand (i.e. remove ..\ and .\ because they would be treated as actual folder names)
>
>
> To change a server path such as
>
> \\share\public
>
> you change it to
>
> \\?\UNC\share\public
>
>
>
> There are also some other useful things in there you might want to look at too like the special files such as COM1


Yes, I did that and it worked for some(most) things it seems but rmdir, for example, seems to fail.

Also, windows 10 does not have this problem nor does unicode so maybe phobos needs to automatically do everything? If it did I wouldn't have had this problem and wasted a day of my life trying to figure out what is going on(I didn't design my program around having to hack such things, I just assumed they would work, because, after all, they should, right?). I then used execute shell and had to work around that but still had problems because I didn't think dirEntries would be failing.

Basically every file function will fail in some odd way(depending on it's use) leaving one to deal with a whole complex of "bugs" when there is really a common bug.

rmdir, now is failing but this may be some other bug introduced by me in trying to fix other bugs.

On Saturday, 15 September 2018 at 10:05:26 UTC, Josphe Brigmo wrote:
> Yes, I did that and it worked for some(most) things it seems but rmdir, for example, seems to fail.

If the file path is passed verbatim to the OS API, and it still doesn't work, then the problem is with the OS or the API, not D.

> Also, windows 10 does not have this problem

What do you mean by "windows 10"? Do you mean Explorer, the default file manager?

Some reasons why it might work whereas the API doesn't:

- It's using workarounds, like the \\?\ prefix, or moving/renaming the object to a shorter name before deleting it.
- It's using a newer API, like WinRT (introduced in Windows 8).
- It's using a secret Microsoft API.

> If it did I wouldn't have had this problem and wasted a day of my life trying to figure out what is going on(I didn't design my program around having to hack such things, I just assumed they would work, because, after all, they should, right?).

I, too, spent a LOT of time fighting the Windows filesystem APIs. See e.g. * https://dump.thecybershadow.net/d78d9911adc16ec749914b6923759454/longpathdelete.d (that also sets ownership/ACLs via external processes, as the C API is unreasonably complicated).

The problem is 100% due to Windows.

It was one of the big reasons why I moved to Linux for software development. Such problems do not exist there.

On Friday, 14 September 2018 at 19:42:39 UTC, Josphe Brigmo wrote:
> "It doesn't matter. When I compile a program or DLL in C/C++ and many other languages, I use the Windows headers. These headers define MAX_PATH to 260.
> So the program will have the limit set by compile time, no matter what you do in the OS later on. There is no advantage with this setting for now, except that you now *may* pass long paths to API functions w/o the infamous \\?\ prefix, but you have no guarantee for it.
>
> So to make this have the advantage that some people think it will provide, we'd need:
>
>     wait until the Windows headers (in Visual Studio, or the DDK, or the Platform SDK or whatever) is properly updated, i.e. the runtime libs may take advantage of it
>     wait until MS forces this setting to be enabled all the time, otherwise it's useless when it stays optional
>     not checking any more for the target OS in the application, assuming Win 10 with the mentioned update
>
> The latter will only be the case when an app can't target an Windows version below 10. So all in all it will take years until we get the advantage for standalone applications, and for TC these things don't matter for now anyway.
> "
>
> enum size_t MAX_PATH = 260;

MAX_PATH only applies to static arrays (usually on the stack), i.e. code which does this:

void doSomething()
{
    char fileName[MAX_PATH];
    ...
    SomeWindowsAPI(fileName);
}

D almost never does this - instead, it uses dynamically sized buffers which fit the entire file name. The only times MAX_PATH is used in Phobos is:

- thisExePath (return path to current .exe file)
- tempDir (return path to temporary directory)

Obviously neither of these is related to the problems you're seeing.