Am Fri, 27 Dec 2013 22:12:37 -0800
schrieb Jonathan M Davis <jmdavisProg@gmx.com>:

> On Saturday, December 28, 2013 06:54:53 Marco Leise wrote:
> > For Phobos we need portable solutions. But it is also clear that std.file.setAttributes cannot be replaced to 100% by a portable solution. The question is: Does portable code _care_ about setting each possible chmod flag or Windows file attribute? Or are the use cases much more limited there, like making a file read-only or querying if it is a directory?
> 
> We need to try hard to make Phobos cross-platform and portable, but some stuff just can't be, and std.file already has some functions which fall in that category (e.g. anything symlink related or dealing with file times). And having a D wrapper around that functionality can make code much cleaner, so I'm all for having a limited number of system-specific functions in std.file if that's what it takes to get the job done (and file permissions tend to fall in that category).
> 
> - Jonathan M Davis

But SetFileAttributes doesn't set file permissions (except for
"read-only")! Chmod does. If you want a D wrapper for
POSIX chmod in std.file, it should be exactly that. For
Windows you need to work with ACLs to change permissions.

-- 
Marco

On Saturday, December 28, 2013 07:21:28 Marco Leise wrote:
> schrieb Jonathan M Davis <jmdavisProg@gmx.com>:
> > We need to try hard to make Phobos cross-platform and portable, but some stuff just can't be, and std.file already has some functions which fall in that category (e.g. anything symlink related or dealing with file times). And having a D wrapper around that functionality can make code much cleaner, so I'm all for having a limited number of system-specific functions in std.file if that's what it takes to get the job done (and file permissions tend to fall in that category).
> > 
> > - Jonathan M Davis
> 
> But SetFileAttributes doesn't set file permissions (except for
> "read-only")! Chmod does. If you want a D wrapper for
> POSIX chmod in std.file, it should be exactly that. For
> Windows you need to work with ACLs to change permissions.

I'm not arguing the exact API for setFileAttributes, since I haven't spent the time as of yet to look it over and therefore do not feel qualified to comment on it specifically. My main point was that some stuff in std.file is system- specific and that if it has to be, it's better to have it in std.file as system- specific rather than not having it at all just because it couldn't be completely cross-platform and portable. So, just because setFileAttributes is doing something system-specific does not mean that it's bad. But its validity may very well be argued on other counts (e.g. whether it's actually a good API for what it does).

- Jonathan M Davis

On Saturday, 28 December 2013 at 08:18:44 UTC, Jonathan M Davis wrote:
> on it specifically. My main point was that some stuff in std.file is system-
> specific and that if it has to be, it's better to have it in std.file as system-
> specific rather than not having it at all just because it couldn't be
> completely cross-platform and portable. So, just because setFileAttributes is
> doing something system-specific does not mean that it's bad.

It is bad. If you want system-specific behaviour you should have a separate interface that provide all the advantages that going to a lower level provides. Having half-assed OS-specific support is too pragmatic and will lead to a legacy mess in the long run when those interfaces become obsolete.

A good file abstraction should also support newer file systems like Google Cloud Storage, though. GCS does not support append() or directories.

A good file abstraction should also provide mechanisms to deal with different levels of consistency on the underlying filesystem and caching-mechanism:
- read after write might return an old version
- read after write returns the new version if read from the same computer
- read after write always returns the new version

As a new language D should support Cloud based environments out-of-the-box with a hierarchy of functionality down to the peculiarities of Windows/Posix etc so that you can decide to code in a manner that supports all platforms without having to litter your code with ifs and version-statements… and also restrict you to the subset you have "authorized" so that you can develop on your Windows system and port it to the cloud with just a recompile without knowing the details of the cloud file system. (i.e. the compiler warns you when you are using system specific functionality).

I also think good file system support implies built-in caching and let the library decide whether to turn library-level caching on or off based on what the operating system or underlying interface supports. E.g. the ability to tell the system that you want successive reads of a file to be available fast at the cost of consistency.

On 2013-12-28 03:46, Marco Leise wrote:

> Wait a second, what about *setting* attributes? Some difficult
> ones are:
>
> o toggling read-only (for whom? user, group, others?)
> o executable flag
> o hidden flag
>
> On Windows 'executable' is implicit and based on the extension.
> On Posix 'hidden' is implicit for a file name beginning with a
> dot. We can read the hidden bit on POSIX, but we cannot toggle
> it for example. So we can either not expose these attributes
> at all, ignore them where not applicable when setting
> attributes or add a third state "ignore".

In addition to that Mac OS X has an additional way of indicating if a file is is hidden or not. More similar to how it works on Windows then the naming scheme from Posix.

> Or looking at it another way:
> DOS attr < POSIX chmod < ACLs
>
> How do other programming languages find a common ground?

On the top of the documentation of the File class in Ruby, it says the following:

"In the description of File methods, permission bits are a platform-specific set of bits that indicate permissions of a file."

And

"On non-Posix operating systems, there may be only the ability to make a file read-only or read-write. In this case, the remaining permission bits will be synthesized to resemble typical values."

-- 
/Jacob Carlborg

As much as I dislike PHP, I think they are onto something with having an optional stream context object that you can pass onto various file functions. It allows you to treat HTTP and the regular file system in the same manner (by having authentication etc in the context object).

Rather than setting attributes you would populate the context object (you could use a OS-specific factory function/class) with the settings you want and then pass it around. I guess you could then precreate one context for ReadOnly, another for ExecutableBinary, another for MIME text/html etc…

Am Sat, 28 Dec 2013 15:23:55 +0100
schrieb Jacob Carlborg <doob@me.com>:

> On 2013-12-28 03:46, Marco Leise wrote:
> 
> > Wait a second, what about *setting* attributes? Some difficult ones are:
> >
> > o toggling read-only (for whom? user, group, others?)
> > o executable flag
> > o hidden flag
> >
> > On Windows 'executable' is implicit and based on the extension. On Posix 'hidden' is implicit for a file name beginning with a dot. We can read the hidden bit on POSIX, but we cannot toggle it for example. So we can either not expose these attributes at all, ignore them where not applicable when setting attributes or add a third state "ignore".
> 
> In addition to that Mac OS X has an additional way of indicating if a file is is hidden or not. More similar to how it works on Windows then the naming scheme from Posix.

Oh right, they have a hidden attribute as well. I guess if Phobos should expose the 'hidden' state of a file it should write to this attribute, but read both. E.g. for a file named ".hidden" you could do:

  attribs.hidden = false;

and still

  assert(attribs.hidden == true);

due to the POSIX file naming taking precedence then.

> > Or looking at it another way:
> > DOS attr < POSIX chmod < ACLs
> >
> > How do other programming languages find a common ground?
> 
> On the top of the documentation of the File class in Ruby, it says the following:
> 
> "In the description of File methods, permission bits are a platform-specific set of bits that indicate permissions of a file."
> 
> And
> 
> "On non-Posix operating systems, there may be only the ability to make a file read-only or read-write. In this case, the remaining permission bits will be synthesized to resemble typical values."

Ruby assumes POSIX semantics and directly exposes chmod() and
chown(). Probably because the level of permission control
that POSIX offers is good enough? (Even though that only
allows setting the read-only bit on Windows.)

o they offer one method to set all permission bits in one go
  (chmod)
o other bits (e.g. hidden) cannot be set
o attributes can be queried through a series of separate
  methods: readable, readable_real, world_readable, writable,
  writable_real, world_writable, executable, owned, grpowned.

On Windows grpowned is just: return false;

-- 
Marco

On Saturday, 28 December 2013 at 08:18:44 UTC, Jonathan M Davis wrote:
> on it specifically. My main point was that some stuff in
> std.file is system-
> specific and that if it has to be, it's better to have it in
> std.file as system-
> specific rather than not having it at all just because it
> couldn't be
> completely cross-platform and portable. So, just because
> setFileAttributes is
> doing something system-specific does not mean that it's bad.

Agreed. And really I didn't want to make a point about how it should be done right when I started this thread. This particular function just seemed not in line with Phobos principles as I remember them. :)

Am Sat, 28 Dec 2013 09:02:01 +0000
schrieb "Ola Fosheim Grøstad"
<ola.fosheim.grostad+dlang@gmail.com>:
> It is bad. If you want system-specific behaviour you should have a separate interface that provide all the advantages that going to a lower level provides. Having half-assed OS-specific support is too pragmatic and will lead to a legacy mess in the long run when those interfaces become obsolete.
> 
> A good file abstraction should also support newer file systems like Google Cloud Storage, though. GCS does not support append() or directories.
> 
> A good file abstraction should also provide mechanisms to deal
> with different levels of consistency on the underlying filesystem
> and caching-mechanism:
> - read after write might return an old version
> - read after write returns the new version if read from the same
> computer
> - read after write always returns the new version
> 
> As a new language D should support Cloud based environments out-of-the-box with a hierarchy of functionality down to the peculiarities of Windows/Posix etc so that you can decide to code in a manner that supports all platforms without having to litter your code with ifs and version-statements… and also restrict you to the subset you have "authorized" so that you can develop on your Windows system and port it to the cloud with just a recompile without knowing the details of the cloud file system. (i.e. the compiler warns you when you are using system specific functionality).
> 
> I also think good file system support implies built-in caching and let the library decide whether to turn library-level caching on or off based on what the operating system or underlying interface supports. E.g. the ability to tell the system that you want successive reads of a file to be available fast at the cost of consistency.

Are you hijacking this thread to ask for Google Cloud
support? ;) Don't forget FTP, SSH, WebDAV, ...
Implementing all of this can take quite some time and will
result in something like the "Gnome virtual file-system". It
might become so large that std.file is easily reimplemented as
a tiny part of it for the native file-system module.

Basic OS level file-system I/O support is useful on its own, especially in a systems programming language. You don't need to pull in a whole bunch of dependencies to read a text file.

-- 
Marco

On Sunday, 29 December 2013 at 11:01:05 UTC, Marco Leise wrote:
> Basic OS level file-system I/O support is useful on its own,
> especially in a systems programming language. You don't need
> to pull in a whole bunch of dependencies to read a text file.

Yes, it is useful to have good bindings for OS-level apis. However, Posix is no longer an adequate abstraction for cross-platform file systems. In the cloud or in clusters you mount network drives with other properties than a local drive.

It is much better to encourage a high-level interface for general file access. That encourage a more portable design for an application. That makes porting much easier. Porting software that assumes a local drive to the cloud is tedious.

You don't pull in whole bunch of dependencies. You include the modules you need and they share the same interface. That's a clean design.

The file module reminds me of Perl and old Php, that was ok in the 1990s, but it is the wrong way of creating a file system abstraction in 2014.

If you just want to read and write a file with a specific path there is no reason for using a non-portable interface. Encourage a portable interface and porting to the cloud becomes trivial: you just swap out the context-object.

On 2013-12-29 11:36, Marco Leise wrote:

> Oh right, they have a hidden attribute as well. I guess if
> Phobos should expose the 'hidden' state of a file it should
> write to this attribute, but read both. E.g. for a file
> named ".hidden" you could do:
>
>    attribs.hidden = false;
>
> and still
>
>    assert(attribs.hidden == true);
>
> due to the POSIX file naming taking precedence then.

Yeah, but that seems quite confusing. I don't know if it would be wise to use the same function for that. It would probably also be confusing if "attribs.hidden = false" renamed the file to remove the leading dot, if present.

-- 
/Jacob Carlborg

On 2013-12-29 12:00, Marco Leise wrote:

> Are you hijacking this thread to ask for Google Cloud
> support? ;) Don't forget FTP, SSH, WebDAV, ...
> Implementing all of this can take quite some time and will
> result in something like the "Gnome virtual file-system". It
> might become so large that std.file is easily reimplemented as
> a tiny part of it for the native file-system module.

Tango already contains a virtual file system interface. If I recall correctly it supports FTP and regular, local, file systems.

-- 
/Jacob Carlborg