Am Sun, 29 Dec 2013 13:44:02 +0100
schrieb Jacob Carlborg <doob@me.com>:

> On 2013-12-29 11:36, Marco Leise wrote:
> 
> > Oh right, they have a hidden attribute as well. I guess if Phobos should expose the 'hidden' state of a file it should write to this attribute, but read both. E.g. for a file named ".hidden" you could do:
> >
> >    attribs.hidden = false;
> >
> > and still
> >
> >    assert(attribs.hidden == true);
> >
> > due to the POSIX file naming taking precedence then.
> 
> Yeah, but that seems quite confusing. I don't know if it would be wise to use the same function for that. It would probably also be confusing if "attribs.hidden = false" renamed the file to remove the leading dot, if present.

File renaming is out of question. Just like you wouldn't remove the .exe extension to make a file non-executable on Windows.

-- 
Marco

On 12/29/13 4:20 AM, "Ola Fosheim Grøstad" <ola.fosheim.grostad+dlang@gmail.com>" wrote:
> On Sunday, 29 December 2013 at 11:01:05 UTC, Marco Leise wrote:
>> Basic OS level file-system I/O support is useful on its own,
>> especially in a systems programming language. You don't need
>> to pull in a whole bunch of dependencies to read a text file.
>
> Yes, it is useful to have good bindings for OS-level apis. However,
> Posix is no longer an adequate abstraction for cross-platform file
> systems. In the cloud or in clusters you mount network drives with other
> properties than a local drive.

Well one question is what other successful designs could we use as precedent? I don't know of any successful unified APIs for regular/remote filesystems that also allow full local file functionality. The closest abstraction I know of is the FUSE-installed filesystems, but those adapt foreign file systems to the Posix interface.


Andrei

On Sunday, 29 December 2013 at 15:05:31 UTC, Andrei Alexandrescu wrote:
> Well one question is what other successful designs could we use as precedent? I don't know of any successful unified APIs for regular/remote filesystems that also allow full local file functionality. The closest abstraction I know of is the FUSE-installed filesystems, but those adapt foreign file systems to the Posix interface.

I guess file storage design is currently in flux and a bit lacking in terms of APIs, partially because of cloud computing and SSDs, so my suggestion would be to:

1. Provide os.linux, os.windows, os.osx, os.ios, os.posix for low level local-storage access which closely resembles the native/standard APIs.

2. Provide a novel hierarchy of abstraction for basic file I/O that is highly portable and afford modular and extensible implementation, and make those really easy to use in order to encourage portable libraries for D. I don't think the overhead of function calls through interfaces matters all that much because the cost associated with context shifts for system calls will dominate that by a solid margin (my assumption).

So basically on the most abstract and limited level you get a key/value cache (memcache). The next level is key/value datastore (couchdb/mongodb/amazon/google etc).
Then the ability to list all keys.
Then the ability list keys based on hiearchy (directories) etc.

If you can specify what kind of functionality your application needs and set those constraints in a file-system context object then you can set the level of portability for your application. I think this would be a real advantage. Then let that file-system object have a factory for "file-mode" and let that do the right thing when you attempt to set the filetype as a MIME-constant (e.g set the right extension or set the right attribute).

Another advantage is that you could create "safe virtual filesystems" e.g. create a file-system context object that resolve ".." and prefix all paths with "/tmp/" and set default "file-mode" for created objects (like read-only and owner). Which would make it easy to write more secure web-servers. You could even specify a name-mangler and create a db-like filesystem for arbitrary keys with automatic creation of directories (for efficiency) if the abstraction level is right.

I don't think the overhead is all that much if it is done in a modular fashion. You only import what you need. Finding the right abstraction is not trivial, I agree. So you need to do some analysis of what current storage-solutions provide to find the common denominators.

(Having done some work with ndb on App Engine and the HttpRequest stuff in Dart I see the advantage of wrapping up resource loading as Futures. Mostly sugar, but uniform and easy to remember if used on all resources. E.g. request all resources as futures, then process them as they become available.)

On 12/29/13 7:43 AM, "Ola Fosheim Grøstad" <ola.fosheim.grostad+dlang@gmail.com>" wrote:
> On Sunday, 29 December 2013 at 15:05:31 UTC, Andrei Alexandrescu wrote:
>> Well one question is what other successful designs could we use as
>> precedent? I don't know of any successful unified APIs for
>> regular/remote filesystems that also allow full local file
>> functionality. The closest abstraction I know of is the FUSE-installed
>> filesystems, but those adapt foreign file systems to the Posix interface.
>
> I guess file storage design is currently in flux and a bit lacking in
> terms of APIs, partially because of cloud computing and SSDs, so my
> suggestion would be to:
>
> 1. Provide os.linux, os.windows, os.osx, os.ios, os.posix for low level
> local-storage access which closely resembles the native/standard APIs.

I think we (and others) have done a fine job so far at abstracting away e.g. very different ways of figuring whether an entry is a directory on Windows vs. Posix. Now we're to throw all that away and go back to silex stones and bear claws?

> 2. Provide a novel hierarchy of abstraction for basic file I/O that is
> highly portable and afford modular and extensible implementation, and
> make those really easy to use in order to encourage portable libraries
> for D.

Yah, it would be great if that were awfully more detailed :o).

> I don't think the overhead of function calls through interfaces
> matters all that much because the cost associated with context shifts
> for system calls will dominate that by a solid margin (my assumption).

That's exactly what they said when they designed iostreams.

> So basically on the most abstract and limited level you get a key/value
> cache (memcache). The next level is key/value datastore
> (couchdb/mongodb/amazon/google etc).
> Then the ability to list all keys.
> Then the ability list keys based on hiearchy (directories) etc.
>
> If you can specify what kind of functionality your application needs and
> set those constraints in a file-system context object then you can set
> the level of portability for your application. I think this would be a
> real advantage. Then let that file-system object have a factory for
> "file-mode" and let that do the right thing when you attempt to set the
> filetype as a MIME-constant (e.g set the right extension or set the
> right attribute).
>
> Another advantage is that you could create "safe virtual filesystems"
> e.g. create a file-system context object that resolve ".." and prefix
> all paths with "/tmp/" and set default "file-mode" for created objects
> (like read-only and owner). Which would make it easy to write more
> secure web-servers. You could even specify a name-mangler and create a
> db-like filesystem for arbitrary keys with automatic creation of
> directories (for efficiency) if the abstraction level is right.
>
> I don't think the overhead is all that much if it is done in a modular
> fashion. You only import what you need. Finding the right abstraction is
> not trivial, I agree. So you need to do some analysis of what current
> storage-solutions provide to find the common denominators.
>
> (Having done some work with ndb on App Engine and the HttpRequest stuff
> in Dart I see the advantage of wrapping up resource loading as Futures.
> Mostly sugar, but uniform and easy to remember if used on all resources.
> E.g. request all resources as futures, then process them as they become
> available.)

Again I think we'd need a ton of detail here to even assess whether this has merit. So far it's an interesting brain dump.


Andrei

On Sunday, 29 December 2013 at 16:06:32 UTC, Andrei Alexandrescu wrote:
> I think we (and others) have done a fine job so far at abstracting away e.g. very different ways of figuring whether an entry is a directory on Windows vs. Posix. Now we're to throw all that away and go back to silex stones and bear claws?

I am not telling anyone what to do. I am saying what I would prefer. I would prefer a standard low-level layer and a flexible high-level layer.

> Yah, it would be great if that were awfully more detailed :o).

I am not coming with a finished design, but I think it is doable.

> That's exactly what they said when they designed iostreams.

Buffered i/o is another issue because you either have a high interface-call/system call ratio, or make too many system-calls.

> Again I think we'd need a ton of detail here to even assess whether this has merit. So far it's an interesting brain dump.

Ok, let me give you some motivation:

Old software bases tend to evolve and make assumptions that do not hold over time. I recently ported phpbb to App Engine, which provides a read-only filesystem. Phpbb used the following:

1. templates: on filesystem, or if modified in mysql
2. avatar images: on filesystem
3. processed templates: cached on filesystem

In order to port I had to create my own filesystem abstraction and map processed templates onto memcache and avatar images onto Cloud Storage. Basically sifting through the code and hoping I didn't overlook anything.

This file-system port would have been trivial if phpbb had instantiated separate "virtual filesystems" for templates, avatars and processed templates.

I could then have specified that the avatar-filesystem should default to MIME-type 'image/jpeg' which would let me serve the images directly from Cloud Storage (through Google's efficient infrastructure) with little extra work (low chance of creating bugs).

In most cases you don't need to know what filesystem different entities reside on. When moving into the cloud you most likely will gain access to a diverse set of storage mechanisms that are optimized for different usage patterns (like on-the-fly rescaling of images, direct http interfaces, memcaches, redundante high-availability storage etc).