Maybe we should have cast a vote to convince Walter to deprecate std.stream and add mango.io to phobos. Please reply only with +1 / -1 / 0.

+1

+1
"Unknown lurker" <Unknown_member@pathlink.com> wrote in message
news:cbepgc$1bbh$1@digitaldaemon.com...
> Maybe we should have cast a vote to convince Walter to deprecate
std.stream and
> add mango.io to phobos. Please reply only with +1 / -1 / 0.
>
> +1
>
>

Unknown lurker wrote:

> Maybe we should have cast a vote to convince Walter to deprecate std.stream and add mango.io to phobos. Please reply only with +1 / -1 / 0.
> 
> +1

I'd first like to see what Sean does with std.stream plus I tend to agree with Matthew that more discussion is needed before jumping too soon in any one direction. There's a lot of cool stuff in mango that I'd love to see somehow merged with std.stream if possible. Maybe I'll give a poke at porting Kris's tokenizers and endian stuff to std.stream just to see what it looks like.

This will probably open up rat-holes, but two quick examples of things to discuss:

1) in mango it looks like to open a file and read it you need to create a FileConduit and pass that to a Reader constructor. So you have to grok the difference between Conduits and Readers/Writers (and maybe Buffers? I notice IConduit has a createBuffer method so is it not buffered by default? I'm not sure). In std.stream you make one object and there is less to grok. The flexibility of mango is probably nice but it adds complexity. Each person has a different notion of where to draw the boundaries.

2) in mango to use object serialization/deserialization you register an instance of a class so that means at startup you basically have to instantiate one of every class that might want to be deserialized. Seems wastful and it could affect class design to avoid having classes that have interdependencies.

In article <cbf0lp$1lvi$1@digitaldaemon.com>, Ben Hinkle says...

>I'd first like to see what Sean does with std.stream plus I tend to agree with Matthew that more discussion is needed before jumping too soon in any one direction. There's a lot of cool stuff in mango that I'd love to see somehow merged with std.stream if possible. Maybe I'll give a poke at porting Kris's tokenizers and endian stuff to std.stream just to see what it looks like.

I'd be quite happy if std.stream were to be improved. Here are some suggestions. You'll probably think that many of them are trivial, but each, in their own way, contributes a small amount of annoyance, and I'm sure these things could be easily got rid of.

1) Since it is more normal to want buffered file access than non-buffered file
access (in C, fopen() is called more often than open()), it makes sense that
File should be buffered by default, and there should be a separate class, maybe
called RawFileStream or something, for the unbuffered case.

2) File should in any case be renamed FileStream

3) FileMode.In and FileMode.Out should be renamed Filemode.IN and Filemode.OUT respectively.

4) It should be possible to construct a File object in create mode, in one step.
As in File f = new File(filename, FileMode.CREATE);

5) In fact, all possible combinations of file opening supported by fopen() should be supported by File. It should be possible to assert that the file does or does not exist before opening it (atomically), to truncate or not truncate, to position the file pointer at the start or end of the file, to allow append-only access, etc.

6) The destructor should always close the file

7) EITHER Stream classes should be auto (likely to be an unpopular suggestion, I know), OR there should be an auto wrapper class that you can construct from a Stream, in order to guarantee that the file will be closed in the event of an exception (which could of course be thrown by ANY piece of code). Currently we have to either roll our own auto wrapper, or use a try/catch block.

8) Documentation should be complete and accurate.

9) There should be a FilterStream class, from which BufferedStream inherits, so
that we can write our own stream filters. (Java does this. It's neat).

10) Streams don't necessarily have to do transcoding (see - I learnt a new word), but nonetheless it should be POSSIBLE to construct them from a Reader/Writer in order to make such extensions possible in the future.

11) I want the function available(), as Java has. A buffered stream always knows how much it's got left in its buffer, and I have no problem with an unbuffered stream returning zero.

12) stdin, stdout and stderr should be globally available D streams. (Maybe they
are already, but point (8) means there's a lot I don't know about existing
capabilities)

13) Streams should overload the << and >> operators. (Someone suggested using ~.
That would be fine too).


None of these is particularly difficult in and of itself, but together they add up to a frustrating gripe list. But I'm fairly confident that if these flaws are fixed (along with any other gripes which others may mention in the course of this thread) then I imagine that most people will be pretty happy with new improved std.stream.





>This will probably open up rat-holes, but two quick examples of things to discuss:
>
>1) in mango it looks like to open a file and read it you need to create a FileConduit and pass that to a Reader constructor. So you have to grok the difference between Conduits and Readers/Writers (and maybe Buffers? I notice IConduit has a createBuffer method so is it not buffered by default? I'm not sure). In std.stream you make one object and there is less to grok. The flexibility of mango is probably nice but it adds complexity. Each person has a different notion of where to draw the boundaries.

But there is logic behind it. Currently, D does no transcoding - that is, writeLine() will spit out raw UTF-8. Now that's fine if your output is going to a text file, but if it's going to a console, you're screwed. Now you COULD simplify this a bit by "automatically" encoding the output in the operating system default encoding - but that would just reverse the problem. Now, output to the console would be fine, but output destined to leave your machine and end up on someone else's machine (e.g. text file, socket, etc.) would also be similarly munged. UTF-8 is pretty much the best portable format, so ideally you only want to encode at the last minute, just before the stream hits the user.



>2) in mango to use object serialization/deserialization you register an instance of a class so that means at startup you basically have to instantiate one of every class that might want to be deserialized. Seems wastful and it could affect class design to avoid having classes that have interdependencies.

I'm not convinced that serialization necessarily has anything to do with streams. You could serialize to a string, or an in-memory buffer. I guess that would be faster for small objects but disadvantageous for very large ones. In any case, you don't need to decide on a firm serialization policy in order to make streams feel nice. That can come later, once we're happy with the basics.

Arcane Jill

In article <cbf0lp$1lvi$1@digitaldaemon.com>, Ben Hinkle says...
>
>I'd first like to see what Sean does with std.stream plus I tend to agree with Matthew that more discussion is needed before jumping too soon in any one direction. There's a lot of cool stuff in mango that I'd love to see somehow merged with std.stream if possible. Maybe I'll give a poke at porting Kris's tokenizers and endian stuff to std.stream just to see what it looks like.

I've gotten a bit bogged down with formatted i/o, but perhaps I'll cut some corners and just get a proof of concept done.  I'm mostly interested in this as something to spur further discussion anyway.

>This will probably open up rat-holes, but two quick examples of things to discuss:
>
>1) in mango it looks like to open a file and read it you need to create a FileConduit and pass that to a Reader constructor. So you have to grok the difference between Conduits and Readers/Writers (and maybe Buffers? I notice IConduit has a createBuffer method so is it not buffered by default? I'm not sure). In std.stream you make one object and there is less to grok. The flexibility of mango is probably nice but it adds complexity. Each person has a different notion of where to draw the boundaries.

Seems you have the same issues with Mango that I do :)  Though in this case I think it would be fairly simple to provide some standard wrapper classes that take care of this all invisibly.  If Mango were incorporated, this is something I'd push for.

>2) in mango to use object serialization/deserialization you register an instance of a class so that means at startup you basically have to instantiate one of every class that might want to be deserialized. Seems wastful and it could affect class design to avoid having classes that have interdependencies.

Haven't gotten this far with the design model, but thre must be a way to change this.  As you say, I can see this not working with interdependent classes.


Sean

In article <cbf8jg$221b$1@digitaldaemon.com>, Arcane Jill says...
>
>I'd be quite happy if std.stream were to be improved. Here are some suggestions. You'll probably think that many of them are trivial, but each, in their own way, contributes a small amount of annoyance, and I'm sure these things could be easily got rid of.
>
>1) Since it is more normal to want buffered file access than non-buffered file
>access (in C, fopen() is called more often than open()), it makes sense that
>File should be buffered by default, and there should be a separate class, maybe
>called RawFileStream or something, for the unbuffered case.

I was actually going to take a different approach and modify BufferedStream like so:

BuferedStream( BaseStream ) : BaseStream {
// override the low-level i/o methods to do buffering
}

So a buffered file stream would be:

alias BufferedStream!(FileStream) BufferedFileStream;

But as you say, file i/o is almost always buffered, so it may make sense to change the name of "FileStream" to "UnbufferedFileStream" and thus make buffered file i/o the default.

>2) File should in any case be renamed FileStream
>
>3) FileMode.In and FileMode.Out should be renamed Filemode.IN and Filemode.OUT respectively.

Both already done :)  Well, it was going to be Stream.IN and Stream.OUT, but same thing.

>5) In fact, all possible combinations of file opening supported by fopen() should be supported by File. It should be possible to assert that the file does or does not exist before opening it (atomically), to truncate or not truncate, to position the file pointer at the start or end of the file, to allow append-only access, etc.

Right now the file stuff uses CreateFile in Windows.  Would it be better to use fopen and the other ANSI calls instead?

>7) EITHER Stream classes should be auto (likely to be an unpopular suggestion, I know), OR there should be an auto wrapper class that you can construct from a Stream, in order to guarantee that the file will be closed in the event of an exception (which could of course be thrown by ANY piece of code). Currently we have to either roll our own auto wrapper, or use a try/catch block.

Interesting idea.  This may be another good template wrapper:

auto class MakeAuto( BaseClass ) : BaseClass {}

>9) There should be a FilterStream class, from which BufferedStream inherits, so
>that we can write our own stream filters. (Java does this. It's neat).
>
>10) Streams don't necessarily have to do transcoding (see - I learnt a new word), but nonetheless it should be POSSIBLE to construct them from a Reader/Writer in order to make such extensions possible in the future.

This kid of stuff should be saved for a later discussion.  All great ideas but they're the tip of a rather large iceberg.

>11) I want the function available(), as Java has. A buffered stream always knows how much it's got left in its buffer, and I have no problem with an unbuffered stream returning zero.

Easy enough.  I was going to add this to the BufferedStream class, though perhaps it would be useful everywhere?

>12) stdin, stdout and stderr should be globally available D streams. (Maybe they
>are already, but point (8) means there's a lot I don't know about existing
>capabilities)

I think they are.  I kind of consider Phobos to still be in the state where looking at the source files is best way to find out what's available.  Doxygen or other documentation is crucial, but I have a feeling that the stream API will be in flux for quite some time yet.

>13) Streams should overload the << and >> operators. (Someone suggested using ~.
>That would be fine too).

Overloading ~ wouldn't work I'm afraid, unless there's something I'm missing. Say I have a FileStream and I want to both read and write from it.  How do I know which I want to do if I'm using the same operator for both?

>None of these is particularly difficult in and of itself, but together they add up to a frustrating gripe list. But I'm fairly confident that if these flaws are fixed (along with any other gripes which others may mention in the course of this thread) then I imagine that most people will be pretty happy with new improved std.stream.

I agree.  But as D is still pretty early in its development I don't really want people to be happy with anything if it means losing constructive dialog.  I'm willing to sacrifice productivity in the short term if it means a better library in the long term.

>But there is logic behind it. Currently, D does no transcoding - that is, writeLine() will spit out raw UTF-8. Now that's fine if your output is going to a text file, but if it's going to a console, you're screwed. Now you COULD simplify this a bit by "automatically" encoding the output in the operating system default encoding - but that would just reverse the problem. Now, output to the console would be fine, but output destined to leave your machine and end up on someone else's machine (e.g. text file, socket, etc.) would also be similarly munged. UTF-8 is pretty much the best portable format, so ideally you only want to encode at the last minute, just before the stream hits the user.

This is the start of a rather long discussion.  I've had enough issues come up with formatted i/o that I'm going to leave that aspect rather bare and see what develops.  For the moment, I'm starting to think that handling plain ol' ASCII is probably enough until the rest can be worked out.


Sean

Oops, something went wrong.  That 's' person is me.

Sean

-1

Premature

"Unknown lurker" <Unknown_member@pathlink.com> wrote in message news:cbepgc$1bbh$1@digitaldaemon.com...
> Maybe we should have cast a vote to convince Walter to deprecate std.stream and add mango.io to phobos. Please reply only with +1 / -1 / 0.
>
> +1
>
>

"Ben Hinkle"  wrote
> I'd first like to see what Sean does with std.stream plus I tend to agree with Matthew that more discussion is needed before jumping too soon in any one direction. There's a lot of cool stuff in mango that I'd love to see somehow merged with std.stream if possible. Maybe I'll give a poke at porting Kris's tokenizers and endian stuff to std.stream just to see what it looks like.

While it's truly great (and encouraging) to see support for mango.io, I also fully agree with Matthew on this point also. It's too soon for either package to be the "standard", although I do have a bias. I would encourage those who like mango.io to please lend a hand in making it even better <G>

> 1) in mango it looks like to open a file and read it you need to create a FileConduit and pass that to a Reader constructor. So you have to grok the difference between Conduits and Readers/Writers (and maybe Buffers? I notice IConduit has a createBuffer method so is it not buffered by
default?
> I'm not sure). In std.stream you make one object and there is less to
grok.
> The flexibility of mango is probably nice but it adds complexity. Each person has a different notion of where to draw the boundaries.

Good point. The long way is to do the following:

FileConduit fc = new FileConduit ("foo.bar", FileStyle.ReadExisting);
Reader reader = new Reader (fc);

reader >> x >> y >> x; // or reader.get(x).get(y).get(z);

The reason for the split is to allow flexibility in what kind of reader you want to use (binary, text, regex, endian, whatever). It also opens up a nice way of doing Reader chaining, plus it allows one to pass any kind of IConduit to some method, and said method can decide how it wishes to 'communicate' by applying a Reader (and/or Writer) of its choice. I've always preferred having the choice of decoupled processing in this manner.

There's nothing to prevent some simple wrappers being provided, such as this:

TextReader tr = new TextFileReader ("foo.bar");  // or whatever

There is also a split underneath the Conduit covers which seperates the Buffer itself from the Conduit. This allows one to apply any and all Readers & Writers directly upon a simple Buffer (without a socket or file attached) so there's complete symmetry in how you manipulate memory-based idioms. This is quite different from std.outbuffer, and also directly supports the notion of asking the FileConduit for a memory-mapped view instead of a standard (buffered) view.

There's usually a trade-off regarding flexibility and ease-of-use.Wrappers are a great idea to make it trivial for newbies. But the "long way" is pretty simple too. I really hope mango.io can achieve both :-)

The dependency chain is thus: IReader/IWriter => IBuffer => IConduit; where both left and right sides are optional. That is, the central crux of the mango.io is IBuffer. There a document over here that discusses it a bit further: http://svn.dsource.org/svn/projects/mango/trunk/Mango%20IO%20overview.pdf

>
> 2) in mango to use object serialization/deserialization you register an instance of a class so that means at startup you basically have to instantiate one of every class that might want to be deserialized. Seems wastful and it could affect class design to avoid having classes that have interdependencies.

Yeah, there's been talk of using an Interface to do that instead. There's more on that over here http://www.dsource.org/forums/viewtopic.php?p=1264#1264

I'd also like to add that mango.io does not grok unicode at this time, cos' I think it's better to take advantage of what Hauke and AJ provide. There's a number of other missing or bogus items that you can check out over here (in the "Mango Sucks ..." thread) http://www.dsource.org/forums/viewtopic.php?t=157

I would encourage folks to add their own gripes to the (growing) list <g>

- Kris

On Thu, 24 Jun 2004 19:04:16 +0000 (UTC), Arcane Jill <Arcane_member@pathlink.com> wrote:
> In article <cbf0lp$1lvi$1@digitaldaemon.com>, Ben Hinkle says...
>
>> I'd first like to see what Sean does with std.stream plus I tend to agree
>> with Matthew that more discussion is needed before jumping too soon in any
>> one direction. There's a lot of cool stuff in mango that I'd love to see
>> somehow merged with std.stream if possible. Maybe I'll give a poke at
>> porting Kris's tokenizers and endian stuff to std.stream just to see what
>> it looks like.
>
> I'd be quite happy if std.stream were to be improved. Here are some suggestions.
> You'll probably think that many of them are trivial, but each, in their own way,
> contributes a small amount of annoyance, and I'm sure these things could be
> easily got rid of.
>
> 1) Since it is more normal to want buffered file access than non-buffered file
> access (in C, fopen() is called more often than open()), it makes sense that
> File should be buffered by default, and there should be a separate class, maybe
> called RawFileStream or something, for the unbuffered case.

I'd call it RawFile, and it should mirror open/close etc.

> 2) File should in any case be renamed FileStream

I think it should stay File, and should mirror fopen etc.

If we want to stream it, we pass it into the constructor of a Stream or BufferedStream (File being buffered would not generally get passed to this as it would be a waste of time) etc.

> 3) FileMode.In and FileMode.Out should be renamed Filemode.IN and Filemode.OUT
> respectively.

I agree.

> 4) It should be possible to construct a File object in create mode, in one step.
> As in File f = new File(filename, FileMode.CREATE);

We need flags to allow us to mirror these fopen modes:

"r"
Opens for reading. If the file does not exist or cannot be found, the fopen call fails.

"w"
Opens an empty file for writing. If the given file exists, its contents are destroyed.

"a"
Opens for writing at the end of the file (appending) without removing the EOF marker before writing new data to the file; creates the file first if it doesn’t exist.

"r+"
Opens for both reading and writing. (The file must exist.)

"w+"
Opens an empty file for both reading and writing. If the given file exists, its contents are destroyed.

"a+"
Opens for reading and appending; the appending operation includes the removal of the EOF marker before new data is written to the file and the EOF marker is restored after writing is complete; creates the file first if it doesn’t exist.

So it seems we need:

"r"  - READ  - read, fails if file does not exist.
"w"  - CREATE - write, overwrite existing.
"a"  - APPEND - write, create if not exist.
"r+" - READ|WRITE - read, write, fails if file does not exist.
"w+" - READ|CREATE - read, write, overwrite existing.
"a+" - READ|APPEND - read, append, create if not exist.

> 5) In fact, all possible combinations of file opening supported by fopen()
> should be supported by File. It should be possible to assert that the file does
> or does not exist before opening it (atomically), to truncate or not truncate,
> to position the file pointer at the start or end of the file, to allow
> append-only access, etc.

Agreed. See above. Also might want

WRITE|NEW - write, fail if file exist.

> 6) The destructor should always close the file

As long as there is a way to dup the underlying handle so we can store and re-use it if desired then I agree.

> 7) EITHER Stream classes should be auto (likely to be an unpopular suggestion, I
> know), OR there should be an auto wrapper class that you can construct from a
> Stream, in order to guarantee that the file will be closed in the event of an
> exception (which could of course be thrown by ANY piece of code). Currently we
> have to either roll our own auto wrapper, or use a try/catch block.

Shouldn't file be auto? And closed in the destructor.

> 8) Documentation should be complete and accurate.

But of course. In an ideal world.. give it a little time. Perhaps the docs are not there because the author of std.stream is not yet happy with it?

> 9) There should be a FilterStream class, from which BufferedStream inherits, so
> that we can write our own stream filters. (Java does this. It's neat).

Definately.

> 10) Streams don't necessarily have to do transcoding (see - I learnt a new
> word), but nonetheless it should be POSSIBLE to construct them from a
> Reader/Writer in order to make such extensions possible in the future.

Why not simply use a filter as mentioned above to transcode?

> 11) I want the function available(), as Java has. A buffered stream always knows
> how much it's got left in its buffer, and I have no problem with an unbuffered
> stream returning zero.

Isn't this true for a normal unbuffered file as well. at the point of opening you know how big it is. it could grow.. but until you reach that initial size you know there is more or not etc.

> 12) stdin, stdout and stderr should be globally available D streams. (Maybe they
> are already, but point (8) means there's a lot I don't know about existing
> capabilities)

They are.

> 13) Streams should overload the << and >> operators. (Someone suggested using ~.
> That would be fine too).

I have never liked << and >>. I dont think ~ is quite the right thing. It mostly makes sense for writing to a stream, but not reading.

Whatever we use it should be efficient i.e. this statement

Stream s;

s ~= "regan" ~ "was" ~ "here"

should not append "here" to "was" then that to "regan" then send it to the stream, as this is inefficient esp in comparrison to a buffered stream.

> None of these is particularly difficult in and of itself, but together they add
> up to a frustrating gripe list. But I'm fairly confident that if these flaws are
> fixed (along with any other gripes which others may mention in the course of
> this thread) then I imagine that most people will be pretty happy with new
> improved std.stream.

Definately.

>> This will probably open up rat-holes, but two quick examples of things to
>> discuss:
>>
>> 1) in mango it looks like to open a file and read it you need to create a
>> FileConduit and pass that to a Reader constructor. So you have to grok the
>> difference between Conduits and Readers/Writers (and maybe Buffers? I
>> notice IConduit has a createBuffer method so is it not buffered by default?
>> I'm not sure). In std.stream you make one object and there is less to grok.
>> The flexibility of mango is probably nice but it adds complexity. Each
>> person has a different notion of where to draw the boundaries.
>
> But there is logic behind it. Currently, D does no transcoding - that is,
> writeLine() will spit out raw UTF-8. Now that's fine if your output is going to
> a text file, but if it's going to a console, you're screwed. Now you COULD
> simplify this a bit by "automatically" encoding the output in the operating
> system default encoding - but that would just reverse the problem. Now, output
> to the console would be fine, but output destined to leave your machine and end
> up on someone else's machine (e.g. text file, socket, etc.) would also be
> similarly munged. UTF-8 is pretty much the best portable format, so ideally you
> only want to encode at the last minute, just before the stream hits the user.

I think a stream should write raw what you give it, meaning you have to use a filter to trancode/convert etc your data to the correct format for the destination.

We can design standard filters for common things like the console etc, and these filters can be automatically plugged into stdin/stdout etc.

>> 2) in mango to use object serialization/deserialization you register an
>> instance of a class so that means at startup you basically have to
>> instantiate one of every class that might want to be deserialized. Seems
>> wastful and it could affect class design to avoid having classes that have
>> interdependencies.
>
> I'm not convinced that serialization necessarily has anything to do with
> streams. You could serialize to a string, or an in-memory buffer. I guess that
> would be faster for small objects but disadvantageous for very large ones. In
> any case, you don't need to decide on a firm serialization policy in order to
> make streams feel nice. That can come later, once we're happy with the basics.

I think serialization could be done with filters. Just like transcoding can. You just plug all the filters together in between your source and destination streams. simple as that.

Regan

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/