August 14, 2013
On Wednesday, 14 August 2013 at 19:23:51 UTC, Jacob Carlborg wrote:
> On 2013-08-14 21:11, Andrei Alexandrescu wrote:
>
>> I'm thinking some people may need to stream to/from large files and
>> would find the requirement of in-core representation limiting.
>
> Yes, I understand that. But currently I'm limited by std.xml.

Can you use another serialization format and supports file output for it? For example, can you use JSON, BSON or binary format?
August 15, 2013
On Wednesday, 14 August 2013 at 19:55:52 UTC, ilya-stromberg wrote:
> On Wednesday, 14 August 2013 at 19:23:51 UTC, Jacob Carlborg wrote:
>> On 2013-08-14 21:11, Andrei Alexandrescu wrote:
>>
>>> I'm thinking some people may need to stream to/from large files and
>>> would find the requirement of in-core representation limiting.
>>
>> Yes, I understand that. But currently I'm limited by std.xml.
>
> Can you use another serialization format and supports file output for it? For example, can you use JSON, BSON or binary format?

That's often not possible, especially when working with an external API.

When working with large files, it's much better to read the file in chunks so you can be processing the data while the platters are seeking. This isn't as big of a problem with SSDs, but you still have to wait for the OS. RAM usage is also an issue, but for me it's less of an issue than waiting for I/O.

Even if rotating media were to be phased out, there's still the problem of streaming data over a network.

std.xml will be replaced, but it shouldn't require breaking code to fix std.serialize.
August 15, 2013
On Wednesday, August 14, 2013 16:54:59 Dicebot wrote:
> std.serialization is not Orange and should not be considered as one. Once it is included into Phobos it is a brand new library and must be treated as such, no matter what origin it has. Users of Orange expect compatibility from Orange, not Phobos.

Agreed.

- Jonathan M Davis
August 15, 2013
On 2013-08-14 21:55, ilya-stromberg wrote:

> Can you use another serialization format and supports file output for
> it? For example, can you use JSON, BSON or binary format?

The idea of the library is that it can support multiple archive types. Currently only XML is implemented. I have been working on a binary archive for a while but I haven't finished it yet.

The actual format has nothing to do with file output.

-- 
/Jacob Carlborg
August 17, 2013
On Wednesday, 14 August 2013 at 13:43:50 UTC, Dicebot wrote:
> On Wednesday, 14 August 2013 at 13:28:42 UTC, glycerine wrote:
>> Wishful thinking aside, they are competitors.
>
> They are not. `std.serialization` does not and should not compete in Thrift domain.

Huh? Do you know what thrift does? Summary: Everything that
Orange/std.serialization does and more. To the point: Thrift
provides data versioning, std.serialization does not. In my book:
end of story, game over. Thrift is preffered choice. If you
are going to standardize something, standardize the Thrift
bindings so that the compiler doesn't introduce regressions
that break them, like happened from dmd 2.062 - present.

You don't provide any rationale for your assertion, so I can't
really respond more constructively until you do. Please
familiarize yourself with D's Thrift bindings, which work well
with dmd 2.061. Then provide a rationale for your conjecture.
August 17, 2013
On Saturday, 17 August 2013 at 08:29:37 UTC, glycerine wrote:
> On Wednesday, 14 August 2013 at 13:43:50 UTC, Dicebot wrote:
>> On Wednesday, 14 August 2013 at 13:28:42 UTC, glycerine wrote:
>>> Wishful thinking aside, they are competitors.
>>
>> They are not. `std.serialization` does not and should not compete in Thrift domain.
>
> Huh? Do you know what thrift does? Summary: Everything that
> Orange/std.serialization does and more. To the point: Thrift
> provides data versioning, std.serialization does not. In my book:
> end of story, game over. Thrift is preffered choice. If you
> are going to standardize something, standardize the Thrift
> bindings so that the compiler doesn't introduce regressions
> that break them, like happened from dmd 2.062 - present.
>
> You don't provide any rationale for your assertion, so I can't
> really respond more constructively until you do. Please
> familiarize yourself with D's Thrift bindings, which work well
> with dmd 2.061. Then provide a rationale for your conjecture.

I agree built in version support is important.

As for your other issues you mention:

a) do one thing, do it well.
b) modular is better than monolithic.
c) std.serialization is for serialization, no more no less.
d) Thrift is for scalable cross-language services development.
(and much more http://thrift.apache.org/)

Just because Thrift can serialize classes/structs doesn't mean std.serialization  should support RPC services or transport of serialized data.

I'd rather that was left for a separate module (or two or three) built on top of std.serialization.



August 17, 2013
On Saturday, 17 August 2013 at 08:29:37 UTC, glycerine wrote:
> On Wednesday, 14 August 2013 at 13:43:50 UTC, Dicebot wrote:
>> On Wednesday, 14 August 2013 at 13:28:42 UTC, glycerine wrote:
>>> Wishful thinking aside, they are competitors.
>>
>> They are not. `std.serialization` does not and should not compete in Thrift domain.
>
> Huh? Do you know what thrift does? Summary: Everything that
> Orange/std.serialization does and more. To the point: Thrift
> provides data versioning, std.serialization does not. In my book:
> end of story, game over. Thrift is preffered choice. If you
> are going to standardize something, standardize the Thrift
> bindings so that the compiler doesn't introduce regressions
> that break them, like happened from dmd 2.062 - present.
>
> You don't provide any rationale for your assertion, so I can't
> really respond more constructively until you do. Please
> familiarize yourself with D's Thrift bindings, which work well
> with dmd 2.061. Then provide a rationale for your conjecture.

Yes I know what Thrift does and that is not what it is needed here. Important things to consider:

1) Having bindings in standard library is discouraged, we have Deimos for that. There is only curl stuff and it is considered a bad solution as far as I am aware of.

2) Thrift covers very wide domain of tasks - protocol descriptions, inter-operation between different versions, cross-language operation. `std.serialization` is about one simple task - taking care of D type reflection to load/store them in some way.

3) UNIX-way. Standard library must provide small self-sufficient components that focus on doing one job and doing it good. It is up to user to combine those components to build behavior he needs in actual application. In that sense core of std.serialization must be evaluated from the point of view "does it allow me to add feature X?" instead of "does it have feature X?"

4) There are lot of different serialization use cases and often having something like Thrift is a huge overkill. Good standard library allows user to chose only functionality he actually needs, no more.

There is nothing wrong with your choice of Thrift - it just does not belong to std.serialization
August 17, 2013
On Saturday, 17 August 2013 at 08:29:37 UTC, glycerine wrote:
> On Wednesday, 14 August 2013 at 13:43:50 UTC, Dicebot wrote:
>> On Wednesday, 14 August 2013 at 13:28:42 UTC, glycerine wrote:
>>> Wishful thinking aside, they are competitors.
>>
>> They are not. `std.serialization` does not and should not compete in Thrift domain.
>
> Huh? Do you know what thrift does? Summary: Everything that
> Orange/std.serialization does and more. To the point: Thrift
> provides data versioning, std.serialization does not. In my book:
> end of story, game over. Thrift is preffered choice.

Thrift is the preferred choice when choosing a library for ALL your possible serialization needs and more. However, standard library modules are not about including every possible convenience, it's about providing solid building blocks for creating larger frameworks.

What you're suggesting leads directly to the clearly idiotic like "std.stdio sucks because it's doesn't have a printRainbows feature"
August 17, 2013
I'd like to start off by saying I don't really know what I want from a std.serialize module. I've done some work with a C# JSON serializer and dealt with Protocol Buffers.

This library looks to be providing a means to serialize any D data structure. It deals with pointers/class/struct/arrays... It is export format agnostic, while currently only XML is available, allowing for export to JSON or some binary form. Afterwards the data can return to the program through deserialization.

This is a use-case I don't think I've needed. Though I do see the value in it and would expect Phobos to provide such functionality.

What I'm not finding in this library is a way to support a 3rd party protocol. Such as those used in Thrift or Protocol Buffers. These specify some aspects of data layout, for example in Protocol Buffers arrays of primitives can be laid out in two forms [ID][value][ID][value] or [ID][Length][value][value].

Thrift and Protocol Buffers use code generation to create the language data type, and at least for Protocol Buffers a method contains all the logic for deserializing a collection of bytes, and one for serializing. I'm not seeing how std.serialize would make this easier or more usable.

When looking at the Archive module, I see that all the specific types get their own void function. I'm unclear on where these functions are supposed to archive to, and container types take a delegate which I suppose is a means for the archiver to place output around the field data.

In conclusion, I don't feel like I've said very much. I don't think std.serialize is applicable to Protocol Buffers or Thrift, and I don't know what benefit there would be if it was.
August 18, 2013
On Saturday, 17 August 2013 at 08:29:37 UTC, glycerine wrote:
> On Wednesday, 14 August 2013 at 13:43:50 UTC, Dicebot wrote:
>> On Wednesday, 14 August 2013 at 13:28:42 UTC, glycerine wrote:
>>> Wishful thinking aside, they are competitors.
>>
>> They are not. `std.serialization` does not and should not compete in Thrift domain.
>
> Huh? Do you know what thrift does? Summary: Everything that
> Orange/std.serialization does and more.

That's actually not true. Thrift does not serialize arbitrary object graphs, or any types with indirections, for that matter. This is by design, it would be hard to do this efficiently in all target languages, and contrary to Orange, performance is the main focus of Thrift.

> If you
> are going to standardize something, standardize the Thrift
> bindings so that the compiler doesn't introduce regressions
> that break them, like happened from dmd 2.062 - present.

On a related note, we desperately need to do something about this, especially since there seems to be an increased amount of interest in Thrift lately. For 2.061 and the previous releases, I always tested every beta against Thrift, and almost invariably found at least one bug/regression per release. However, for 2.062 and 2.063, I was busy with LDC (and other things) at the time and it seems like I forgot to run the tests.

The DMD 2.062+ error message (see https://issues.apache.org/jira/browse/THRIFT-2130) doesn't make much sense; I guess the best way of going about this would be to try to DustMite-reduce the problem first or to fire up DMD in gdb to see what exactly is tripping the recursive alias error.

David