August 19, 2013
On Monday, 19 August 2013 at 12:49:48 UTC, Jacob Carlborg wrote:
> I have had a brief look at Protocol Buffers and I don't see why it wouldn't work as an archive. I would probably need to implement a Protocol Buffers archive type to see what the limitations of std.serialization and Protocol Buffers are.

You can find the Protocol Buffers library here, may be it helps:
https://256.makerslocal.org/wiki/index.php/ProtocolBuffer
August 19, 2013
On 2013-08-19 15:03, Dicebot wrote:

> Great! Are there any difficulties with the input?

It just that I don't clearly know how the code will need to look like, and I'm not particular familiar with implementing range based code.

-- 
/Jacob Carlborg
August 19, 2013
On Monday, 19 August 2013 at 13:31:27 UTC, Jacob Carlborg wrote:
> On 2013-08-19 15:03, Dicebot wrote:
>
>> Great! Are there any difficulties with the input?
>
> It just that I don't clearly know how the code will need to look like, and I'm not particular familiar with implementing range based code.

Ok, I'll investigate related part of package a bit more in details during this week and see if I can suggest something.
August 19, 2013
On Monday, 19 August 2013 at 13:31:27 UTC, Jacob Carlborg wrote:
> On 2013-08-19 15:03, Dicebot wrote:
>
>> Great! Are there any difficulties with the input?
>
> It just that I don't clearly know how the code will need to look like, and I'm not particular familiar with implementing range based code.

Maybe we need some kind of doc explaining the idiomatic usage of ranges?

Personally, I'd like to do something like this:

    auto archive = new XmlArchive!(char); // create an XML archive
    auto serializer = new Serializer(archive); // create the serializer
    serializer.serialize(foo);

    pipe(archive.out, someFile);

Where pipe would read from the left and write to the right. My idea for an implementation is through using take():

    void pipe(R) (R input, File output) // isInputRange(R)...
    {
        while (!input.empty) {
            // if Serializer has no data cached, goes through one step
            // and returns what it has
            auto arr = input.take(BUF_SIZE);
            input.popFrontN(arr.length);
            output.write(arr);
        }
    }

For now, I'd be happy for serializer to process all data in serialize(), but change the behavior later to do step through computation when calling take().

I don't know if this helps, and others are very likely to have better ideas.
August 19, 2013
On Sunday, 18 August 2013 at 20:33:01 UTC, Jonathan M Davis wrote:
> On Sunday, August 18, 2013 21:45:59 Jacob Carlborg wrote:
>> If versioning is crucial it can be added.
>
> I don't know if it's crucial or not, but I know that the Java guys didn't have
> it initially but ended up adding it later, which would imply that they ran
> into problems that made them decide that it should be there. I'd certainly be
> inclined to think that it's better to have it, and it's probably easier to add
> it before it's merged than later. But I don't know how crucial it is.
>
> - Jonathan M Davis

I think this versioning idea is more important for protocol buffers, msgpck, thrift like libraries that use a separate IDL schema and IDL-compiled code. std.serialization uses the D code itself to serialize so the version is practically dictated by the user. It may as well be manually handled....as long as it throws/returns error and doesn't crash if one tries to deserialize an archive type into a different/modified D type.

From memory the Protocol Buffers versioning is to ensure schema generated code and library are compatible. You get compile errors if you try to compile IDL generated code with a newer version of the library. Similarly you get runtime errors if you deserialize data that was serialized with an older version of the library. This is all from memory so I could be wrong...

Orange seems/feels more like the BOOST.serialization to me but much better. It's D for a start and allows custom archive types. In BOOST, the library stores a version number in the archive for each class serialized. This number defaults to 0 but can be set by the user via a #define.

http://www.boost.org/doc/libs/1_54_0/libs/serialization/doc/tutorial.html#versioning

I think adding it later can be done without breaking existing API, if it is deemed necessary. It just needs to default to 0 or something similar when missing from an archive and ensure it won't clash with any fields in existing archives.

August 19, 2013
On 2013-08-19 15:47, Dicebot wrote:

> Ok, I'll investigate related part of package a bit more in details
> during this week and see if I can suggest something.

What I have now is something like this:

auto foo = new Foo;
foo.a = 3;

auto archive = new XmlArchive!(string); // string is the range type
auto serializer = new Serializer(archive);

serializer.serialize(foo);
auto data = archive.data; // returns a range, typed as XmlArchiveData

The problem now is that the range type is "string", so I can't set the data using any other range type:

archive.data = data;

Results in:

Error: cannot implicitly convert expression (range) of type XmlArchiveData to string

How can I handle that?

-- 
/Jacob Carlborg
August 19, 2013
On Monday, 19 August 2013 at 12:49:48 UTC, Jacob Carlborg wrote:
> I have had a brief look at Protocol Buffers and I don't see why it wouldn't work as an archive. I would probably need to implement a Protocol Buffers archive type to see what the limitations of std.serialization and Protocol Buffers are.

I not familiar with the interaction of Archive and Serializer. I was overwhelmed by the number of functions I'd have to implement (or in my case ignore) and ultimately I didn't know what my serialized data would look like.

I think it is possible to output a binary format which uses the same translation as Protocol Buffers, but I wouldn't expect it to resemble a message.

>> Thrift and Protocol Buffers use code generation to create the language
>> data type, and at least for Protocol Buffers a method contains all the
>> logic for deserializing a collection of bytes, and one for serializing.
>> I'm not seeing how std.serialize would make this easier or more usable.
>
> If a Thrift or Protocol Buffers archive would be used with std.serialization I'm thinking that one would skip that step and have the data types defined directly in D.

I'll see if I can push my way through creating an Archive type.
August 19, 2013
On Monday, 19 August 2013 at 13:17:48 UTC, ilya-stromberg wrote:
> On Monday, 19 August 2013 at 12:49:48 UTC, Jacob Carlborg wrote:
>> I have had a brief look at Protocol Buffers and I don't see why it wouldn't work as an archive. I would probably need to implement a Protocol Buffers archive type to see what the limitations of std.serialization and Protocol Buffers are.
>
> You can find the Protocol Buffers library here, may be it helps:
> https://256.makerslocal.org/wiki/index.php/ProtocolBuffer

Code has moved to https://github.com/opticron/ProtocolBuffer
August 19, 2013
On 2013-08-19 17:40, Jesse Phillips wrote:

> I not familiar with the interaction of Archive and Serializer. I was
> overwhelmed by the number of functions I'd have to implement (or in my
> case ignore) and ultimately I didn't know what my serialized data would
> look like.

std.serialization basically support any type in D (except for delegates and function pointers). If a particular method doesn't make sense to implement for a given archive, just implement a dummy function to satisfy the interface. The documentation for Archive says so:

"When implementing a new archive type, if any of these methods do not make sense for that particular implementation just implement an empty method and return T.init, if the method returns a value."

If something breaks due to this please let me know.

> I think it is possible to output a binary format which uses the same
> translation as Protocol Buffers, but I wouldn't expect it to resemble a
> message.

In the binary archive I'm working on I have chosen to ignore some parts of the implicit contract between the serializer and the archive. For example, I'm not planning to support slices, pointers to fields and similar complex features.

-- 
/Jacob Carlborg
August 19, 2013
On 2013-08-19 15:03, Dicebot wrote:

> This thread:
> http://forum.dlang.org/post/xqklcesoguxujifijadp@forum.dlang.org

I have removed all uses of "mixin annotations".

-- 
/Jacob Carlborg