I believe it's worth the try to get Python-style serialization simplicity to D by default. We use Python cpickle at work and its fantastically simple and fast. It handles just about everthing you need by default as long as you have imported the modules that contain the types involved in the serialization.

It would attract more people from dynamical languages like Python if D got default-powers in std.serialization that does everything that msgpack does plus followings references.

This would make it possible to serialize arbitrary graph-likes structures with potential reference cycles. Resolving these cycles can be solved by using a map that contain a list of references/classs that already have serialized.

I'm however not sure how serialization of base and superclasses should be implemented in the most convenient way. Maybe somehow has a good suggestion for this.

On 2/16/14, "Nordlöw" <per.nordlow@gmail.com> wrote:
> I'm however not sure how serialization of base and superclasses should be implemented in the most convenient way. Maybe somehow has a good suggestion for this.

One way to do it: https://github.com/msgpack/msgpack-d#use-own-deserialization-routine-for-class-and-struct

On 2014-02-16 21:45, "Nordlöw" wrote:
> I believe it's worth the try to get Python-style serialization
> simplicity to D by default. We use Python cpickle at work and its
> fantastically simple and fast. It handles just about everthing you need
> by default as long as you have imported the modules that contain the
> types involved in the serialization.
>
> It would attract more people from dynamical languages like Python if D
> got default-powers in std.serialization that does everything that
> msgpack does plus followings references.
>
> This would make it possible to serialize arbitrary graph-likes
> structures with potential reference cycles. Resolving these cycles can
> be solved by using a map that contain a list of references/classs that
> already have serialized.

I already have all this in Orange [1], which I'm in progress of adapting to a package for Phobos.

> I'm however not sure how serialization of base and superclasses should
> be implemented in the most convenient way. Maybe somehow has a good
> suggestion for this.

This requires registering the subclass in some way. In my implementation one needs to call:

Serializer.register!(Sub);

For full example see [2].

[1] https://github.com/jacob-carlborg/orange
[2] https://github.com/jacob-carlborg/orange/wiki/Base-Class

-- 
/Jacob Carlborg

> I already have all this in Orange [1], which I'm in progress of adapting to a package for Phobos.
>
...
>
> This requires registering the subclass in some way. In my implementation one needs to call:
>
> Serializer.register!(Sub);
>
> For full example see [2].
>
> [1] https://github.com/jacob-carlborg/orange
> [2] https://github.com/jacob-carlborg/orange/wiki/Base-Class
>

The problem with the way your doing it though is that it requires that the library doing the deserialization is fully aware of the semantics used in the serialization implementation, rather than just the syntax and semantics of the selected serialization format.

> Why not? Think of languages like C and C++, they only support pointers. Pointers to basic types are not so interesting but pointers to structs are.

Because, by serializing a pointer, you are implying that mechanism that will be deserializing the value both exists on the same machine, and lies within the same address space, otherwise it will be referencing incorrect data.


> If the same reference value is encountered multiple times, is it serialized once or multiple times?

It is currently serialized multiple times. Serializing once would require a mechanism to exist on both the serializer and deserializer that understands and can interpret those references. As there is not a standard mechanism in JSON to support this, I haven't actually gotten around to implementing that.

> What? I'm referring to methods being called before and after serialization of a given value.

Woops, that means I simply mis-understood your question. The answer to your actual question is somewhat, there is a single code path for dynamic types, I've only implemented support in a modified version of Destructionator's JSVar library, but it should be possible to add support to Variant without any real issue, that does support javascript's 2 parameter json serialization, there are not however callbacks for the start and end of serialization.


> I prefer opt-out rather than opt-in.
Well, if I ever get around to cleaning up the codebase enough to submit it for inclusion in Phobos, there can be a nice long discussion about the pro's and con's of each, because it's a very simple check to remove.

> Can it serialize through base class references?
It currently retrieves all fields present in the class heirarchy, so if you have classes A, B, and C defined as follows:
class A
{
    @optional
    int aA = 10;
}
class B : A
{
    @serializeAs("Bob")
    int bob;
}
class C : B
{
    int cA;
    @nonSerialized
    int cB;
}


Then calling toJSON on an instance of class C, and have modified the value of aA, it will produce a result containing fields defined as aA, Bob, and cA.

Because cB is marked as @nonSerialized, it is ignored during serialization.

Next, bob is serialized as Bob because it is marked as @serializeAs which is intended to account for the difference in naming conventions between different languages, in this case the source of the serialized data may very well be C#, where the convention is typically PascalCase. We however are programming in D, where, if you're me at least, you use camelCase for fields.

Lastly, if we hadn't modified the value of aA, and it was still 10, it would not be included in serialization results, because it is marked as @optional, and contains the same value it would were it default constructed.

On Sunday, 16 February 2014 at 22:41:59 UTC, Orvid King wrote:
>> Why not? Think of languages like C and C++, they only support pointers. Pointers to basic types are not so interesting but pointers to structs are.
>
> Because, by serializing a pointer, you are implying that mechanism that will be deserializing the value both exists on the same machine, and lies within the same address space, otherwise it will be referencing incorrect data.

No, it should just serialize the pointed value and make the same difference upon deserialization - if it is a value, write to it, otherwise allocate new instance on heap and assign its address.

On 2/16/14, Dicebot <public@dicebot.lv> wrote:
> No, it should just serialize the pointed value and make the same difference upon deserialization - if it is a value, write to it, otherwise allocate new instance on heap and assign its address.

Speaking of related things like pointers and cyclic references I have support for this in my fork of msgpack, but the problem is that msgpack is a defined format, so you can't arbitrarily implement your own features without breaking the format:

https://github.com/msgpack/msgpack-d/issues/7

It's 11:48 PM here in case I'm completely off topic. :P

On Sunday, 16 February 2014 at 22:48:51 UTC, Andrej Mitrovic wrote:
> On 2/16/14, Dicebot <public@dicebot.lv> wrote:
>> No, it should just serialize the pointed value and make the same
>> difference upon deserialization - if it is a value, write to it,
>> otherwise allocate new instance on heap and assign its address.
>
> Speaking of related things like pointers and cyclic references I have
> support for this in my fork of msgpack, but the problem is that
> msgpack is a defined format, so you can't arbitrarily implement your
> own features without breaking the format:
>
> https://github.com/msgpack/msgpack-d/issues/7
>
> It's 11:48 PM here in case I'm completely off topic. :P

I just commented.
Maybe, we can support cyclic references by using ext type.

https://github.com/msgpack/msgpack-d/issues/7#issuecomment-35230360

I will try it.

A base class reference is: from your example an A which actually contains a C or B.

Honestly I haven't tried this I would have assumed that D still gives you the real type when using reflection but have you tried it?


On Mon, Feb 17, 2014 at 12:18 AM, Orvid King <blah38621@gmail.com> wrote:

> Why not? Think of languages like C and C++, they only support pointers.
>> Pointers to basic types are not so interesting but pointers to structs are.
>>
>
> Because, by serializing a pointer, you are implying that mechanism that will be deserializing the value both exists on the same machine, and lies within the same address space, otherwise it will be referencing incorrect data.
>
>
>
>  If the same reference value is encountered multiple times, is it
>> serialized once or multiple times?
>>
>
> It is currently serialized multiple times. Serializing once would require a mechanism to exist on both the serializer and deserializer that understands and can interpret those references. As there is not a standard mechanism in JSON to support this, I haven't actually gotten around to implementing that.
>
>
>  What? I'm referring to methods being called before and after
>> serialization of a given value.
>>
>
> Woops, that means I simply mis-understood your question. The answer to your actual question is somewhat, there is a single code path for dynamic types, I've only implemented support in a modified version of Destructionator's JSVar library, but it should be possible to add support to Variant without any real issue, that does support javascript's 2 parameter json serialization, there are not however callbacks for the start and end of serialization.
>
>
>
>  I prefer opt-out rather than opt-in.
>>
> Well, if I ever get around to cleaning up the codebase enough to submit it for inclusion in Phobos, there can be a nice long discussion about the pro's and con's of each, because it's a very simple check to remove.
>
>
>  Can it serialize through base class references?
>>
> It currently retrieves all fields present in the class heirarchy, so if
> you have classes A, B, and C defined as follows:
> class A
> {
>     @optional
>     int aA = 10;
> }
> class B : A
> {
>     @serializeAs("Bob")
>     int bob;
> }
> class C : B
> {
>     int cA;
>     @nonSerialized
>     int cB;
> }
>
>
> Then calling toJSON on an instance of class C, and have modified the value of aA, it will produce a result containing fields defined as aA, Bob, and cA.
>
> Because cB is marked as @nonSerialized, it is ignored during serialization.
>
> Next, bob is serialized as Bob because it is marked as @serializeAs which is intended to account for the difference in naming conventions between different languages, in this case the source of the serialized data may very well be C#, where the convention is typically PascalCase. We however are programming in D, where, if you're me at least, you use camelCase for fields.
>
> Lastly, if we hadn't modified the value of aA, and it was still 10, it would not be included in serialization results, because it is marked as @optional, and contains the same value it would were it default constructed.
>

Do you have a JSON driver for Orange yet? Would be interesting to benchmark
Orange against a purpose designed JSON serialization library.
If your design is mostly compile time the experiment would be very
interesting (I think).




On Mon, Feb 17, 2014 at 12:03 AM, Jacob Carlborg <doob@me.com> wrote:

> On 2014-02-16 21:45, "Nordlöw" wrote:
>
>> I believe it's worth the try to get Python-style serialization simplicity to D by default. We use Python cpickle at work and its fantastically simple and fast. It handles just about everthing you need by default as long as you have imported the modules that contain the types involved in the serialization.
>>
>> It would attract more people from dynamical languages like Python if D got default-powers in std.serialization that does everything that msgpack does plus followings references.
>>
>> This would make it possible to serialize arbitrary graph-likes structures with potential reference cycles. Resolving these cycles can be solved by using a map that contain a list of references/classs that already have serialized.
>>
>
> I already have all this in Orange [1], which I'm in progress of adapting to a package for Phobos.
>
>
>  I'm however not sure how serialization of base and superclasses should
>> be implemented in the most convenient way. Maybe somehow has a good suggestion for this.
>>
>
> This requires registering the subclass in some way. In my implementation one needs to call:
>
> Serializer.register!(Sub);
>
> For full example see [2].
>
> [1] https://github.com/jacob-carlborg/orange
> [2] https://github.com/jacob-carlborg/orange/wiki/Base-Class
>
> --
> /Jacob Carlborg
>