Is there any interest in having a serializer in Phobos? I have a serializer compatible with D2 which I licensed under the Boost license. This is the description from the project page:

Orange is a serialization library for D1 and D2, supporting both Tango and Phobos. It can serialize most of the available types in D, including third party types and can serialize through base class references. It supports fully automatic serialization of all supported types and also supports several ways to customize the serialization. Orange has a separate front end (the serializer) and back end (the archive) making it possible for the user to create new archive types that can be used with the existing serializer.

It's not very well tested but if there's some interest I'm hoping on getting more people to test the library. The project page is: http://dsource.org/projects/orange/

/Jacob Carlborg

Absolutely.  I'm not sure Orange is ready for prime time yet (not saying it isn't, just that I haven't tested/reviewed it yet) but IMHO serialization definitely belongs in the standard lib.  It's a relatively small (in the sense that a whole app would never be built around it; I'm not implying it's trivial), widely useful piece of functionality.  For such pieces of functionality having to go out and find and install a 3rd party library for even simple one-off use cases is a major hassle.  If you're writing a library, as opposed to an application, that needs serialization, using a third party serialization library is basically asking for dependency hell. For these reasons pretty much any language that claims to be mature, modern and high level has serialization in the standard lib.  I know Java, PHP and Python do.

One specific use case of where I'd love a serialization lib in Phobos but would never use one if it were a 3rd party lib is my Plot2kill lib.  I'd love to use serialization for completely lossless saving of plots (i.e. you save a plot, you open it and can do everything you could before you saved it), but not enough to introduce a dependency to the project.

On Sat, Aug 7, 2010 at 11:19 AM, Jacob <doob at me.com> wrote:

> Is there any interest in having a serializer in Phobos? I have a serializer compatible with D2 which I licensed under the Boost license. This is the description from the project page:
>
> Orange is a serialization library for D1 and D2, supporting both Tango and Phobos. It can serialize most of the available types in D, including third party types and can serialize through base class references. It supports fully automatic serialization of all supported types and also supports several ways to customize the serialization. Orange has a separate front end (the serializer) and back end (the archive) making it possible for the user to create new archive types that can be used with the existing serializer.
>
> It's not very well tested but if there's some interest I'm hoping on getting more people to test the library. The project page is: http://dsource.org/projects/orange/
>
> /Jacob Carlborg
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20100807/b9ef5eae/attachment.html>

I think that would be great. Knowing nothing about Orange, I visited the website and read the feature lists and the tutorial (the reference seems to be missing for now). The latter contains:

auto a2 = serializer.deserialize!(A)(data);

which seems to require compile-time knowledge of the deserialized type. I'd expect the library to support something like

Object a2 = serializer.deserialize!Object(data);

and fill the object with an A. I'm pretty certain you've done that, it would be great to feature that within the tutorials and documentation. I'd also expect Variant to play a role there, e.g. you deserialize something and you get a Variant.


Andrei

On 08/07/2010 10:19 AM, Jacob wrote:
> Is there any interest in having a serializer in Phobos? I have a serializer compatible with D2 which I licensed under the Boost license. This is the description from the project page:
>
> Orange is a serialization library for D1 and D2, supporting both Tango and Phobos. It can serialize most of the available types in D, including third party types and can serialize through base class references. It supports fully automatic serialization of all supported types and also supports several ways to customize the serialization. Orange has a separate front end (the serializer) and back end (the archive) making it possible for the user to create new archive types that can be used with the existing serializer.
>
> It's not very well tested but if there's some interest I'm hoping on getting more people to test the library. The project page is: http://dsource.org/projects/orange/
>
> /Jacob Carlborg
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

I think D should support language-specific serialization.
In particular, std.concurrency needs serialization function for object
passing.
Orange seems to be one of the candidates.


Masahiro

On Sun, 08 Aug 2010 00:19:17 +0900, Jacob <doob at me.com> wrote:

> Is there any interest in having a serializer in Phobos? I have a serializer compatible with D2 which I licensed under the Boost license. This is the description from the project page:
>
> Orange is a serialization library for D1 and D2, supporting both Tango and Phobos. It can serialize most of the available types in D, including third party types and can serialize through base class references. It supports fully automatic serialization of all supported types and also supports several ways to customize the serialization. Orange has a separate front end (the serializer) and back end (the archive) making it possible for the user to create new archive types that can be used with the existing serializer.
>
> It's not very well tested but if there's some interest I'm hoping on getting more people to test the library. The project page is: http://dsource.org/projects/orange/
>
> /Jacob Carlborg
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

On 08/08/2010 01:42 AM, Masahiro Nakagawa wrote:
> I think D should support language-specific serialization.
> In particular, std.concurrency needs serialization function for object
> passing.
> Orange seems to be one of the candidates.

Good point! That way we can use messages without having to change the type system (e.g. make shared more sophisticated).

Andrei

I'm glad to see that there is some interest in this. Yes, most languages have support for serialization in the standard library, Ruby, .NET, Objective-C to mention a few more.

On 8 aug 2010, at 01:44, David Simcha wrote:

> Absolutely.  I'm not sure Orange is ready for prime time yet (not saying it isn't, just that I haven't tested/reviewed it yet) but IMHO serialization definitely belongs in the standard lib.  It's a relatively small (in the sense that a whole app would never be built around it; I'm not implying it's trivial), widely useful piece of functionality.  For such pieces of functionality having to go out and find and install a 3rd party library for even simple one-off use cases is a major hassle.  If you're writing a library, as opposed to an application, that needs serialization, using a third party serialization library is basically asking for dependency hell.  For these reasons pretty much any language that claims to be mature, modern and high level has serialization in the standard lib.  I know Java, PHP and Python do.
> 
> One specific use case of where I'd love a serialization lib in Phobos but would never use one if it were a 3rd party lib is my Plot2kill lib.  I'd love to use serialization for completely lossless saving of plots (i.e. you save a plot, you open it and can do everything you could before you saved it), but not enough to introduce a dependency to the project.
> 
> On Sat, Aug 7, 2010 at 11:19 AM, Jacob <doob at me.com> wrote:
> Is there any interest in having a serializer in Phobos? I have a serializer compatible with D2 which I licensed under the Boost license. This is the description from the project page:
> 
> Orange is a serialization library for D1 and D2, supporting both Tango and Phobos. It can serialize most of the available types in D, including third party types and can serialize through base class references. It supports fully automatic serialization of all supported types and also supports several ways to customize the serialization. Orange has a separate front end (the serializer) and back end (the archive) making it possible for the user to create new archive types that can be used with the existing serializer.
> 
> It's not very well tested but if there's some interest I'm hoping on getting more people to test the library. The project page is: http://dsource.org/projects/orange/
> 
> /Jacob Carlborg
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20100808/84408041/attachment-0001.html>

On 8 aug 2010, at 07:47, Andrei Alexandrescu wrote:

> I think that would be great. Knowing nothing about Orange, I visited the website and read the feature lists and the tutorial (the reference seems to be missing for now). The latter contains:
> 
> auto a2 = serializer.deserialize!(A)(data);
> 
> which seems to require compile-time knowledge of the deserialized type. I'd expect the library to support something like
> 
> Object a2 = serializer.deserialize!Object(data);

This is currently not possible in the library. I'm not sure if that would be possible, how would you deserialize a struct for example? There is no factory function for structs like there is for classes. Since all the static types of the objects would be Object how would I set the values when deserializing? Or would Variant be useful here? I have not used Variant. Also due to issue 2844 (and one about getMembers returns any empty array, can't find it right now) I don't think this is currently possible.

> and fill the object with an A. I'm pretty certain you've done that, it would be great to feature that within the tutorials and documentation. I'd also expect Variant to play a role there, e.g. you deserialize something and you get a Variant.
> 
> Andrei
> 
> On 08/07/2010 10:19 AM, Jacob wrote:
>> Is there any interest in having a serializer in Phobos? I have a serializer compatible with D2 which I licensed under the Boost license. This is the description from the project page:
>> 
>> Orange is a serialization library for D1 and D2, supporting both Tango and Phobos. It can serialize most of the available types in D, including third party types and can serialize through base class references. It supports fully automatic serialization of all supported types and also supports several ways to customize the serialization. Orange has a separate front end (the serializer) and back end (the archive) making it possible for the user to create new archive types that can be used with the existing serializer.
>> 
>> It's not very well tested but if there's some interest I'm hoping on getting more people to test the library. The project page is: http://dsource.org/projects/orange/
>> 
>> /Jacob Carlborg
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos

Le 2010-08-08 ? 1:47, Andrei Alexandrescu a ?crit :

> I think that would be great. Knowing nothing about Orange, I visited the website and read the feature lists and the tutorial (the reference seems to be missing for now). The latter contains:
> 
> auto a2 = serializer.deserialize!(A)(data);
> 
> which seems to require compile-time knowledge of the deserialized type. I'd expect the library to support something like
> 
> Object a2 = serializer.deserialize!Object(data);
> 
> and fill the object with an A. I'm pretty certain you've done that, it would be great to feature that within the tutorials and documentation. I'd also expect Variant to play a role there, e.g. you deserialize something and you get a Variant.

My own unreleased, unfinished and in-need-of-a-refactoring serialization module does that... but unfortunately dynamically recreating the right type cannot be so straightforward in the current state of runtime reflection.

This post turned out longer that I expected, please stay with me.

Runtime reflection currently gives you access *only* to the default constructor, so this is what my module do internally when unserializing a class:

	ClassInfo c = findClass(classNameFromSerializationStream);
	Object o = c.create();
	(cast(Unserializable)o).unserialize(serialiationStream);

Since we can't access a constructor with a different signature, we can't unserialize directly from the constructor. This is rather a weak point as it forces all objects to have a default constructor. Another options is for the user to manually register his own constructor with the serialization system prior unserializing, but that's much less convenient.

The unserialize member function called above must be explicitly added by the user (either manually or with a mixin) because the fields don't reflect at runtime and the actual class is unknown at compile-time. And the class needs to conform to an interface that contains that unserialize function so we can find it at runtime.

So before adding a serialization library, I would suggest we solve the runtime-reflection problem and find a standard way to attach various attributes to types and members. That could be done as a library, but ideally it'd have some help from the compiler which could put this stuff where it really belongs: ClassInfo. Currently, QtD has its own mixins for that, my D/Objective-C bridge has its own mixins and class registration system, my serialization module has its own, surely Orange has its own, I believe PyD has its own... this is going to be a mess pretty soon if it isn't already.

Once we have a proper standardized runtime-reflection and attribute system, then the serialization module can focus on serialization instead of implementing various hacks to add and get to the information it needs.

-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/

Le 2010-08-08 ? 6:46, Jacob a ?crit :

> On 8 aug 2010, at 07:47, Andrei Alexandrescu wrote:
> 
>> I think that would be great. Knowing nothing about Orange, I visited the website and read the feature lists and the tutorial (the reference seems to be missing for now). The latter contains:
>> 
>> auto a2 = serializer.deserialize!(A)(data);
>> 
>> which seems to require compile-time knowledge of the deserialized type. I'd expect the library to support something like
>> 
>> Object a2 = serializer.deserialize!Object(data);
> 
> This is currently not possible in the library. I'm not sure if that would be possible, how would you deserialize a struct for example? There is no factory function for structs like there is for classes.

But there's no concept of derived struct. For a struct you always know the type at compile-time. The only way to hide a struct would be behind a void* or void[], but trying to serialize/unserialize that type automatically (without the user writing the serialization code itself) is pointless.

Or you could hide it behind a variant, in which case the variant's serialization should remember the type name so it can find a proper deserializer on the other side. How exactly it does that? Either with better runtime-reflection, or with pre-registered handlers on the unserializer's side (not very convenient).


> Since all the static types of the objects would be Object how would I set the values when deserializing? Or would Variant be useful here? I have not used Variant.

At this point I've been unable to serialize/unserialize a variant.

Another interesting point: it's probably necessary to be tolerant of type differences. For instance, if I serialize a size_t on a machine and unserialize it elsewhere, it might not be the same underlying integral type.


-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/

On 8 aug 2010, at 14:26, Michel Fortin wrote:

> Le 2010-08-08 ? 1:47, Andrei Alexandrescu a ?crit :
> 
>> I think that would be great. Knowing nothing about Orange, I visited the website and read the feature lists and the tutorial (the reference seems to be missing for now). The latter contains:
>> 
>> auto a2 = serializer.deserialize!(A)(data);
>> 
>> which seems to require compile-time knowledge of the deserialized type. I'd expect the library to support something like
>> 
>> Object a2 = serializer.deserialize!Object(data);
>> 
>> and fill the object with an A. I'm pretty certain you've done that, it would be great to feature that within the tutorials and documentation. I'd also expect Variant to play a role there, e.g. you deserialize something and you get a Variant.
> 
> My own unreleased, unfinished and in-need-of-a-refactoring serialization module does that... but unfortunately dynamically recreating the right type cannot be so straightforward in the current state of runtime reflection.
> 
> This post turned out longer that I expected, please stay with me.
> 
> Runtime reflection currently gives you access *only* to the default constructor, so this is what my module do internally when unserializing a class:
> 
> 	ClassInfo c = findClass(classNameFromSerializationStream);
> 	Object o = c.create();
> 	(cast(Unserializable)o).unserialize(serialiationStream);
> 
> Since we can't access a constructor with a different signature, we can't unserialize directly from the constructor. This is rather a weak point as it forces all objects to have a default constructor. Another options is for the user to manually register his own constructor with the serialization system prior unserializing, but that's much less convenient.

Currently I don't call the constructor, just creating an instance of the class and sets its fields. I don't know how good or bad that actually is. Another option would be to use the __ctor and call one of the constructors (if it has multiple constructors) with the default values for the signature.

> The unserialize member function called above must be explicitly added by the user (either manually or with a mixin) because the fields don't reflect at runtime and the actual class is unknown at compile-time. And the class needs to conform to an interface that contains that unserialize function so we can find it at runtime.

I think that is too much extra work. One of my goals was to be able to serialize third party types.

> So before adding a serialization library, I would suggest we solve the runtime-reflection problem and find a standard way to attach various attributes to types and members. That could be done as a library, but ideally it'd have some help from the compiler which could put this stuff where it really belongs: ClassInfo. Currently, QtD has its own mixins for that, my D/Objective-C bridge has its own mixins and class registration system, my serialization module has its own, surely Orange has its own, I believe PyD has its own... this is going to be a mess pretty soon if it isn't already.
> 
> Once we have a proper standardized runtime-reflection and attribute system, then the serialization module can focus on serialization instead of implementing various hacks to add and get to the information it needs.

That is absolutely the best solution. I tried to do the best I could with the current compiler/runtime.

> -- 
> Michel Fortin
> michel.fortin at michelf.com
> http://michelf.com/
> 
> 
> 
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos