I've almost finished the rewrite of my serialization library Orange. I'm hoping that someone wants to give it a try and see what issues/bugs are found.

Project page: http://dsource.org/projects/orange
Source code: https://github.com/jacob-carlborg/orange

There are two usage examples on the project page. For more examples I recommend looking at the unit tests in the "tests" directory.

Description:

Orange is a serialization library for D1 and D2, supporting both Tango and Phobos. It can serialize most of the available types in D, including third party types and can serialize through base class references. It supports fully automatic serialization of all supported types and also supports several ways to customize the serialization process. Orange has a separate front end (the serializer) and back end (the archive) making it possible for the user to create new archive types that can be used with the existing serializer.

Features:

* Automatically serializes the base classes
* Supports events (before and after (de)serializing)

* Supports non-serialized fields and classes (you can say that some fields in a class should not be serialized)

* Licensed under the Boost license
* Std/runtime library independent

* Extendable - possible to create new archive types and use them with the existing serializer

* Serializes through base class references
* Serializes third party types

* Customization of the (de)serialization process, both intrusive and non-intrusive

* Properly (de)serializes slices and pointers

Known Issues/Limitations:

* Due to limitations in the XML module provided by Phobos the XMLArchive will only work with "char" as the template type with D2

* Due to several bugs/limitations in the compiler/runtime even the D2 version requires you to register the type when serializing through base class references

* No built-in support for versioning
* Floating point numbers are not serialized as hexadecimal (D1)

-- 
/Jacob Carlborg

This looks really great!  I'll try it out sometime later.  A few questions:

1.  What other archiver formats besides XML might be useful?  (I remember binary can't work because of the way keys work, though messagepack is probably a better option for lightweight serialization in general.  Orange's niche is portability and serializing as many types as possible.)

2.  What are the prospects for submitting this for inclusion in Phobos?  Serialization is something basic and universally needed enough that it should not require a third-party library.

3.  Given that all of the necessary introspection already works, what are the prospects for creating a Clone archiver?  A Clone archiver would not actually serialize anything and instead would deep clone whatever data structure it receives.  If Orange gets into Phobos, cloning could be recognized by std.concurrency as a safe, simple way to pass complex object graphs between threads.  I'm picturing an API something like this:

import std.stdio, whateverOrangeEndsUpBeingNamed.

struct Cloned(T) {
    T obj;
}

Cloned!T cloned(T)(T obj) { return typeof(return)(obj); }

void spawmMe() {
    receive((int[][]) { writeln("Received an int[][]."); });
}

void main() {
    auto tid = spawn(&spawnMe);
    auto mat = new int[][](42, 42);

    // FAILS:  Implicit sharing.
    tid.send(mat);

    // Works.  mat is automatically deep cloned and the clone is sent
    // to tid.  send recognized the Cloned struct as special and
    // processes the payload accordingly.
    tid.send(cloned(mat));
}



On 8/20/2011 11:13 AM, Jacob Carlborg wrote:
> I've almost finished the rewrite of my serialization library Orange. I'm
> hoping that someone wants to give it a try and see what issues/bugs are
> found.
>
> Project page: http://dsource.org/projects/orange
> Source code: https://github.com/jacob-carlborg/orange
>
> There are two usage examples on the project page. For more examples I
> recommend looking at the unit tests in the "tests" directory.
>
> Description:
>
> Orange is a serialization library for D1 and D2, supporting both Tango
> and Phobos. It can serialize most of the available types in D, including
> third party types and can serialize through base class references. It
> supports fully automatic serialization of all supported types and also
> supports several ways to customize the serialization process. Orange has
> a separate front end (the serializer) and back end (the archive) making
> it possible for the user to create new archive types that can be used
> with the existing serializer.
>
> Features:
>
> * Automatically serializes the base classes
> * Supports events (before and after (de)serializing)
>
> * Supports non-serialized fields and classes (you can say that some
> fields in a class should not be serialized)
>
> * Licensed under the Boost license
> * Std/runtime library independent
>
> * Extendable - possible to create new archive types and use them with
> the existing serializer
>
> * Serializes through base class references
> * Serializes third party types
>
> * Customization of the (de)serialization process, both intrusive and
> non-intrusive
>
> * Properly (de)serializes slices and pointers
>
> Known Issues/Limitations:
>
> * Due to limitations in the XML module provided by Phobos the XMLArchive
> will only work with "char" as the template type with D2
>
> * Due to several bugs/limitations in the compiler/runtime even the D2
> version requires you to register the type when serializing through base
> class references
>
> * No built-in support for versioning
> * Floating point numbers are not serialized as hexadecimal (D1)
>

Masahiro Nakagawa seems to be working on msgpack: https://bitbucket.org/repeatedly/msgpack4d/

Is RIFF considered a good format? I've ran into it when porting C code,  it seems it's also used by Google and probably other companies. http://en.wikipedia.org/wiki/Resource_Interchange_File_Format

On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
> Masahiro Nakagawa seems to be working on msgpack:
> https://bitbucket.org/repeatedly/msgpack4d/

Yeah, this does look quite useful.  I tried it a few weeks ago and ran into a couple showstopper bugs.  I filed them and he fixed them, but I never got around to trying it again.

I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos.  Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.

>
> Is RIFF considered a good format? I've ran into it when porting C
> code,  it seems it's also used by Google and probably other companies.
> http://en.wikipedia.org/wiki/Resource_Interchange_File_Format

From reading the Wikipedia article (I knew nothing about RIFF before this discussion) it sounds like it's in a valley in between two utility peaks.  XML provides maximum portability.  msgpack provides maximum efficiency.  RIFF doesn't really provide either all that well.

On 2011-08-20 18:16, dsimcha wrote:
> This looks really great! I'll try it out sometime later. A few questions:
>
> 1. What other archiver formats besides XML might be useful? (I remember
> binary can't work because of the way keys work, though messagepack is
> probably a better option for lightweight serialization in general.
> Orange's niche is portability and serializing as many types as possible.)

I assume any archive type that can unpack a value given a key will work. I assume JSON and YAML will work as well. I no nothing about binary serialization but I think messagepack can work.

Another problem might be pointers and slices. For example, at the first step every array is assumed to be pointing to its own memory, i.e. not a slice. After everything is serialized the serializer post process the serialized data and replaces all arrays that points into another array with a slice.

> 2. What are the prospects for submitting this for inclusion in Phobos?
> Serialization is something basic and universally needed enough that it
> should not require a third-party library.

From my part:

* Remove all Tango/D1 specific code
* Add documentation
* There are some few things left in the rewrite process
* I've made some modifications to the XML module in Phobos, these need to be applied as well
* Probably something else I can't think of right now

I'm not particular happy about removing the Tango/D1 specific code if I'm not sure that the library can be included in Phobos.

> 3. Given that all of the necessary introspection already works, what are
> the prospects for creating a Clone archiver? A Clone archiver would not
> actually serialize anything and instead would deep clone whatever data
> structure it receives. If Orange gets into Phobos, cloning could be
> recognized by std.concurrency as a safe, simple way to pass complex
> object graphs between threads. I'm picturing an API something like this:
>
> import std.stdio, whateverOrangeEndsUpBeingNamed.
>
> struct Cloned(T) {
> T obj;
> }
>
> Cloned!T cloned(T)(T obj) { return typeof(return)(obj); }
>
> void spawmMe() {
> receive((int[][]) { writeln("Received an int[][]."); });
> }
>
> void main() {
> auto tid = spawn(&spawnMe);
> auto mat = new int[][](42, 42);
>
> // FAILS: Implicit sharing.
> tid.send(mat);
>
> // Works. mat is automatically deep cloned and the clone is sent
> // to tid. send recognized the Cloned struct as special and
> // processes the payload accordingly.
> tid.send(cloned(mat));
> }

Currently you could serialize the data an pass it send it over to another thread and deserialize on the other side, but I'm guessing this would be not very effective. I'm not exactly sure what's needed to be done to make a clone archive and what can be shared but as you say, the necessary introspection already work.

> On 8/20/2011 11:13 AM, Jacob Carlborg wrote:
>> I've almost finished the rewrite of my serialization library Orange. I'm
>> hoping that someone wants to give it a try and see what issues/bugs are
>> found.
>>
>> Project page: http://dsource.org/projects/orange
>> Source code: https://github.com/jacob-carlborg/orange
>>
>> There are two usage examples on the project page. For more examples I
>> recommend looking at the unit tests in the "tests" directory.
>>
>> Description:
>>
>> Orange is a serialization library for D1 and D2, supporting both Tango
>> and Phobos. It can serialize most of the available types in D, including
>> third party types and can serialize through base class references. It
>> supports fully automatic serialization of all supported types and also
>> supports several ways to customize the serialization process. Orange has
>> a separate front end (the serializer) and back end (the archive) making
>> it possible for the user to create new archive types that can be used
>> with the existing serializer.
>>
>> Features:
>>
>> * Automatically serializes the base classes
>> * Supports events (before and after (de)serializing)
>>
>> * Supports non-serialized fields and classes (you can say that some
>> fields in a class should not be serialized)
>>
>> * Licensed under the Boost license
>> * Std/runtime library independent
>>
>> * Extendable - possible to create new archive types and use them with
>> the existing serializer
>>
>> * Serializes through base class references
>> * Serializes third party types
>>
>> * Customization of the (de)serialization process, both intrusive and
>> non-intrusive
>>
>> * Properly (de)serializes slices and pointers
>>
>> Known Issues/Limitations:
>>
>> * Due to limitations in the XML module provided by Phobos the XMLArchive
>> will only work with "char" as the template type with D2
>>
>> * Due to several bugs/limitations in the compiler/runtime even the D2
>> version requires you to register the type when serializing through base
>> class references
>>
>> * No built-in support for versioning
>> * Floating point numbers are not serialized as hexadecimal (D1)
>>
>


-- 
/Jacob Carlborg

On 2011-08-20 19:45, dsimcha wrote:
> On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
>> Masahiro Nakagawa seems to be working on msgpack:
>> https://bitbucket.org/repeatedly/msgpack4d/
>
> Yeah, this does look quite useful. I tried it a few weeks ago and ran
> into a couple showstopper bugs. I filed them and he fixed them, but I
> never got around to trying it again.
>
> I actually think that since both are very commonly needed things and
> satisfy completely different niches, Orange and msgpack4d might both
> eventually have a rightful place in Phobos. Orange would be recommended
> when portability and serializing as many types as possible are
> important, and msgpack4d would be recommended when speed and space
> efficiency are the primary concerns.

If it's possible to extract a value given a key with messagepack I think it would be possible to use messagepack as an archive for Orange. Since messagepack supports maps every type Orange can serialize should be possible serialize with messagepack, although probably not as efficient as "regular" messagepack.

-- 
/Jacob Carlborg

On 8/20/2011 4:24 PM, Jacob Carlborg wrote:
>> 2. What are the prospects for submitting this for inclusion in Phobos?
>> Serialization is something basic and universally needed enough that it
>> should not require a third-party library.
>
>  From my part:
>
> * Remove all Tango/D1 specific code
> * Add documentation
> * There are some few things left in the rewrite process

Ok, it sounds like this isn't going to be ready that soon.  I'll start using it as-is, as a third party library.  Then I can make suggestions, file bug reports, etc. and streamline its eventual Phobos review.

> * I've made some modifications to the XML module in Phobos, these need
> to be applied as well

If we're really lucky, by the time your turn in the review queue comes along, Phobos will have a new and much better XML module, since Thomas Sowinki is apparently working on one.

> * Probably something else I can't think of right now
>
> I'm not particular happy about removing the Tango/D1 specific code if
> I'm not sure that the library can be included in Phobos.

Hmm. We could review the module before you remove it, with the assumption that this code will be removed if accepted.


> Currently you could serialize the data an pass it send it over to
> another thread and deserialize on the other side, but I'm guessing this
> would be not very effective. I'm not exactly sure what's needed to be
> done to make a clone archive and what can be shared but as you say, the
> necessary introspection already work.

Right.  The idea would be to bypass serializing and unserializing and leverage the introspection that Orange provides to create an efficient clone() function.  This is mostly a long-term prospect, to be considered more seriously if/when Orange gets into Phobos.  All we'd need from orange is an archiver that efficiently clones without actually serializing or deserializing anything.  Given this, the changes to std.concurrency would be trivial.

On 2011-08-20 23:29, dsimcha wrote:
> On 8/20/2011 4:24 PM, Jacob Carlborg wrote:
>>> 2. What are the prospects for submitting this for inclusion in Phobos?
>>> Serialization is something basic and universally needed enough that it
>>> should not require a third-party library.
>>
>> From my part:
>>
>> * Remove all Tango/D1 specific code
>> * Add documentation
>> * There are some few things left in the rewrite process
>
> Ok, it sounds like this isn't going to be ready that soon. I'll start
> using it as-is, as a third party library. Then I can make suggestions,
> file bug reports, etc. and streamline its eventual Phobos review.

I will add documentation and finish the rewrite regardless so it's only the D1/Tango specific code.

Oh, one more thing. I'm using my own kind of unit test "framework", don't know what people think about that. It could easily be removed though.

>> * I've made some modifications to the XML module in Phobos, these need
>> to be applied as well
>
> If we're really lucky, by the time your turn in the review queue comes
> along, Phobos will have a new and much better XML module, since Thomas
> Sowinki is apparently working on one.

Yes, hopefully. Also, some of the changes I've made are because of differences with the Tango and Phobos XML modules. Since it will only support the Phobos module it will be easier and hopefully not require as many changes.

>> * Probably something else I can't think of right now
>>
>> I'm not particular happy about removing the Tango/D1 specific code if
>> I'm not sure that the library can be included in Phobos.
>
> Hmm. We could review the module before you remove it, with the
> assumption that this code will be removed if accepted.

Exactly, sounds like a good idea.

>> Currently you could serialize the data an pass it send it over to
>> another thread and deserialize on the other side, but I'm guessing this
>> would be not very effective. I'm not exactly sure what's needed to be
>> done to make a clone archive and what can be shared but as you say, the
>> necessary introspection already work.
>
> Right. The idea would be to bypass serializing and unserializing and
> leverage the introspection that Orange provides to create an efficient
> clone() function. This is mostly a long-term prospect, to be considered
> more seriously if/when Orange gets into Phobos. All we'd need from
> orange is an archiver that efficiently clones without actually
> serializing or deserializing anything. Given this, the changes to
> std.concurrency would be trivial.

Yeah, I wonder if any data needs to be stored in the archive and in that case, how it could be stored.

-- 
/Jacob Carlborg

On 20.08.2011 19:45, dsimcha wrote:
> On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
>> Masahiro Nakagawa seems to be working on msgpack:
>> https://bitbucket.org/repeatedly/msgpack4d/
>
> Yeah, this does look quite useful. I tried it a few weeks ago and ran
> into a couple showstopper bugs. I filed them and he fixed them, but I
> never got around to trying it again.
>
> I actually think that since both are very commonly needed things and
> satisfy completely different niches, Orange and msgpack4d might both
> eventually have a rightful place in Phobos. Orange would be recommended
> when portability and serializing as many types as possible are
> important, and msgpack4d would be recommended when speed and space
> efficiency are the primary concerns.
>
>>
>> Is RIFF considered a good format? I've ran into it when porting C
>> code, it seems it's also used by Google and probably other companies.
>> http://en.wikipedia.org/wiki/Resource_Interchange_File_Format
>
>  From reading the Wikipedia article (I knew nothing about RIFF before
> this discussion) it sounds like it's in a valley in between two utility
> peaks. XML provides maximum portability. msgpack provides maximum
> efficiency. RIFF doesn't really provide either all that well.

Agreed. I've worked a lot with RIFF in the past, and it only specifies a high level structure by having sections with a known size. It doesn't give much over a plain binary file.

On Sun, 21 Aug 2011 02:45:20 +0900, dsimcha <dsimcha@yahoo.com> wrote:

> On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
>> Masahiro Nakagawa seems to be working on msgpack:
>> https://bitbucket.org/repeatedly/msgpack4d/
>
> Yeah, this does look quite useful.  I tried it a few weeks ago and ran into a couple showstopper bugs.  I filed them and he fixed them, but I never got around to trying it again.
>
> I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos.

I agree. In the past, I mentioned this point ;)

> Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.

Yes for MessagePack.
In addition, communicating other language is important like JSON.

P.S.
My D libraries moved to github. msgpack4d is here.

https://github.com/msgpack/msgpack-d

FTTB, bitbucket is available.