On Sun, 21 Aug 2011 05:28:33 +0900, Jacob Carlborg <doob@me.com> wrote:

> On 2011-08-20 19:45, dsimcha wrote:
>> On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
>>> Masahiro Nakagawa seems to be working on msgpack:
>>> https://bitbucket.org/repeatedly/msgpack4d/
>>
>> Yeah, this does look quite useful. I tried it a few weeks ago and ran
>> into a couple showstopper bugs. I filed them and he fixed them, but I
>> never got around to trying it again.
>>
>> I actually think that since both are very commonly needed things and
>> satisfy completely different niches, Orange and msgpack4d might both
>> eventually have a rightful place in Phobos. Orange would be recommended
>> when portability and serializing as many types as possible are
>> important, and msgpack4d would be recommended when speed and space
>> efficiency are the primary concerns.
>
> If it's possible to extract a value given a key with messagepack I think it would be possible to use messagepack as an archive for Orange. Since messagepack supports maps every type Orange can serialize should be possible serialize with messagepack, although probably not as efficient as "regular" messagepack.
>

I agree.
Unfortunately, Orange with MessagePack decreases MessagePack merits...

On 2011-08-21 23:16, Masahiro Nakagawa wrote:
> On Sun, 21 Aug 2011 05:28:33 +0900, Jacob Carlborg <doob@me.com> wrote:
> 
>> On 2011-08-20 19:45, dsimcha wrote:
>>> On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:
>>>> Masahiro Nakagawa seems to be working on msgpack: https://bitbucket.org/repeatedly/msgpack4d/
>>>
>>> Yeah, this does look quite useful. I tried it a few weeks ago and ran into a couple showstopper bugs. I filed them and he fixed them, but I never got around to trying it again.
>>>
>>> I actually think that since both are very commonly needed things and satisfy completely different niches, Orange and msgpack4d might both eventually have a rightful place in Phobos. Orange would be recommended when portability and serializing as many types as possible are important, and msgpack4d would be recommended when speed and space efficiency are the primary concerns.
>>
>> If it's possible to extract a value given a key with messagepack I think it would be possible to use messagepack as an archive for Orange. Since messagepack supports maps every type Orange can serialize should be possible serialize with messagepack, although probably not as efficient as "regular" messagepack.
>>
> 
> I agree.
> Unfortunately, Orange with MessagePack decreases MessagePack merits...

I suspected that.

-- 
/Jacob Carlborg

On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob@me.com> wrote:

> I've almost finished the rewrite of my serialization library Orange.

While I've never had the chance to use Orange myself, one of the problems I've often heard being associated with using Orange was large binary file sizes. What are your thoughts about this issue? If this is a real problem, do you think the toolchain could be improved in this regard (perhaps ideas such as merging bitwise-identical functions at link time)?

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On 2011-08-22 21:50, Vladimir Panteleev wrote:
> On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob@me.com> wrote:
>
>> I've almost finished the rewrite of my serialization library Orange.
>
> While I've never had the chance to use Orange myself, one of the
> problems I've often heard being associated with using Orange was large
> binary file sizes. What are your thoughts about this issue? If this is a
> real problem, do you think the toolchain could be improved in this
> regard (perhaps ideas such as merging bitwise-identical functions at
> link time)?

I never heard anything being associated with using Orange. The library takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World app using Orange takes up 1.2 mb, I don't know if that's a problem.

I don't have any thoughts, I didn't know it was an issue. Yes I think the tool chain can be improved, at least I hope it can.

-- 
/Jacob Carlborg

On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob@me.com> said:

> On 2011-08-22 21:50, Vladimir Panteleev wrote:
>> On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob@me.com> wrote:
>> 
>>> I've almost finished the rewrite of my serialization library Orange.
>> 
>> While I've never had the chance to use Orange myself, one of the
>> problems I've often heard being associated with using Orange was large
>> binary file sizes. What are your thoughts about this issue? If this is a
>> real problem, do you think the toolchain could be improved in this
>> regard (perhaps ideas such as merging bitwise-identical functions at
>> link time)?
> 
> I never heard anything being associated with using Orange. The library takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World app using Orange takes up 1.2 mb, I don't know if that's a problem.
> 
> I don't have any thoughts, I didn't know it was an issue. Yes I think the tool chain can be improved, at least I hope it can.

If there's any truth in that (which I don't know), it'd probably have something to do with the amount of generated code through templates.

When I was still developing the D/Objective-C bridge as a bunch of templates and stub objects, the binary size was growing significantly for each new Objective-C class I added (due to the insane number of stub functions instantiated). And at some point it became totally impractical to use, even though it worked.

I doubt this is a problem for Orange because of its more limited scope, but it'd nevertheless be wise to measure how much the binary size grows when you add more types to serialize and unserialize and more code is generated as a result.

-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

On 2011-08-24 02:53, Michel Fortin wrote:
> On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob@me.com> said:
>
>> On 2011-08-22 21:50, Vladimir Panteleev wrote:
>>> On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob@me.com> wrote:
>>>
>>>> I've almost finished the rewrite of my serialization library Orange.
>>>
>>> While I've never had the chance to use Orange myself, one of the
>>> problems I've often heard being associated with using Orange was large
>>> binary file sizes. What are your thoughts about this issue? If this is a
>>> real problem, do you think the toolchain could be improved in this
>>> regard (perhaps ideas such as merging bitwise-identical functions at
>>> link time)?
>>
>> I never heard anything being associated with using Orange. The library
>> takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World
>> app using Orange takes up 1.2 mb, I don't know if that's a problem.
>>
>> I don't have any thoughts, I didn't know it was an issue. Yes I think
>> the tool chain can be improved, at least I hope it can.
>
> If there's any truth in that (which I don't know), it'd probably have
> something to do with the amount of generated code through templates.
>
> When I was still developing the D/Objective-C bridge as a bunch of
> templates and stub objects, the binary size was growing significantly
> for each new Objective-C class I added (due to the insane number of stub
> functions instantiated). And at some point it became totally impractical
> to use, even though it worked.

Yeah, that was insane.

> I doubt this is a problem for Orange because of its more limited scope,
> but it'd nevertheless be wise to measure how much the binary size grows
> when you add more types to serialize and unserialize and more code is
> generated as a result.

I guess I should try serializing a big data structure of some kind.

-- 
/Jacob Carlborg

On Wed, 24 Aug 2011 02:25:19 -0400, Jacob Carlborg <doob@me.com> wrote:

> On 2011-08-24 02:53, Michel Fortin wrote:
>> On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob@me.com> said:
>>
>>> On 2011-08-22 21:50, Vladimir Panteleev wrote:
>>>> On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob@me.com> wrote:
>>>>
>>>>> I've almost finished the rewrite of my serialization library Orange.
>>>>
>>>> While I've never had the chance to use Orange myself, one of the
>>>> problems I've often heard being associated with using Orange was large
>>>> binary file sizes. What are your thoughts about this issue? If this is a
>>>> real problem, do you think the toolchain could be improved in this
>>>> regard (perhaps ideas such as merging bitwise-identical functions at
>>>> link time)?
>>>
>>> I never heard anything being associated with using Orange. The library
>>> takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World
>>> app using Orange takes up 1.2 mb, I don't know if that's a problem.
>>>
>>> I don't have any thoughts, I didn't know it was an issue. Yes I think
>>> the tool chain can be improved, at least I hope it can.
>>
>> If there's any truth in that (which I don't know), it'd probably have
>> something to do with the amount of generated code through templates.
>>
>> When I was still developing the D/Objective-C bridge as a bunch of
>> templates and stub objects, the binary size was growing significantly
>> for each new Objective-C class I added (due to the insane number of stub
>> functions instantiated). And at some point it became totally impractical
>> to use, even though it worked.
>
> Yeah, that was insane.
>
>> I doubt this is a problem for Orange because of its more limited scope,
>> but it'd nevertheless be wise to measure how much the binary size grows
>> when you add more types to serialize and unserialize and more code is
>> generated as a result.
>
> I guess I should try serializing a big data structure of some kind.
>

Actually, it's the number of types rather than the amount of memory which causes issues. You can try something like this:

    template Qual(T) {
        alias TypeTuple!(T,T[],const(T)[],immutable(T)[],shared(T)[], const T, immutable T, shared T, const T[], immutable T[], shared T[]) Qual;
    }
    alias TypeTuple!(bool,ubyte,byte,ushort,short,uint,int,ulong,long,float,double,real,char,wchar,dchar) ValueTypes;

    alias staticMap!(Qual,ValueTypes) QualifiedTypes;

To generate a large tuple of all the basic D types and then see how much code is generated.

On Aug 20, 2011, at 9:16 AM, dsimcha wrote:

> This looks really great!  I'll try it out sometime later.  A few questions:
> 
> 1.  What other archiver formats besides XML might be useful?  (I remember binary can't work because of the way keys work, though messagepack is probably a better option for lightweight serialization in general.  Orange's niche is portability and serializing as many types as possible.)

JSON, Protobuf…

> 2.  What are the prospects for submitting this for inclusion in Phobos?  Serialization is something basic and universally needed enough that it should not require a third-party library.

Some form of serialization definitely needs to be in Phobos/Druntime.  The other day I was thinking that the basic framework for struct/class serialization might be nice in Druntime as a replacement for x.toString, with most of the actual functional bits in Phobos.

On 2011-08-29 19:19, Sean Kelly wrote:
> On Aug 20, 2011, at 9:16 AM, dsimcha wrote:
>
>> This looks really great!  I'll try it out sometime later.  A few questions:
>>
>> 1.  What other archiver formats besides XML might be useful?  (I remember binary can't work because of the way keys work, though messagepack is probably a better option for lightweight serialization in general.  Orange's niche is portability and serializing as many types as possible.)
>
> JSON, Protobuf…

YAML as well.

>> 2.  What are the prospects for submitting this for inclusion in Phobos?  Serialization is something basic and universally needed enough that it should not require a third-party library.
>
> Some form of serialization definitely needs to be in Phobos/Druntime.  The other day I was thinking that the basic framework for struct/class serialization might be nice in Druntime as a replacement for x.toString, with most of the actual functional bits in Phobos.

I've been think the same several times before and think it would be really useful for debugging.

-- 
/Jacob Carlborg