http://code.dlang.org/packages/cerealed

What's new?

* Performance improvements
* New UDAs for networking packets for even less required boilerplate

The first new thing is self-explanatory. The second one is explained briefly in this blog post:

https://www.reddit.com/r/programming/comments/3flnlt/cerealed_a_d_library_for_declarative_binary/

(also on HN but you know how that goes)

The summary is you can now write this:

    struct UdpPacket {
        static struct Header {
            ushort srcPort;
            ushort dstPort;
            ushort length;
            ushort checksum;
        }

        enum headerSize = unalignedSizeof!Header;
        alias header this;

        Header header;
        @LengthInBytes("length - headerSize") ubyte[] data;
    }

Code? Who needs code when the compiler can write it for you?

Atila

On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
> The summary is you can now write this:
>
>     struct UdpPacket {
>         static struct Header {
>             ushort srcPort;
>             ushort dstPort;
>             ushort length;
>             ushort checksum;
>         }
>
>         enum headerSize = unalignedSizeof!Header;
>         alias header this;
>
>         Header header;
>         @LengthInBytes("length - headerSize") ubyte[] data;
>     }
>

This deserialization will be identical to casting like this, right? (Not trying to diminish your work, just making sure I get semantics :))

align(1)
struct UdpPacket
{
    align(1)
    static struct Header {
        ushort srcPort;
        ushort dstPort;
        ushort length;
        ushort checksum;
    }

    Header   header;
    ubyte[0] data;
}

// ...

auto packet = cast(UdpPacket*) raw_data.ptr;

On Monday, 3 August 2015 at 09:27:03 UTC, Dicebot wrote:
> On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
>> The summary is you can now write this:
>>
>>     struct UdpPacket {
>>         static struct Header {
>>             ushort srcPort;
>>             ushort dstPort;
>>             ushort length;
>>             ushort checksum;
>>         }
>>
>>         enum headerSize = unalignedSizeof!Header;
>>         alias header this;
>>
>>         Header header;
>>         @LengthInBytes("length - headerSize") ubyte[] data;
>>     }
>>
>
> This deserialization will be identical to casting like this, right? (Not trying to diminish your work, just making sure I get semantics :))
>
> align(1)
> struct UdpPacket
> {
>     align(1)
>     static struct Header {
>         ushort srcPort;
>         ushort dstPort;
>         ushort length;
>         ushort checksum;
>     }
>
>     Header   header;
>     ubyte[0] data;
> }
>
> // ...
>
> auto packet = cast(UdpPacket*) raw_data.ptr;

In this case, yes. In the real-life case I was simplifying, it wasn't a ubyte[] array, it was an array of structs with non-trivial serialisation that also depended on a previous deserialised variable. It was more like this:

struct Outer {
   static struct Header { ... }
   Header header;
   @LengthInBytes("length - headerSize") Inner[] array;
}

struct Inner {
    static struct Header { ... }
    Header header;
    @ArrayLength("length") Unit[] units; //actual length of the array instead of in bytes
}

struct Unit { ... }

So maybe not as useless after all ;)

Atila

On 03-Aug-2015 12:27, Dicebot wrote:
> On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
>> The summary is you can now write this:
>>
>>     struct UdpPacket {
>>         static struct Header {
>>             ushort srcPort;
>>             ushort dstPort;
>>             ushort length;
>>             ushort checksum;
>>         }
>>
>>         enum headerSize = unalignedSizeof!Header;
>>         alias header this;
>>
>>         Header header;
>>         @LengthInBytes("length - headerSize") ubyte[] data;
>>     }
>>
>
> This deserialization will be identical to casting like this, right? (Not
> trying to diminish your work, just making sure I get semantics :))
>
> align(1)
> struct UdpPacket
> {
>      align(1)
>      static struct Header {
>          ushort srcPort;
>          ushort dstPort;
>          ushort length;
>          ushort checksum;
>      }
>
>      Header   header;
>      ubyte[0] data;
> }
>
> // ...
>
> auto packet = cast(UdpPacket*) raw_data.ptr;

Plus/minus network byte order.

-- 
Dmitry Olshansky

On 8/3/2015 2:21 AM, Atila Neves wrote:
>[...]

Please put this as the first comment on the reddit post.

On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
> http://code.dlang.org/packages/cerealed
>
> What's new?
>
> * Performance improvements
> * New UDAs for networking packets for even less required boilerplate
>
> The first new thing is self-explanatory. The second one is explained briefly in this blog post:
>
> https://www.reddit.com/r/programming/comments/3flnlt/cerealed_a_d_library_for_declarative_binary/
>
> (also on HN but you know how that goes)
>
> The summary is you can now write this:
>
>     struct UdpPacket {
>         static struct Header {
>             ushort srcPort;
>             ushort dstPort;
>             ushort length;
>             ushort checksum;
>         }
>
>         enum headerSize = unalignedSizeof!Header;
>         alias header this;
>
>         Header header;
>         @LengthInBytes("length - headerSize") ubyte[] data;
>     }
>
> Code? Who needs code when the compiler can write it for you?
>
> Atila

1. Are there any convenience functions similar to msgpack's pack() and unpack() providing compactness and elegance as

    import msgpack;
    auto x  = [1,2];
    assert(x.pack.unpack!typeof(x) == x);

2. How does the Cereal performance compare to Msgpack after the recent optimizations?

On Monday, 3 August 2015 at 10:37:05 UTC, Dmitry Olshansky wrote:
> On 03-Aug-2015 12:27, Dicebot wrote:
>> On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
>>>[...]
>>
>> This deserialization will be identical to casting like this, right? (Not
>> trying to diminish your work, just making sure I get semantics :))
>>
>> align(1)
>> struct UdpPacket
>> {
>>      align(1)
>>      static struct Header {
>>          ushort srcPort;
>>          ushort dstPort;
>>          ushort length;
>>          ushort checksum;
>>      }
>>
>>      Header   header;
>>      ubyte[0] data;
>> }
>>
>> // ...
>>
>> auto packet = cast(UdpPacket*) raw_data.ptr;
>
> Plus/minus network byte order.

Oh yeah, and that. Cerealed always does network order.

Atila

On Monday, 3 August 2015 at 11:13:50 UTC, Walter Bright wrote:
> On 8/3/2015 2:21 AM, Atila Neves wrote:
>>[...]
>
> Please put this as the first comment on the reddit post.

Which one?

Atila

On Monday, 3 August 2015 at 11:43:15 UTC, Per Nordlöw wrote:
> On Monday, 3 August 2015 at 09:21:50 UTC, Atila Neves wrote:
>> http://code.dlang.org/packages/cerealed
>>
>> What's new?
>>
>> * Performance improvements
>> * New UDAs for networking packets for even less required boilerplate
>>
>> The first new thing is self-explanatory. The second one is explained briefly in this blog post:
>>
>> https://www.reddit.com/r/programming/comments/3flnlt/cerealed_a_d_library_for_declarative_binary/
>>
>> (also on HN but you know how that goes)
>>
>> The summary is you can now write this:
>>
>>     struct UdpPacket {
>>         static struct Header {
>>             ushort srcPort;
>>             ushort dstPort;
>>             ushort length;
>>             ushort checksum;
>>         }
>>
>>         enum headerSize = unalignedSizeof!Header;
>>         alias header this;
>>
>>         Header header;
>>         @LengthInBytes("length - headerSize") ubyte[] data;
>>     }
>>
>> Code? Who needs code when the compiler can write it for you?
>>
>> Atila
>
> 1. Are there any convenience functions similar to msgpack's pack() and unpack() providing compactness and elegance as
>
>     import msgpack;
>     auto x  = [1,2];
>     assert(x.pack.unpack!typeof(x) == x);


Yes. `cerealise` and `decerealise`. The former is slightly weird for performance reasons. It takes a lambda that tells it what to do with the resulting bytes.

import cerealed;
auto x = MyStruct();
x.cerealise!(bytes => writeln(bytes));

ubyte[] bytes = [...];
auto x = bytes.decerealise!MyStruct;


>
> 2. How does the Cereal performance compare to Msgpack after the recent optimizations?

Close with LDC and DMD, faster with GDC: http://forum.dlang.org/post/nkcelouzpjsgmqtvnonq@forum.dlang.org

I haven't even tried optimising it myself though, I just merged a contribution from someone else.

Atila

On Monday, 3 August 2015 at 11:51:24 UTC, Atila Neves wrote:
> Yes. `cerealise` and `decerealise`. The former is slightly weird for performance reasons. It takes a lambda that tells it what to do with the resulting bytes.
>
> Close with LDC and DMD, faster with GDC: http://forum.dlang.org/post/nkcelouzpjsgmqtvnonq@forum.dlang.org

Nice!

Are there any plans to add different backends (for instance msgpack) to Cereal? Then we could have one package to rule them all!

I'll try Cereal in favor of msgpack next time!

Thanks, Atila!