Jump to page: 1 2
Thread overview
Implementing serialisation with minmal boilerplate and template overhead using core.reflect
Aug 15, 2021
Stefan Koch
Aug 15, 2021
Stefan Koch
Aug 15, 2021
drug
Aug 15, 2021
Stefan Koch
Aug 16, 2021
Temtaime
Aug 16, 2021
Bruce Carneal
Aug 17, 2021
russhy
Aug 17, 2021
12345swordy
Aug 17, 2021
russhy
Aug 17, 2021
Adam Ruppe
Aug 17, 2021
12345swordy
Aug 17, 2021
Alexandru Ermicioi
Aug 17, 2021
12345swordy
Aug 17, 2021
Alexandru Ermicioi
Aug 18, 2021
Arafel
Aug 17, 2021
max haughton
August 15, 2021

Good Day everyone,

A friend asked me recently of core.reflect could be used for serialization.

The code to do that generically is about 100 lines of code.
Since it's so little I am just going to post it here verbatim.

class C { @("NoSerialize") ubyte a; uint b; @("NoSerialize") ulong c;
    this(ubyte a, uint b, ulong c)
    {
        this.a = a; this.b = b; this.c = c;
    }
}
struct S { @("NoSerialize") ubyte a; uint b; ubyte c; @("NoSerialize") ulong d; }

import core.reflect.reflect;
/// if a type is an aggregate type return the aggregate declaration
/// otherwise return null
AggregateDeclaration aggregateFromType(const Type type)
{
    AggregateDeclaration result = null;
    if (auto ts = cast(TypeStruct)type)
    {
        result = ts.sym;
    } else if (auto tc = cast(TypeClass)type)
    {
        result = tc.sym;
    }
    return result;
}

bool hasNoSerializeAttrib(const Declaration d)
{
    bool NoSerialize = false;
    foreach(attr;d.attributes)
    {
        auto se = cast(StringLiteral)attr;
        if (se && se.string_ == "NoSerialize")
        {
          NoSerialize = true;
          break;
        }
    }
    return NoSerialize;
}
  /// the only template in here. only one instances per serialzed-root-type
  const(ubyte[]) serialize(T)(T value) {
    static immutable Type type = cast(immutable Type) nodeFromName("T");
    return serializeType(cast(ubyte*)&value, type);
  }

  const(ubyte[]) serializeType(const ubyte* ptr, const Type type)
  {
    // writeShallowTypeDescriptor(Type);
    // printf("Serializing type: %s\n", type.identifier.ptr);
    if (auto sa = cast (TypeArray) type)
    {
        ulong length = sa.dim;
        const void* values = ptr;
        auto elemType = sa.nextOf;
        return serializeArray(length, ptr, elemType);
    }
    else if (auto da = cast(TypeSlice) type)
    {
        ulong length = *cast(size_t*) ptr;
        const void* values = *cast(const ubyte**)(ptr + size_t.sizeof);
        auto elemType = da.nextOf;
        return serializeArray(length, ptr, elemType);
    }
    else if (auto ts = cast(TypeStruct) type)
    {
        auto fields = ts.sym.fields;
        return serializeAggregate(ptr, fields);
    }
    else if (auto tc = cast(TypeClass) type)
    {
        auto fields = tc.sym.fields;
        return serializeAggregate(*cast(const ubyte**)ptr, fields);
    }
    else if (auto tb = cast(TypeBasic) type)
    {
        // writeTypeTag ?
        return ptr[0 .. type.size];
    }
    else assert(0, "Serialisation for " ~ type.identifier ~ " not implemented .. " ~ (cast()type).toString());
  }

  ubyte[] serializeArray(ulong length, const ubyte* ptr, const Type elemType)
  {
    ubyte[] result;
    // writeLength(length)
    foreach(i; 0 .. length)
    {
      result ~= serializeType(ptr, elemType);
    }
    return result;
  }

  ubyte[] serializeAggregate(const ubyte* ptr, VariableDeclaration[] fields)
  {
    ubyte[] result;
    foreach(f; fields)
    {
      if (hasNoSerializeAttrib(f))
      {
        // skip fields which are annotated with noSerialize;
        continue;
      }
      result ~= serializeType(ptr + f.offset, f.type);
    }
    return result;
  }


void main()
{
    S s = S(72, 19992034, 98);
    C c = new C(72, 19992039, 98);
    auto buffer = serialize(s);
    assert(buffer.length == 5);
    assert ((buffer[0] | buffer[1] << 8 | buffer[2] << 16 | buffer[3] << 24) == 19992034);
    assert (buffer[4] == 98);
    auto buffer_c = serialize(c);
    assert(buffer_c.length == 4);
    assert ((buffer_c[0] | buffer_c[1] << 8 | buffer_c[2] << 16 | buffer_c[3] << 24) == 19992039);
    // we can see the fields annotated with @("NoSerialize") are skipped
}

I would like to know what you think about this

In my next example I will show how you can modify serialization for library types source-code you can't control.

However for that to work core.reflect needs to be extended a little ;)

Cheers and have a nice day,

Stefan

August 15, 2021

On Sunday, 15 August 2021 at 11:18:57 UTC, Stefan Koch wrote:

>

Good Day everyone,

A friend asked me recently of core.reflect could be used for serialization.
[ ... ]

I would like to know what you think about this

There was a bug in the code I posted.
I should have run every path before calling it a day ;)
The code for dynamic array serialization

        ulong length = *cast(size_t*) ptr;
        const void* values = *cast(const ubyte**)(ptr + size_t.sizeof);
        auto elemType = da.nextOf;
        return serializeArray(length, ptr, elemType);

has to be

        ulong length = *cast(size_t*) ptr;
        const ubyte* values = *cast(const ubyte**)(ptr + size_t.sizeof);
        auto elemType = da.nextOf;
        return serializeArray(length, ptr, values);

and of course serializeArray has to write the length as well for this to work as without the length information you cannot de-serialize.

August 15, 2021
15.08.2021 14:18, Stefan Koch пишет:
> Good Day everyone,
> 
> A friend asked me recently of core.reflect could be used for serialization.
> 
> The code to do that generically is about 100 lines of code.
> Since it's so little I am just going to post it here verbatim.
> 
> ```D
> class C { @("NoSerialize") ubyte a; uint b; @("NoSerialize") ulong c;
>      this(ubyte a, uint b, ulong c)
>      {
>          this.a = a; this.b = b; this.c = c;
>      }
> }
> struct S { @("NoSerialize") ubyte a; uint b; ubyte c; @("NoSerialize") ulong d; }
> 
> import core.reflect.reflect;
> /// if a type is an aggregate type return the aggregate declaration
> /// otherwise return null
> AggregateDeclaration aggregateFromType(const Type type)
> {
>      AggregateDeclaration result = null;
>      if (auto ts = cast(TypeStruct)type)
>      {
>          result = ts.sym;
>      } else if (auto tc = cast(TypeClass)type)
>      {
>          result = tc.sym;
>      }
>      return result;
> }
> 
> bool hasNoSerializeAttrib(const Declaration d)
> {
>      bool NoSerialize = false;
>      foreach(attr;d.attributes)
>      {
>          auto se = cast(StringLiteral)attr;
>          if (se && se.string_ == "NoSerialize")
>          {
>            NoSerialize = true;
>            break;
>          }
>      }
>      return NoSerialize;
> }
>    /// the only template in here. only one instances per serialzed-root-type
>    const(ubyte[]) serialize(T)(T value) {
>      static immutable Type type = cast(immutable Type) nodeFromName("T");
>      return serializeType(cast(ubyte*)&value, type);
>    }
> 
>    const(ubyte[]) serializeType(const ubyte* ptr, const Type type)
>    {
>      // writeShallowTypeDescriptor(Type);
>      // printf("Serializing type: %s\n", type.identifier.ptr);
>      if (auto sa = cast (TypeArray) type)
>      {
>          ulong length = sa.dim;
>          const void* values = ptr;
>          auto elemType = sa.nextOf;
>          return serializeArray(length, ptr, elemType);
>      }
>      else if (auto da = cast(TypeSlice) type)
>      {
>          ulong length = *cast(size_t*) ptr;
>          const void* values = *cast(const ubyte**)(ptr + size_t.sizeof);
>          auto elemType = da.nextOf;
>          return serializeArray(length, ptr, elemType);
>      }
>      else if (auto ts = cast(TypeStruct) type)
>      {
>          auto fields = ts.sym.fields;
>          return serializeAggregate(ptr, fields);
>      }
>      else if (auto tc = cast(TypeClass) type)
>      {
>          auto fields = tc.sym.fields;
>          return serializeAggregate(*cast(const ubyte**)ptr, fields);
>      }
>      else if (auto tb = cast(TypeBasic) type)
>      {
>          // writeTypeTag ?
>          return ptr[0 .. type.size];
>      }
>      else assert(0, "Serialisation for " ~ type.identifier ~ " not implemented .. " ~ (cast()type).toString());
>    }
> 
>    ubyte[] serializeArray(ulong length, const ubyte* ptr, const Type elemType)
>    {
>      ubyte[] result;
>      // writeLength(length)
>      foreach(i; 0 .. length)
>      {
>        result ~= serializeType(ptr, elemType);
>      }
>      return result;
>    }
> 
>    ubyte[] serializeAggregate(const ubyte* ptr, VariableDeclaration[] fields)
>    {
>      ubyte[] result;
>      foreach(f; fields)
>      {
>        if (hasNoSerializeAttrib(f))
>        {
>          // skip fields which are annotated with noSerialize;
>          continue;
>        }
>        result ~= serializeType(ptr + f.offset, f.type);
>      }
>      return result;
>    }
> 
> 
> void main()
> {
>      S s = S(72, 19992034, 98);
>      C c = new C(72, 19992039, 98);
>      auto buffer = serialize(s);
>      assert(buffer.length == 5);
>      assert ((buffer[0] | buffer[1] << 8 | buffer[2] << 16 | buffer[3] << 24) == 19992034);
>      assert (buffer[4] == 98);
>      auto buffer_c = serialize(c);
>      assert(buffer_c.length == 4);
>      assert ((buffer_c[0] | buffer_c[1] << 8 | buffer_c[2] << 16 | buffer_c[3] << 24) == 19992039);
>      // we can see the fields annotated with @("NoSerialize") are skipped
> }
> ```
> 
> I would like to know what you think about this
> 
> In my next example I will show how you can modify serialization for library types source-code you can't control.
> 
> However for that to work `core.reflect` needs to be extended a little ;)
> 
> Cheers and have a nice day,
> 
> Stefan

I'm impressed by your work. But I have one question - when ordinary people like me can be able to use all these amazing things you did? Iterating over aggregate members considering their type, attributes and runtime value is so often in my practice that I'm really interesting in your core.reflect but I can use only official compiler w/o any customization.
August 15, 2021
On Sunday, 15 August 2021 at 12:27:44 UTC, drug wrote:
>
> I'm impressed by your work. But I have one question - when ordinary people like me can be able to use all these amazing things you did? Iterating over aggregate members considering their type, attributes and runtime value is so often in my practice that I'm really interesting in your core.reflect but I can use only official compiler w/o any customization.

I have to write a DIP and have it approved.

I am already starting as this one is much simpler than the type function project.
Essentially the changes to the core language spec is just a few lines.

I suspect the runtime documentation will be more challenging.
But for that at least I have already specified a data-structure of there is not much variability to take into account.

I cannot give a definitive timeline but I would hope for this to go in before 2022 is over.
August 16, 2021
On Sunday, 15 August 2021 at 14:21:00 UTC, Stefan Koch wrote:
> On Sunday, 15 August 2021 at 12:27:44 UTC, drug wrote:
>> [...]
>
> I have to write a DIP and have it approved.
>
> I am already starting as this one is much simpler than the type function project.
> Essentially the changes to the core language spec is just a few lines.
>
> I suspect the runtime documentation will be more challenging.
> But for that at least I have already specified a data-structure of there is not much variability to take into account.
>
> I cannot give a definitive timeline but I would hope for this to go in before 2022 is over.

Hello.
Take a look at https://github.com/Temtaime/utile/blob/main/source/utile/binary/tests.d :)
Maybe someone will found this library of me useful
August 16, 2021
On Monday, 16 August 2021 at 16:53:54 UTC, Temtaime wrote:
> On Sunday, 15 August 2021 at 14:21:00 UTC, Stefan Koch wrote:
>> On Sunday, 15 August 2021 at 12:27:44 UTC, drug wrote:
>>> [...]
>>
>> I have to write a DIP and have it approved.
>>
>> I am already starting as this one is much simpler than the type function project.
>> Essentially the changes to the core language spec is just a few lines.
>>
>> I suspect the runtime documentation will be more challenging.
>> But for that at least I have already specified a data-structure of there is not much variability to take into account.
>>
>> I cannot give a definitive timeline but I would hope for this to go in before 2022 is over.
>
> Hello.
> Take a look at https://github.com/Temtaime/utile/blob/main/source/utile/binary/tests.d :)
> Maybe someone will found this library of me useful

dlang sure seems to inspire run time serializers: https://code.dlang.org/search?q=serialization

My home brew serialization is not as sophisticated as what you've written, let alone what Stefan is doing at compile time.

August 16, 2021

On 8/16/21 5:18 PM, Bruce Carneal wrote:

>

On Monday, 16 August 2021 at 16:53:54 UTC, Temtaime wrote:

>

On Sunday, 15 August 2021 at 14:21:00 UTC, Stefan Koch wrote:

>

On Sunday, 15 August 2021 at 12:27:44 UTC, drug wrote:

>

[...]

I have to write a DIP and have it approved.

I am already starting as this one is much simpler than the type function project.
Essentially the changes to the core language spec is just a few lines.

I suspect the runtime documentation will be more challenging.
But for that at least I have already specified a data-structure of there is not much variability to take into account.

I cannot give a definitive timeline but I would hope for this to go in before 2022 is over.

Hello.
Take a look at https://github.com/Temtaime/utile/blob/main/source/utile/binary/tests.d :)

Maybe someone will found this library of me useful

dlang sure seems to inspire run time serializers: https://code.dlang.org/search?q=serialization

My home brew serialization is not as sophisticated as what you've written, let alone what Stefan is doing at compile time.

D is for (de)serialization

;)

There's at least one talk about serialization in almost every dconf I think.

-Steve

August 17, 2021

Is this runtime reflection?

Will this depend on the GC? if so does it add pressure to the GC?

We already have compile time type introspection, i don't think it's wise to move things to runtime, we have a poor GC adding more pressure to it is just bad

Compile time reflection already proved to be superior in heavy workloads

Never been a fan of runtime reflection in java/c#, not good examples to follow

Also if it uses the GC, i'm not sure "core" package is the go, should be put on "std", or as a library imo

August 17, 2021

On Tuesday, 17 August 2021 at 13:52:19 UTC, russhy wrote:

>

Is this runtime reflection?

Will this depend on the GC? if so does it add pressure to the GC?

We already have compile time type introspection, i don't think it's wise to move things to runtime, we have a poor GC adding more pressure to it is just bad

Compile time reflection already proved to be superior in heavy workloads

Never been a fan of runtime reflection in java/c#, not good examples to follow

Also if it uses the GC, i'm not sure "core" package is the go, should be put on "std", or as a library imo

The reason that they use runtime reflection in java/c# is because of the basic principles of OOP.

-Alex

August 17, 2021

On Tuesday, 17 August 2021 at 13:52:19 UTC, russhy wrote:

>

Is this runtime reflection?

Will this depend on the GC? if so does it add pressure to the GC?

We already have compile time type introspection, i don't think it's wise to move things to runtime, we have a poor GC adding more pressure to it is just bad

Compile time reflection already proved to be superior in heavy workloads

Never been a fan of runtime reflection in java/c#, not good examples to follow

Also if it uses the GC, i'm not sure "core" package is the go, should be put on "std", or as a library imo

It's for introspecting over code at compile time, not at runtime. Stefan and I have been mulling over this for ages and I think we both think it can do more than reflection as currently know it at least. This let's you drink from the firehose, so to speak.

« First   ‹ Prev
1 2