On Monday, 4 November 2019 at 08:25:11 UTC, Patrick Schluter wrote:
> On Sunday, 3 November 2019 at 21:35:18 UTC, JN wrote:
>> On Sunday, 3 November 2019 at 08:37:07 UTC, SealabJaster wrote:
>>> On Sunday, 3 November 2019 at 08:35:42 UTC, SealabJaster wrote:
>>>> On Friday, 1 November 2019 at 21:14:56 UTC, SealabJaster wrote:
>>>>> ...
>>>
>>> Sorry, seems it cut out the first half of that reply.
>>>
>>> New posts are out, and I don't want to spam Announce with new threads, so I'm just replying to this one.
>>>
>>> #1.1 https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser1_1
>>> #2 https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser2
>>
>> "This often seems to confuse people at first, especially those coming from other languages"
>>
>> I think what's confusing people is that enum (short for ENUMERATION) is suddenly used like a constant/alias.
>
> I don't get why it confuses people.
> In all languages I know (C, C++, Java, Pascal, etc..) they are used to associate a compile time symbols with some quantities, i.e. the definition of constants.
> When an enumeration only consists of 1 value, then the enumeration is this value itself.

Yes and no, because the fist case for using enum described at [1] is something very different:

> This defines a new type X which has values X.A=0, X.B=1, X.C=2:
> enum X { A, B, C }  // named enum

[1] https://dlang.org/spec/enum.html

And it might be good to change the docs to point out the very often taken use case for declaring a singe compile time constant.

On Monday, 4 November 2019 at 08:25:11 UTC, Patrick Schluter wrote:
> I don't get why it confuses people.
> In all languages I know (C, C++, Java, Pascal, etc..) they are used to associate a compile time symbols with some quantities, i.e. the definition of constants.
> When an enumeration only consists of 1 value, then the enumeration is this value itself.

If I'm not mistaken `enum` must come from "enumerate" or "enumeration", so it's easy to instinctively think of enums as having multiple possible values; using such a word for a single value is a bit counter-intuitive IMO.

On Sunday, 3 November 2019 at 08:37:07 UTC, SealabJaster wrote:
> ...

Next post is out, I feel kind of iffy over how I went over classes, but I think for the most part it came out ok. I probably won't touch classes specifically in any future post though, unless I can think of something interesting to do with them.

#3: https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser3

I'm considering making a #3.1 post (similar to the #1.1 post) that shows how __traits(hasMember), __traits(getMember)/__traits(allMembers), alongside some things in std.traits like Parameters and ReturnType, could be used instead of __traits(compiles).

Also, until run.dlang.io's ability to create links is fixed, the usual links to snippets won't take you anywhere.

On Sunday, 10 November 2019 at 10:22:17 UTC, SealabJaster wrote:
> Next post is out, I feel kind of iffy over how I went over classes, but I think for the most part it came out ok. I probably won't touch classes specifically in any future post though, unless I can think of something interesting to do with them.

It's outside of the scope of the tutorial, but the "tough" part in serializing classes is realizing that they're a reference type rather than value type. Just like with serializing pointers, I could have two references pointing to the same class object, and a smart deserializer would create only one class object and point both references to it.

On 10/30/19 8:05 PM, SealabJaster wrote:
> https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser1
> 
> Currently only the first post is out, as I'd like to collect feedback before writing any more.
> 
> The series is aimed at people new to D, or people who have heard of D, but haven't really explored its metaprogramming too much, hence certain things such as calling D's metaprogramming "often overlooked" as that tends to be true for non/new users of D.
> 
> Sorry in advance if this is the wrong forum group to post to, I didn't know whether to put it here or in the Learn group.
> 
> Future plans for this series:
>   * Serialising structs (next up).
> 
>   * Serialising enums via their names, rather than values (and certain complications such as is(SomeEnum == int) being true).
> 
>   * Serialising classes.
> 
>   * Using UDAs to let classes and structs customise their fields.
> 
>   * Using __traits(compiles) to determine if a struct or class implements a custom (de)serialisation function.
> 
>   * Trying to think of how I can shoe horn mixin templates in, just so they're shown off.
> 
> I'm welcome to any other ideas to try and fit into the series. The serialiser won't be overly robust in terms of edge cases (such as nested classes/structs needing to be filtered out sometimes), but I'll try to mention them and provide a workaround where possible.
> 
> I've also taken to certain decisions in the code snippets, to try and reduce the amount of things I need to explain at once, and to hopefully make things more readable. e.g. Using "const string" instead of just "const" or "enum" (and then having to explain why not to use "enum" for strings), giving all if and else statements brackets to make things more readable (that's the plan at least), etc.
> 
> As for release schedule, I'm planning on getting at least one post out a week.
> 
> Other than feedback for the actual content of the blog, I'd also appreciate any feedback on visual style of the blog, since I'm quite a novice at CSS, let alone writing things like this.
> 
> While I'm not really capable of contributing to D in terms of improving phobos, or writing quality libraries, I'm hoping to at least be able to provide some educational material on what I feel makes D worth it as a way of contribution.

Looks like a great idea! Sounds very similar to what I've been learning over the past 5 years or so.

See my dconf presentation from this year.

Also, if you want some inspiration for serialization/deserialization, check out jsoniopipe: http://code.dlang.org/packages/jsoniopipe

Simply put, I think it's great to use D overloading as much as possible instead of static if (reading your first article). OK, now reading the "alternate function layout", I see you did do that :) That's how I did mine, and I think it reads much nicer.

-Steve

On 11/10/19 5:50 PM, JN wrote:
> On Sunday, 10 November 2019 at 10:22:17 UTC, SealabJaster wrote:
>> Next post is out, I feel kind of iffy over how I went over classes, but I think for the most part it came out ok. I probably won't touch classes specifically in any future post though, unless I can think of something interesting to do with them.
> 
> It's outside of the scope of the tutorial, but the "tough" part in serializing classes is realizing that they're a reference type rather than value type. Just like with serializing pointers, I could have two references pointing to the same class object, and a smart deserializer would create only one class object and point both references to it.
> 

The tough part about serializing classes is if the class is not final, how do you serialize the derived data. It requires some sort of user help to tell it how to get at the data.

-Steve

On Monday, 11 November 2019 at 16:56:31 UTC, Steven Schveighoffer wrote:
> On 10/30/19 8:05 PM, SealabJaster wrote:
>> ...
>
> Looks like a great idea! Sounds very similar to what I've been learning over the past 5 years or so.
>
> See my dconf presentation from this year.

I had a quick skim, and seems like it touches upon a lot of stuff I want to talk about. Will give it a proper watch later today :)

>
> Also, if you want some inspiration for serialization/deserialization, check out jsoniopipe: http://code.dlang.org/packages/jsoniopipe

Will also give that a look over.

> Simply put, I think it's great to use D overloading as much as possible instead of static if (reading your first article). OK, now reading the "alternate function layout", I see you did do that :) That's how I did mine, and I think it reads much nicer.

I do actually prefer to go with overloads (I usually use a hybrid of overloads + static ifs, depending on the case), but it made the code snippets a bit harder to read and follow (imo) so I went with a single function instead.

> The tough part about serializing classes is if the class is not final, how do you serialize the derived data. It requires some sort of user help to
> tell it how to get at the data.

And if you do allow things such as letting classes have a 'deserialise' member function which can be overloaded, you still need to create or be given an instance of the class beforehand, which brings things back around to the "constructor issue" in the latest post.

I don't quite know if there's actually an automatic solution for classes that covers most cases? Anything I can think of places limitations in some form or another.

On 11/12/19 4:15 AM, SealabJaster wrote:
> On Monday, 11 November 2019 at 16:56:31 UTC, Steven Schveighoffer wrote:
>> The tough part about serializing classes is if the class is not final, how do you serialize the derived data. It requires some sort of user help to
>> tell it how to get at the data.
> 
> And if you do allow things such as letting classes have a 'deserialise' member function which can be overloaded, you still need to create or be given an instance of the class beforehand, which brings things back around to the "constructor issue" in the latest post.
> 
> I don't quite know if there's actually an automatic solution for classes that covers most cases? Anything I can think of places limitations in some form or another.

There's definitely not an automatic solution. However, in my jsoniopipe code, the base class can have a static function fromJSON that initializes the requested type. Such a thing has to be a member, but in theory could just be a registered callback.

In my use case, I have a base class and all the derivatives in the same module (they are just messages). Here is some code that returns the base type that is portable to any such situation:

    static Base fromJSON(JT)(ref JT tokenizer, ReleasePolicy relPol)
    {
        // this creates a rewind point, so I can go back and parse the
        // object after finding the type.
        tokenizer.startCache();
        if(tokenizer.parseTo("type")) // this part depends on the encoding
        {
            string t;
            // get the type name
            tokenizer.deserialize(t);
            import std.meta;
            // alias to the module for Base, which also contains all the
            // derivatives
            alias mod = __traits(parent, Base);
            enum isBase(string s) =
                is(__traits(getMember, mod, s) : Base) &&
                !is(__traits(getMember, mod, s) == Base);

            // this gets all the classes in this module that are
            // derivatives of Base (except Base). D is so cool ;)
            alias Derivatives = Filter!(isBase, __traits(allMembers, mod));
            // reset the tokenizer to read the whole JSON data for this object
            tokenizer.rewind();
            tokenizer.endCache();
            switch(t)
            {
                static foreach(typename; Derivatives)
                {
                // Note: MyType is the JSON string that designates the
                // type, your implementation may vary.
                case __traits(getMember, mod, typename).MyType:
                    {
                        auto result = new __traits(getMember, mod, typename);
                        // defined by the library, does a rote
                        // serialization of members.
                        tokenizer.deserializeAllMembers(result, relPol);
                        return result;
                    }
                }
            default:
                assert(false, "Unknown type identifier: " ~ t);
            }
        }
        assert(false, "Couldn't find type identifier");
    }

Basically, I never have to touch this again, thanks D. Just add a new type that derives from Base with a proper MyType member, and it gets added to the list. It just has to be added to the module itself. If your class hierarchy is spread out in different files, you can make some kind of registration scheme to handle the deserialization.

-Steve

On Tuesday, 12 November 2019 at 09:15:28 UTC, SealabJaster wrote:
> ...

Just wanted to quickly explain the delay for the next post.

Basically just boils down to energy levels being non-existant, and run.dlang.io still being broken (Don't want to have a backlog of snippets to make).

Hopefully both issues are resolved soon.

On Tuesday, 12 November 2019 at 09:15:28 UTC, SealabJaster wrote:

> And if you do allow things such as letting classes have a 'deserialise' member function which can be overloaded, you still need to create or be given an instance of the class beforehand, which brings things back around to the "constructor issue" in the latest post.
>
> I don't quite know if there's actually an automatic solution for classes that covers most cases? Anything I can think of places limitations in some form or another.

They way I handled this in Orange [1] which allows to (de)serialize third party types which you don't have any control of. First, structs have the same problem as classes: there can be private members. They way this is handled is to use `.tupleof`. `.tupleof` will bypass the protection and allows to read and write private fields:

module foo;

class Foo
{
    private int a;
    int getA() { return a; }
}

module main;

import std.stdio;
import foo;

void main()
{
    auto foo = new Foo;
    foo.tupleof[0] = 3;
    assert(foo.getA == 3);
    assert(foo.tupleof[0] == 3);
}

`.tupleof` is used both for serialization and deserialization. To create an instance of a class use the same way as the runtime does. This function [2] is called by the compiler when the code contains "new Object" (extracted to Orange here [3]). This will not call a constructor, but that's fine since we will set the fields anyway directly after.

Orange also offers a way to tweak the (de)serialization process with hooks. You can define methods in your structs or classes with any of the following UDAs: `onSerializing`, `onSerialized`, `onDeserializing` and `onDeserialized`. These method will be called (if they exist) before/after the (de)serialization process [4]. This allows you do preform some post-processing that might be needed since no constructor has been called on deserialization.

When it comes to (de)serializing derived types through a base class reference Orange requires you do register all derived types [5].

Then there's the problem with non-mutable fields and instances as well. But you can just cast away those when deserializing.

There's also the issue with circular references [6]. But that's pretty easy to solve by storing  all serialized instances to see if they have already been serialized or not.

[1] http://github.com/jacob-carlborg/orange

[2] https://github.com/dlang/druntime/blob/873fac33014c5af680c4bed69bb74cb0f192198a/src/rt/lifetime.d#L73

[3] https://github.com/jacob-carlborg/orange/blob/90f1dbb0097ba4a319805bfb7d109f7038418ac6/orange/util/Reflection.d#L149-L183

[4] https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d

[5] https://github.com/jacob-carlborg/orange/blob/90f1dbb0097ba4a319805bfb7d109f7038418ac6/orange/serialization/Serializer.d#L241-L262

[6] https://github.com/jacob-carlborg/orange/blob/master/tests/CircularReference.d

--
/Jacob Carlborg