https://youtu.be/Y-cgh-rwoC8

I was watching Steven's 2019 talk. I'd like to put together a DIP to specify a collection of attributes which target this very common pattern of marking up structure. We see it in c# and Java at runtime, and we see it with different D libraries.

Adding and replacing serialization to the standard library has had its delays due to expectations of how serialization should be designed.

I think it is clear that attributes on structures is a common approach placed on top of the actual implementation.

Phobos updates are not traditionally going through a DIP, but I think this is the right avenue for review.

We see similar attributes used in things like argument parsing. But here is a short list of what I am thinking from an attribute list.

@ignore
@optional
@name("field")

Seeing this work through will be hard with a 3y and 3m old, but I feel this is the type of library support std can provide even with the benefits of dub.

Destroy.

On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips wrote:
> @ignore
> @optional
> @name("field")
>
> Seeing this work through will be hard with a 3y and 3m old, but I feel this is the type of library support std can provide even with the benefits of dub.
>
> Destroy.

This should have a place in phobos, because it makes interoperability across libraries easier.

On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips wrote:

> Destroy.

WIP https://github.com/JesseKPhillips/DIPs/blob/serialize/attribute/DIPs/1NNN-jkp.md

On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips wrote:
> @optional

If we look at e.g. json-schema.org or openapis.org they seem to have optional by default. So we should at least consider this as well, e.g.

1. have optional by default and introduce @required
2. make it configurable on the containing struct / class

enum IfMissing {
    setDefault,
    keep
}

@missing(IfMissing.SetDefault)
struct S
{
    // all fields by default optional due to @missing
    @required
    int x;
    int a = 4; // set to 0 if not provided during deserialization
}

3. something else ..

On 12/8/19 4:19 PM, Tobias Pankrath wrote:
> On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips wrote:
>> @optional
> 
> If we look at e.g. json-schema.org or openapis.org they seem to have optional by default. So we should at least consider this as well, e.g.
> 
> 1. have optional by default and introduce @required
> 2. make it configurable on the containing struct / class
> 
> enum IfMissing {
>      setDefault,
>      keep
> }
> 
> @missing(IfMissing.SetDefault)
> struct S
> {
>      // all fields by default optional due to @missing
>      @required
>      int x;
>      int a = 4; // set to 0 if not provided during deserialization
> }
> 
> 3. something else ..

First, I think the initiative is a good one. But in certain ways it is not enough.

I will tell you from first hand experience. I have types in my proprietary back end which are serialized in 2 ways -- one to the database, and one to a REST route as JSON. Specifically for one method of serialization, some things are ignored. For the other method of serialization, other things are ignored. For example, in the database, I may care very much that the id is serialized. But over Json, not at all.

So, the conclusion I came to (and actually, I use), is that I need to tag my members with BOTH, and have some way to distinguish them. That is, vibe.data.serialization.optional is a DIFFERENT attribute than db.DB.optional. And in my case, I use a DB struct, so all my attributes look like @(DB.optional) or @(DB.name!"class")

Decisions like whether the default is required or optional is very implementation dependent. Even the same mechanisms that are used in one project (i.e. Json serialization) may require a different default mode in another project.

And types may be defined that are used in both.

So how do we sort this out with a "universal" UDA system? I don't really know the answer. So far in my travels, I haven't needed to serialize types other than my own with serialization markers on it. So I control everything.

But that isn't always going to be the case. Perhaps in some cases, we need something other than UDA-based markers. Even setting the default may not be enough, maybe you need to generate a "dummy" struct which has the right UDAs on it. I go over something similar in this d meetup talk I did a while back: https://www.youtube.com/watch?v=ZxzczSDaobw

Part of me really likes the idea of a universal UDA system for serialization, to make things more general. But the other part of me feels like it's not a problem that can be solved universally -- it needs a local aspect. Is there a way to make it work both ways?

-Steve

On Sunday, 8 December 2019 at 22:00:53 UTC, Steven Schveighoffer wrote:
> Part of me really likes the idea of a universal UDA system for serialization, to make things more general. But the other part of me feels like it's not a problem that can be solved universally -- it needs a local aspect. Is there a way to make it work both ways?
>
> -Steve

I think serde in Rust is a good reference to look around. It's the de facto way of doing serialization in Rust and they have a list of attributes that are available: https://serde.rs/attributes.html

On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips wrote:
> https://youtu.be/Y-cgh-rwoC8
>
> I was watching Steven's 2019 talk. I'd like to put together a DIP to specify a collection of attributes which target this very common pattern of marking up structure. We see it in c# and Java at runtime, and we see it with different D libraries.
>
> Adding and replacing serialization to the standard library has had its delays due to expectations of how serialization should be designed.

Then your DIP will not address the problem. As far as I can see it proposes conventions about the attributes to use on fields that have to be read from and written to. Is the lack of attributes really the reason why there's no std.serializer ?

A few remarks however. You want to collect the different attributes while it would be possible to use a single attribute... a class. This would allow to store functions pointers, tables or I don't know what else could be useful for a finely grained control of the serialization.

On Sunday, 8 December 2019 at 22:00:53 UTC, Steven Schveighoffer wrote:

> So, the conclusion I came to (and actually, I use), is that I need to tag my members with BOTH, and have some way to distinguish them. That is, vibe.data.serialization.optional is a DIFFERENT attribute than db.DB.optional. And in my case, I use a DB struct, so all my attributes look like @(DB.optional) or @(DB.name!"class")

I figured this would be likely. I took a look and it would be possible to specify a template argument to the attribute, the libraries could supply a type, we could supply some generic ones (DB, Web, Disk, Unspecified)

I'll watch the video.

On 2019-12-08 09:22, Tobias Pankrath wrote:

> This should have a place in phobos, because it makes interoperability across libraries easier.

No, it should be in druntime. Then it's possible to attach the UDA's to symbols both in Phobos and druntime. For example, adding @ignore to Thread.

-- 
/Jacob Carlborg

On Wednesday, 11 December 2019 at 18:53:56 UTC, Jacob Carlborg wrote:
> On 2019-12-08 09:22, Tobias Pankrath wrote:
>
>> This should have a place in phobos, because it makes interoperability across libraries easier.
>
> No, it should be in druntime. Then it's possible to attach the UDA's to symbols both in Phobos and druntime. For example, adding @ignore to Thread.

This points me in a completely different direction: you can't annotate 3rd party libraries with annotations that these libraries are unaware of.

I worked on a expansive C# project at one time where a 3rd party GUI control was extracting info about presentation of properties from attributes via reflection. The classes I had to feed into it were data model classes that couldn't depend on the GUI. The resulting code bent over backwards to inject the required attributes at runtime based on a separately maintained database that covered thousands of properties. I'm still amazed that this worked. The main issue was keeping the database in sync with the model code. Compile time checks on the consistency of these external attributes would have been great.