Hi,

I'm trying this example:

https://dlang.org/phobos/std_csv.html

```
struct Layout
{
    string name;
    int value;
    double other;

    @csvIgnore
    Layout* parent;  // add extra pointer
}

auto records = text.csvReader!Layout(';');
```

Right now, if we add this extra pointer, the compiler will error out:

https://run.dlang.io/is/EvU6Ee

/dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): Error: template std.conv.toImpl cannot deduce function from argument types !(Layout*)(string), candidates are:
...

I think if we can add add uda (e.g. @csvIgnore) support in std.csv, it will make the library more flexible to use.

Thoughts?

On 2020-09-08 06:53, mw wrote:
> Hi,
> 
> I'm trying this example:
> 
> https://dlang.org/phobos/std_csv.html
> 
> ```
> struct Layout
> {
>      string name;
>      int value;
>      double other;
> 
>      @csvIgnore
>      Layout* parent;  // add extra pointer
> }
> 
> auto records = text.csvReader!Layout(';');
> ```
> 
> Right now, if we add this extra pointer, the compiler will error out:
> 
> https://run.dlang.io/is/EvU6Ee
> 
> /dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): Error: template std.conv.toImpl cannot deduce function from argument types !(Layout*)(string), candidates are:
> ...
> 
> I think if we can add add uda (e.g. @csvIgnore) support in std.csv, it will make the library more flexible to use.

How about a more general attribute in `core.attribute`, like `@nonSerialized`? This could be used by third party serialization libraries. Then it could also be used by druntime and Phobos to ignore things that should not be serializable. Like threads, processes and sockets.

-- 
/Jacob Carlborg

On Thursday, 10 September 2020 at 14:57:11 UTC, Jacob Carlborg wrote:
> On 2020-09-08 06:53, mw wrote:
>> Hi,
>> 
>> I'm trying this example:
>> 
>> https://dlang.org/phobos/std_csv.html
>> 
>> ```
>> struct Layout
>> {
>>      string name;
>>      int value;
>>      double other;
>> 
>>      @csvIgnore
>>      Layout* parent;  // add extra pointer
>> }
>> 
>> auto records = text.csvReader!Layout(';');
>> ```
>> 
>> Right now, if we add this extra pointer, the compiler will error out:
>> 
>> https://run.dlang.io/is/EvU6Ee
>> 
>> /dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): Error: template std.conv.toImpl cannot deduce function from argument types !(Layout*)(string), candidates are:
>> ...
>> 
>> I think if we can add add uda (e.g. @csvIgnore) support in std.csv, it will make the library more flexible to use.
>
> How about a more general attribute in `core.attribute`, like `@nonSerialized`? This could be used by third party serialization libraries. Then it could also be used by druntime and Phobos to ignore things that should not be serializable. Like threads, processes and sockets.

Existing serialization libraries [1][2] already define their own attributes for this sort of thing. I don't think there's anything to be gained by attempting to force standardization here.

If Phobos and/or the D ecosystem does someday decide to standardize on a shared @ignore attribute, std.csv can be easily upgraded for compatibility by adding `alias csvIgnore = core.attribute.ignore`.

[1] https://code.dlang.org/packages/asdf
[2] https://vibed.org/api/vibe.data.serialization/

On 9/10/20 1:57 PM, Paul Backus wrote:
> On Thursday, 10 September 2020 at 14:57:11 UTC, Jacob Carlborg wrote:

>> How about a more general attribute in `core.attribute`, like `@nonSerialized`? This could be used by third party serialization libraries. Then it could also be used by druntime and Phobos to ignore things that should not be serializable. Like threads, processes and sockets.
> 
> Existing serialization libraries already define their own attributes for this sort of thing. I don't think there's anything to be gained by attempting to force standardization here.
> 
> If Phobos and/or the D ecosystem does someday decide to standardize on a shared @ignore attribute, std.csv can be easily upgraded for compatibility by adding `alias csvIgnore = core.attribute.ignore`.
> 

To add to this, I have code that serializes the same types for both vibe.d JSON for REST interfaces, AND to a database. In one case, I want certain fields to be ignored (e.g. no reason to send internal database ids to the browser), and in another case I want other fields to be ignored.

I'd rather have libraries define their own UDAs, and not expect Phobos to design a UDA system that pleases all parties.

-Steve

On Tuesday, 8 September 2020 at 04:53:26 UTC, mw wrote:
> Hi,
>
> I'm trying this example:
>
> https://dlang.org/phobos/std_csv.html
>
> ```
> struct Layout
> {
>     string name;
>     int value;
>     double other;
>
>     @csvIgnore
>     Layout* parent;  // add extra pointer
> }
>
> auto records = text.csvReader!Layout(';');
> ```
>
> Right now, if we add this extra pointer, the compiler will error out:
>
> https://run.dlang.io/is/EvU6Ee
>
> /dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): Error: template std.conv.toImpl cannot deduce function from argument types !(Layout*)(string), candidates are:
> ...
>
> I think if we can add add uda (e.g. @csvIgnore) support in std.csv, it will make the library more flexible to use.
>
> Thoughts?

I do run into this problem today, i.e. my struct need some extra non-simple-scalar fields (e.g. SysTime) other than those defined in the csv file, I'm wondering what kind of work-around people use in such situation?

Thanks.

On Thursday, 10 September 2020 at 19:11:03 UTC, Steven Schveighoffer wrote:
> To add to this, I have code that serializes the same types for both vibe.d JSON for REST interfaces, AND to a database. In one case, I want certain fields to be ignored (e.g. no reason to send internal database ids to the browser), and in another case I want other fields to be ignored.
>
> I'd rather have libraries define their own UDAs, and not expect Phobos to design a UDA system that pleases all parties.
>
> -Steve

I have sometimes used a dedicated struct just to let the serialiser know how to deal with the fields. It would mostly be a copy of the original struct but with some udas here and there. It allows me to have different shapes even when targeting the same format.

Wrt to csv, you could serialise into an intermediate struct first, and from that into your target struct. It wouldn't be hard to convert struct a { string timestamp, name; double value; } into struct ab { Datetime timestamp; string name; double value; }

On Thursday, 10 September 2020 at 14:57:11 UTC, Jacob Carlborg wrote:
> On 2020-09-08 06:53, mw wrote:
>> [...]
>
> How about a more general attribute in `core.attribute`, like `@nonSerialized`? This could be used by third party serialization libraries. Then it could also be used by druntime and Phobos to ignore things that should not be serializable. Like threads, processes and sockets.

Ready to use module for this purpose.
http://mir-algorithm.libmir.org/mir_serde.html

On Saturday, 10 October 2020 at 10:13:55 UTC, 9il wrote:
>
> Ready to use module for this purpose.
> http://mir-algorithm.libmir.org/mir_serde.html

Nice, but this is in a 3rd-party library, and std.csv is the DLang standard library.

Maybe can you create a PR just contribute this module to std?

(I've read there are some issues regarding licensing to contribute the whole mir to stb lib).

On Saturday, 10 October 2020 at 16:52:32 UTC, mw wrote:
> On Saturday, 10 October 2020 at 10:13:55 UTC, 9il wrote:
>>
>> Ready to use module for this purpose.
>> http://mir-algorithm.libmir.org/mir_serde.html
>
> Nice, but this is in a 3rd-party library, and std.csv is the DLang standard library.
>
> Maybe can you create a PR just contribute this module to std?
>
> (I've read there are some issues regarding licensing to contribute the whole mir to stb lib).

We are making Mir the real standard D library.

On Sunday, 11 October 2020 at 14:04:47 UTC, 9il wrote:
> On Saturday, 10 October 2020 at 16:52:32 UTC, mw wrote:
>> On Saturday, 10 October 2020 at 10:13:55 UTC, 9il wrote:
>>>
>>> Ready to use module for this purpose.
>>> http://mir-algorithm.libmir.org/mir_serde.html
>>
>> Nice, but this is in a 3rd-party library, and std.csv is the DLang standard library.
>>
>> Maybe can you create a PR just contribute this module to std?
>>
>> (I've read there are some issues regarding licensing to contribute the whole mir to stb lib).
>
> We are making Mir the real standard D library.

This is excellent news.

BTW, what namespace it will use? std.????

From a purely marketing / promotional perspective, may I suggest numD?

People will immediately recognize numPy => numD, and if we can even make the interface the same, people may adopt it more quickly. I know there is numir project, but there are several repos under https://github.com/libmir, as a user I sometimes got confused which one to use, can they *all* be merged into a single library numD?