Just trying a very simple thing and it's pretty hard: "Read a CSV file (raw_data) that has a ; separator so that I can iterate over the lines and access the fields."

	csv_data = raw_data.byLine.joiner("\n")

From the docs, which I find extremly hard to understand:

auto csvReader(Contents = string, Malformed ErrorLevel = Malformed.throwException, Range, Separator = char)(Range input, Separator delimiter = ',', Separator quote = '"')

So, let's see if I can decyphre this, step-by-step by trying out:

	csv_records = csv_data.csvReader();

Would split the CSV data into iterable CSV records using ',' char as separator using UFCS syntax. When running this I get:

	std.csv.CSVException@/Library/D/dmd/src/phobos/std/csv.d(1283): Row 1's length 0 does not match previous length of 1.

Which indicates some problem because not all fields are set in my CSV data. So let's ignore any error by specifying Malformed.ignore;

	csv_records = csv_data.csvReader(Malformed.ignore);

And now I'm lost (just showing the first candidate):

Error: template std.csv.csvReader cannot deduce function from argument types !()(Result, Malformed), candidates are:
/Library/D/dmd/src/phobos/std/csv.d(327):        csvReader(Contents = string, Malformed ErrorLevel = Malformed.throwException, Range, Separator = char)(Range input, Separator delimiter = ',', Separator quote = '"')
 with Contents = string,
      ErrorLevel = cast(Malformed)1,
      Range = Result,
      Separator = Malformed
 whose parameters have the following constraints:
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   isInputRange!Range
   is(Unqual!(ElementType!Range) == dchar)
 > isSomeChar!Separator
 - !is(Contents T : T[U], U : string)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The docs state Malformed as 2nd parameter, since I use UFCS I assume that this becomes the first parameter. I don't understand what the 3rd parameter (Range) is about. 4th parameter is my separator, which I need to set to ';' somehow.

But from the error message, it looks like DMD tries to use Malformed.ignore as the 4th (!!) Parameter being the Separator.

I'm totally confused:

* What is used as the 3rd parameter by DMD? Where does it come from?
* How to specify a ';' separator?

This is all pretty confusing...

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster

On Thursday, 14 November 2019 at 12:25:30 UTC, Robert M. Münch wrote:

> From the docs, which I find extremly hard to understand:
>
> auto csvReader(Contents = string, Malformed ErrorLevel = Malformed.throwException, Range, Separator = char)(Range input, Separator delimiter = ',', Separator quote = '"')


Contents, ErrorLevel, Range, and Separator are template (i.e. compile-time) parameters. Input, delimiter, and quote are function (i.e. runtime) parameters.

>
> So, let's see if I can decyphre this, step-by-step by trying out:
>
> 	csv_records = csv_data.csvReader();


Here, you aren't providing any template parameters and only the first function parameter, so it's the equivalent to calling the function like so:

csvReader!(string, Malformed.throwException, typeof(csv_data), char)(csv_data, ',', '"');


> Which indicates some problem because not all fields are set in my CSV data. So let's ignore any error by specifying Malformed.ignore;
>
> 	csv_records = csv_data.csvReader(Malformed.ignore);


csv_records = csv_data.csvReader!(string, Malformed.ignore)();



>
> The docs state Malformed as 2nd parameter, since I use UFCS I assume that this becomes the first parameter. I don't


Malformed is the 2nd template parameter, your UFCS value is the first function parameter.

> understand what the 3rd parameter (Range) is about.

Range is the type of the first parameter. It's common outside of Phobos use T and U for template types, but any valid symbol name can be used. This template has three type parameters which are named according to their purpose (Contents, Range, and Separator). Since Range is also the type of the first function parameter, the compiler will infer the type if you don't specify it.


> 4th parameter is my separator, which I need to set to ';' somehow.

The fourth _template_ parameter is the _type_ of your separator (and is set to default to char) not the actual separator. You want to set the delimiter, which is the second _function_ parameter.

csv_records = csv_data.csvReader!(string, Malformed.ignore)(';');

On 2019-11-14 13:08:10 +0000, Mike Parker said:

> Contents, ErrorLevel, Range, and Separator are template (i.e. compile-time) parameters. Input, delimiter, and quote are function (i.e. runtime) parameters.

Mike, thanks a lot... I feel like an idiot. As casual D programmer the template-syntax is not so easy to get used too because it's not so distinguishable.

However, your explanation helps a lot to make things much more clear now.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster

On Thursday, 14 November 2019 at 12:25:30 UTC, Robert M. Münch wrote:
> Just trying a very simple thing and it's pretty hard: "Read a CSV file (raw_data) that has a ; separator so that I can iterate over the lines and access the fields."
>
> 	csv_data = raw_data.byLine.joiner("\n")
>
> From the docs, which I find extremly hard to understand:
>
> auto csvReader(Contents = string, Malformed ErrorLevel = Malformed.throwException, Range, Separator = char)(Range input, Separator delimiter = ',', Separator quote = '"')
>
> So, let's see if I can decyphre this, step-by-step by trying out:
>
> 	csv_records = csv_data.csvReader();
>
> Would split the CSV data into iterable CSV records using ',' char as separator using UFCS syntax. When running this I get:
>
> [...]

Side comment - This code looks like it was taken from the first example in the std.csv documentation. To me, the code in the std.csv example is doing something that might not be obvious at first glance and is potentially confusing.

In particular, 'byLine' is not reading individual CSV records. CSV can have embedded newlines, these are identified by CSV escape syntax. 'byLine' doesn't know the escape syntax. If there are embedded newlines, 'byLine' will read partial records, which may not be obvious at first glance. The .joiner("\n") step puts the newline back, stitching fields and records back together again in the process.

The effect is to create an input range of characters representing the entire file, using 'byLine' to do buffered reads. This input range is passed to CSVReader.

This could also be done using 'byChunk' and 'joiner' (with no separator). This would use a fixed size buffer, no searching for newlines while reading, so it should be faster.

An example:

==== csv_by_chunk.d ====
import std.algorithm;
import std.csv;
import std.conv;
import std.stdio;
import std.typecons;
import std.utf;

void main()
{
    // Small buffer used to show it works. Normally would use a larger buffer.
    ubyte[16] buffer;
    auto stdinBytes = stdin.byChunk(buffer).joiner;
    auto stdinDChars = stdinBytes.map!((ubyte b) => cast(char) b).byDchar;

    writefln("--------------");
    foreach (record; stdinDChars.csvReader!(Tuple!(string, string, string)))
    {
        writefln("Field 0: |%s|", record[0]);
        writefln("Field 1: |%s|", record[1]);
        writefln("Field 2: |%s|", record[2]);
        writefln("--------------");
    }
}

Pass it csv data without embedded newlines:

$ echo $'abc,def,ghi\njkl,mno,pqr' | ./csv_by_chunk
--------------
Field 0: |abc|
Field 1: |def|
Field 2: |ghi|
--------------
Field 0: |jkl|
Field 1: |mno|
Field 2: |pqr|
--------------

Pass it csv data with embedded newlines:

$ echo $'abc,"LINE 1\nLINE 2",ghi\njkl,mno,pqr' | ./csv_by_chunk
--------------
Field 0: |abc|
Field 1: |LINE 1
LINE 2|
Field 2: |ghi|
--------------
Field 0: |jkl|
Field 1: |mno|
Field 2: |pqr|
--------------

An example like this may avoid the confusion about newlines. Unfortunately, the need to do the odd looking conversion from ubyte to char/dchar is undesirable in a code example. I haven't found a cleaner way to write that. If there's a nicer way I'd appreciate hearing about it.

--Jon