Hi

Ok first ... is this possible? Or wise?
Being mostly self taught with regards to programming I sometimes wonder if the way I'm going about something is even the right approach, never mind whether my code is logical. So if this seems ludicrous then please tell me my approach is silly and point me in the right direction ... :)

Ok so ...

I was reading in some data from a csv and wanted to read it into a struct called.
My problem is
1) my full csv is 4000 variables wide, and I really don't want to declare each variable in the struct (I've created a smaller one to test)
2) what if I wanted to read in different csvs of different sizes, I don't want to have to change my program each time?

After much googling, I thought I found something on RosettaCode that would do what I wanted (http://rosettacode.org/wiki/Add_a_variable_to_a_class_instance_at_runtime#D)
but when I try to use it, it doesn't work. (Its actually adding to a class, which may be better than a struct in this instance, but regardless, it doesn't work)

As my programming skills are mainly obtained through finding other work that does something similar to what I desire and stealing/hacking it, please be gentle if there is some glaringly foolish mistake that is obvious. :S

Thanks in advance
Brian

My code is:
module main;

import std.stdio;
import std.csv;
import std.string;
import std.file;
import std.array;
import std.variant;

class dataVector(T)
{
    private T[string] vars;
    @property T opDispatch(string key)() pure nothrow
    {
        return vars[key];
    }

    @property void opDispatch(string key, U)(U value)/*pure*/ nothrow
    {
        vars[key] = value;
    }
};

void fillVector(int size, string filenameToUse)
{
    writeln(size);
    uint loopNum = size;
    do
    {
        writeln(loopNum);
    //    auto test = dataVector!Variant();
/*
        auto file = File(filenameToUse, "r");
        foreach(line;file.byLine())
        {
            foreach(ob;csvReader!Data(text))
            {
                dataVector.size ~= ob;
            }
        }*/
// ****************************************
// according to the RosettaCode this is how I should be able
// to create a variable a in the class test with a value of loopNum + 1
// ****************************************
        test.a = loopNum + 1;

 //       writeln(test.a);
        loopNum--;
    }
    while(loopNum != 0);
}

void main()
{
    string input;

    string filename = "/A/Path/To/The/Dark/Side/output.csv";
    auto file = File(filename, "r");

/* this is super dynamic, but I've managed to create a different function to calculate this, which I have omitted to keep the code concise */
    int numOfVars = 11;
    if (numOfVars > 0)
    fillVector(numOfVars, filename);
}

The csv I would eventually like to read in is of the form:

test1,303,-140,-166,-317,-310,414,-157,-360,-120,-89
test10,-1,70,-57,-101,112,137,-134,9,195,86
test100,367,78,-417,123,220,-234,-170,236,-218,-351
test1000,309,-178,-674,-202,514,218,-165,76,-82,-328
test10000,-131,142,6,-143,80,46,29,48,-84,-113

There is a way to get what you described, and your code is fairly close... but there's a much simpler way too.

from the std.csv docs:

==

 When an input contains a header the $(D Contents) can be specified as an
 associative array. Passing null to signify that a header is present.

 -------
 auto text = "Name,Occupation,Salary\r"
     "Joe,Carpenter,300000\nFred,Blacksmith,400000\r\n";

 foreach(record; csvReader!(string[string])
         (text, null))
 {
     writefln("%s works as a %s and earns $%s per year.",
              record["Name"], record["Occupation"],
              record["Salary"]);
 }
 -------

==

The associative array directly is the simplest way to get this to work. All items read will be of type string if you do it like this, and you access with the [] syntax.

You can convert to other types with to when you want to use it:

import std.conv;

int salary = to!int(record["salary"]);



and that's the simplest way to make it work.

On Mon, Feb 18, 2013 at 01:44:49AM +0100, Brian Brady wrote: [...]
> I was reading in some data from a csv and wanted to read it into a
> struct called.
> My problem is
> 1) my full csv is 4000 variables wide, and I really don't want to
> declare each variable in the struct (I've created a smaller one to
> test)
> 2) what if I wanted to read in different csvs of different sizes, I
> don't want to have to change my program each time?
[...]

It seems to me that what you need is not any struct or class per se, but a runtime container that stores different kinds of data depending on runtime input. I would recommend using an associative array instead of trying to shoehorn dynamic data into struct fields. Say something like:

	import std.variant;

	Variant[string] data;
	data[fieldName] = value;
	...
	auto myField = data[fieldName];
	...

Etc.


T

-- 
There is no gravity. The earth sucks.

On Monday, 18 February 2013 at 00:52:12 UTC, Adam D. Ruppe wrote:
[...]
>  -------
>  auto text = "Name,Occupation,Salary\r"
>      "Joe,Carpenter,300000\nFred,Blacksmith,400000\r\n";
>
>  foreach(record; csvReader!(string[string])
>          (text, null))
>  {
>      writefln("%s works as a %s and earns $%s per year.",
>               record["Name"], record["Occupation"],
>               record["Salary"]);
>  }

doesn't this writefln() assume that I know what the Variables are called? ie "Name", "Occupation", etc.?
I want to be able to run the same program against a file with 4 variables or a file with 400 variables, so specifying names wouldn't work.
Can I somehow use a record[var[a]] where a can be a number between 0 and the count of variables?

Also, if I wanted to store this as I read it in from csvReader, how can I define the array? It'll be of differing sizes if my 2 csv's are of different sizes. This kinda the crux of my problem.

string[string] Data;
is my first guess based on http://dlang.org/hash-map.html

but then this doesn't work:
     foreach(line;file.byLine())
         foreach(ob;csvReader!Data(line))
         {
             // do things
         }

On 02/17/2013 04:44 PM, Brian Brady wrote:

> 1) my full csv is 4000 variables wide, and I really don't want to
> declare each variable in the struct (I've created a smaller one to test)

Looking at the sample file you provide, what you call "variables" look like data points.

> 2) what if I wanted to read in different csvs of different sizes, I
> don't want to have to change my program each time?

Then you need something other than a CSV reader. Ordinarily, records in a CSV files have a known number of fields.

> After much googling, I thought I found something on RosettaCode that
> would do what I wanted
> (http://rosettacode.org/wiki/Add_a_variable_to_a_class_instance_at_runtime#D) 

>
> but when I try to use it, it doesn't work. (Its actually adding to a
> class, which may be better than a struct in this instance, but
> regardless, it doesn't work)

That is not a natural idiom for D and I don't think it is needed here. :)

> The csv I would eventually like to read in is of the form:
>
> test1,303,-140,-166,-317,-310,414,-157,-360,-120,-89
> test10,-1,70,-57,-101,112,137,-134,9,195,86
> test100,367,78,-417,123,220,-234,-170,236,-218,-351
> test1000,309,-178,-674,-202,514,218,-165,76,-82,-328
> test10000,-131,142,6,-143,80,46,29,48,-84,-113

What I see there is a label and a number of integers. Here is a simple parser that takes advantage of the %( and %) grouping format specifiers:

import std.stdio;
import std.format;

struct Data
{
    string label;
    int[] values;
}

int main(string[] args)
{
    if (args.length != 2) {
        stderr.writefln("Usage: %s <input-file-name>", args[0]);
        return 1;
    }

    auto file = File(args[1], "r");

    Data[] data;

    foreach (line; file.byLine) {
        string label;
        int[] values;
        formattedRead(line, "%s,%(%s,%)", &label, &values);
        data ~= Data(label, values);
    }

    writeln(data);

    return 0;
}

Ali

-- 
D Programming Language Tutorial: http://ddili.org/ders/d.en/index.html

On Monday, 18 February 2013 at 05:26:39 UTC, Ali Çehreli wrote:

Thank you for the working solution.

[...]
> Looking at the sample file you provide, what you call "variables" look like data points.
Yes, apologies. Different languages using different terms to describe, essentially the same thing.

[...]
> Then you need something other than a CSV reader. Ordinarily, records in a CSV files have a known number of fields.
While you would assume the lines within a csv contain a specific number of 'data points' surely the width of the csv is as variable as the data within them? This is what I want to code around.

[...]
> That is not a natural idiom for D and I don't think it is needed here. :)

That's ok. Like I said, I open to be pointed in a new, better direction.

[...]
>
> What I see there is a label and a number of integers. Here is a simple parser that takes advantage of the %( and %) grouping format specifiers:
>
Aye, in this instance it is just that, but I was looking for a more generalised code to account for, for example, when the first 2 are labels, or the first and third ...

Thanks everyone for the help.

Regards
Brian

Brian Brady:

> After much googling, I thought I found something on RosettaCode that would do what I wanted (http://rosettacode.org/wiki/Add_a_variable_to_a_class_instance_at_runtime#D)
> but when I try to use it, it doesn't work. (Its actually adding to a class, which may be better than a struct in this instance, but regardless, it doesn't work)

Since some time I am maintaining most of the D code on Rosettacode. What's broken in that program? "it doesn't work" is too much vague.

Bye,
bearophile

On 02/18/2013 02:55 AM, Brian Brady wrote:
> On Monday, 18 February 2013 at 05:26:39 UTC, Ali Çehreli wrote:

>> What I see there is a label and a number of integers. Here is a simple
>> parser that takes advantage of the %( and %) grouping format specifiers:
>>
> Aye, in this instance it is just that, but I was looking for a more
> generalised code to account for, for example, when the first 2 are
> labels, or the first and third ...

You mean like this?

10,20,label,1,2,3,4

Then what are 10 and 20 on that line? Do they belong to the previous label? If so, I think this format is too free-form to be parsed by a general solution like csvReader. It looks like something special needs to be done and it is hard to say without knowing the meanings of 10 and 20 above. :)

Ali

On Monday, 18 February 2013 at 12:28:22 UTC, bearophile wrote:
[...]
> Since some time I am maintaining most of the D code on Rosettacode. What's broken in that program? "it doesn't work" is too much vague.
>
> Bye,
> bearophile

Apologies. My wording was poor. I believe the RosettaCode code worked for that example, but when I tried to get it to work for my purposes I could not, and could not figure out why. Possibly because I was also trying to incorporate reading from a file at the same time.

To be honest, I'm not really sure what is going on in the struct definition to even know where my code was possibly failing. I'd like to take time to figure it out, but it has a lot of parts that I don't understand, so it'll be a while.

Cheers
Brian

On Monday, 18 February 2013 at 16:54:59 UTC, Ali Çehreli wrote:
[...]
>
> You mean like this?
>
> 10,20,label,1,2,3,4
>
> Then what are 10 and 20 on that line? Do they belong to the previous label? If so, I think this format is too free-form to be parsed by a general solution like csvReader. It looks like something special needs to be done and it is hard to say without knowing the meanings of 10 and 20 above. :)
>
> Ali

I don't mind reading them all in as strings, and then converting them. I imagine I can do that relatively easily (famous last words) by checking if all the entries in the string are numeric and if so casting to an int/long/double etc.

My question was aimed at how to store them really. What was the best dynamic container type that I could use to store data points read from csvs.

I want something which is robust enough that I don't have to keep changing the code every time I change my csv.

So a program that reads in a csv of the form:

"string, number, number, number"

and stores it in a container/matrix/associative array

could also read in

"number, number, string, number, string, number, number, number"

and store it in a container with similar properties.
Once I can get both csvs input in the same manner, I can work on determining what is in each container, and go from there.

While this may seem madness, I hope my explanation describes what I am trying to achieve?

Brian