I'm working on a program to do statistics on matrices of different sizes, and I've run into a handful of situations where I just can't seem to find the trick I need.  In general, I'm trying to make my functions work on static arrays of the proper size, and template that up for the different sizes I need to support.  I appeal to your experience and mercy on the various questions below.  Any help on any point appreciated.

1.  How can I cast a single dimension dynamic array to a multi-dimension static array?  I'm trying to do roughly the following:

   auto data = cast(float[100][100])std.file.read(fname, fsize);

which fails on the cast.  Is there any way to treat they memory returned
from read as a static array?  If I want to avoid the copy, am I relegated back
to pointers?  Is there a cast path through a pointer that would work?  I think
I'm essentially asking if I can make a reference to a static array and assign
that to the memory contained in a dynamic array.  I have a suspicion about the
answer....

2.  To work around 1., I've left one dimension of the cast floating.  This
means to bind both dimensions in my template args, I can no longer use
type parameter inference, so I have to explicitly instantiate the template
with the static size, but pass in a type that's dynamic in one dimension.  Is
there a way to deduce the equivalent static array type from the dynamic array?
(See code below).  How about just from a static array?  Even the following
fails:

   U foo(T: U[C][R], U, size_t C, size_t R)(T data, size_t c, size_t r)
   {
      return data[r][c];
   }

   int[4][5] data = [ [...] ];
   int bar = foo(data, 2, 3);

but

   int bar = foo!(int[4][5])(data, 2, 3);

works.  Why?

If I could solve that, then next I'd need to make this work:

   U foo(T: U[C][R], U, size_t C, size_t R)(U[C][] data, size_t c, size_t r)
   {
      return data[r][c];
   }

   int[4][] data = [ [...] ];
   int bar = foo(data, 2, 3);

Extra points for that one!


3.  I wanted to have a const static lookup table in a struct to map a string to an
int.  I figured an Associative array would be a good way to index by string.
But I was not able to give a literal AA.  I had to put it in static this().  I
assume that's because AA's aren't really initialized by the compiler, so
they're not really literals.  They're just arguments to some run-time function
that actually initializes them.  Is that right?  If so, that's too bad--making
different rules for different array types makes them feel like they're not
/really/ part of the language.  Plus, having immutable lookups, fixed at
compile time, would be really useful.

4.  Lastly, compiling.  In src/main.d, I had "import data.stats;", and in
src/data/stats.d, I had "module data.stats;".  On the command line in src, I figured
I'd just have to say "dmd main.d" and I'd be good to go.  Or maybe "dmd -I.
main.d" or "dmd -Idata main.d".  But I had to specify both files for the
compiler, as in "dmd main.d data/stats.d" to avoid undefined symbols from
stat.d at link time.  Is that right?  If so, how does dmd know where to get
phobos files from?  Is there a slicker way to do this with dmd?

Thanks,
Jason

Forgot a couple of things:

- this is all using D2.047.

- Another question (in reference to part 2 before):  I'd like to support about 4 base types and 5 or 6 different matrix sizes.  So that's roughly 20 type combinations for my template.  But I decide these based on command-line arguments at run time.  So I need some slick way of mapping the run-time element size, row count, and column count into a static array type to instantiate my template with. Anybody have any good ideas how to make that table-driven or something?

Thanks again,
Jason

Jason Spencer:
> 1.  How can I cast a single dimension dynamic array to a multi-dimension
> static array?  I'm trying to do roughly the following:
>    auto data = cast(float[100][100])std.file.read(fname, fsize);

I have tried a natural implementation, but it doesn't work and I don't know why it doesn't work or how to fix it:

import std.file, std.stdio;
void main() {
    enum int N = 10;
    auto filename = "data.dat";
    auto data = slurp!(float[N][N])(filename, "%s")[0];
}


So I have written an ugly version:

import std.file, std.stdio, std.contracts;

void main() {
    enum int N = 10;
    alias float DataType;
    auto filename = "data.dat";

    // be careful with endiness
    enum size_t data_size = N * N * DataType.sizeof;
    void[] raw = std.file.read(filename, data_size);
    enforce(raw.length == data_size);
    DataType[N][N] data = void;
    data[0].ptr[0 .. N*N] = (cast(DataType[])raw)[];
}


But that creates dynamic arrays, copies them, etc. Ugly, unsafe and inefficient. So better to have unsafe and efficient code, going back to C:

import std.contracts, std.c.stdio;

void main() {
    enum int N = 10;
    alias float DataType;
    auto filename = "data.dat";

    DataType[N][N] data = void;

    auto fin = fopen((filename ~ '\0').ptr, "rb");
    enforce(fin != null);

    // be careful with endiness
    size_t nread = fread(data[0].ptr, DataType.sizeof, N * N, fin);
    enforce(nread == N * N);

    fclose(fin);
}


In D fixed-sized nD arrays are contiguous, so I think that code is correct.
Be careful with floats endinaness.
Other people maybe can give you better solutions. But the point is that a minimally useful Phobos2 must be able to read/write a binary basic type (like a matrix) from disk with no extra copying and allocations and with a safe and nice syntax.


> They're just arguments to some run-time function
> that actually initializes them.

Currently AAs are (mostly) run-time things. Things can change in future as CTFE gets better.


> 4.  Lastly, compiling.  In src/main.d, I had "import data.stats;", and in
> src/data/stats.d, I had "module data.stats;".  On the command line in src, I figured
> I'd just have to say "dmd main.d" and I'd be good to go.  Or maybe "dmd -I.
> main.d" or "dmd -Idata main.d".  But I had to specify both files for the
> compiler, as in "dmd main.d data/stats.d" to avoid undefined symbols from
> stat.d at link time.  Is that right?  If so, how does dmd know where to get
> phobos files from?  Is there a slicker way to do this with dmd?

<sarcasm>Walter seems to think that Java and Python are too much handy on this, and he thinks it's not the work of the compiler to look for the file it needs by itself. Probably because this can become slow if a project has thousands of source files. So to make things clean and tidy it's better to never make dmd handy, even if your project is 2 files long. So you need to use an external tool to solve this problem. I use one named "bud" not being able to use the built-in one named rdmd, and Andrei didn't help me to learn how to use it and I have not found docs about it. There are other tools good for larger projects that of course are much harder to use.</sarcasm>  I have never appreciated all this situation. Good luck.


> But I decide
> these based on command-line arguments at run time.  So I need some
> slick way of mapping the run-time element size, row count, and column
> count into a static array type to instantiate my template with.

A basic brutal way to do that is to use a switch, that switches according to the runtime value to use the correctly compile-time instantiated template. If the switch becomes too much large you can use a string mixin to shrink it to few lines.

Bye,
bearophile

== Quote from bearophile (bearophileHUGS@lycos.com)'s article
Thanks for all the suggestions!

A little more discussion:

> > So I need some slick way of mapping the run-time element size, row count, and column count into a static array type to instantiate my template with.

> A basic brutal way to do that is to use a switch, that switches according to the
runtime value to use the correctly compile-time instantiated template. If the switch becomes too much large you can use a string mixin to shrink it to few lines.

I had thought of the switch, and it would be too large.  Can you give me an idea of what
the mixin solution would look like?  Suppose I have three strings, type, rows, and cols
that get set at runtime, where type = ["int" | "float" | "short"], rows and cols =
["100" | "200" | "250"].  What I want is to take these 3 variables and come up with:
foo!(int[100][100])(...)
foo!(int[100][200])(...)
...
foo!(short[250][250])(...)

for the right cases, without switching on type, rows, or cols.

Thanks again.
Jason

Jason Spencer:
> I had thought of the switch, and it would be too large.  Can you give me an idea of what
> the mixin solution would look like?  Suppose I have three strings, type, rows, and cols
> that get set at runtime, where type = ["int" | "float" | "short"], rows and cols =
> ["100" | "200" | "250"].  What I want is to take these 3 variables and come up with:
> foo!(int[100][100])(...)
> foo!(int[100][200])(...)
> ...
> foo!(short[250][250])(...)
> 
> for the right cases, without switching on type, rows, or cols.

The brutal way:


import std.stdio: writeln;

void foo(T, int N, int M)() {
    writeln(typeid(T), " ", N, " ", M);
}

enum Type { int_type, float_type, short_type }
enum Size { small = 100, medium = 200, large = 250 }


void main() {
    Type type = Type.int_type;
    Size n = Size.medium;
    Size m = Size.small;


    final switch (type) {
        case Type.int_type:
            final switch (n) {
                case Size.small:
                    final switch (m) {
                        case Size.small:  foo!(int, Size.small, Size.small)(); break;
                        case Size.medium: foo!(int, Size.small, Size.medium)(); break;
                        case Size.large:  foo!(int, Size.small, Size.large)(); break;
                    }
                    break;
                case Size.medium:
                    final switch (m) {
                        case Size.small:  foo!(int, Size.medium, Size.small)(); break;
                        case Size.medium: foo!(int, Size.medium, Size.medium)(); break;
                        case Size.large:  foo!(int, Size.medium, Size.large)(); break;
                    }
                    break;
                case Size.large:
                    final switch (m) {
                        case Size.small:  foo!(int, Size.large, Size.small)(); break;
                        case Size.medium: foo!(int, Size.large, Size.medium)(); break;
                        case Size.large:  foo!(int, Size.large, Size.large)(); break;
                    }
                    break;
            }
            break;

        case Type.float_type:
            final switch (n) {
                case Size.small:
                    final switch (m) {
                        case Size.small:  foo!(float, Size.small, Size.small)(); break;
                        case Size.medium: foo!(float, Size.small, Size.medium)(); break;
                        case Size.large:  foo!(float, Size.small, Size.large)(); break;
                    }
                    break;
                case Size.medium:
                    final switch (m) {
                        case Size.small:  foo!(float, Size.medium, Size.small)(); break;
                        case Size.medium: foo!(float, Size.medium, Size.medium)(); break;
                        case Size.large:  foo!(float, Size.medium, Size.large)(); break;
                    }
                    break;
                case Size.large:
                    final switch (m) {
                        case Size.small:  foo!(float, Size.large, Size.small)(); break;
                        case Size.medium: foo!(float, Size.large, Size.medium)(); break;
                        case Size.large:  foo!(float, Size.large, Size.large)(); break;
                    }
                    break;
            }
            break;

        case Type.short_type:
            final switch (n) {
                case Size.small:
                    final switch (m) {
                        case Size.small:  foo!(short, Size.small, Size.small)(); break;
                        case Size.medium: foo!(short, Size.small, Size.medium)(); break;
                        case Size.large:  foo!(short, Size.small, Size.large)(); break;
                    }
                    break;
                case Size.medium:
                    final switch (m) {
                        case Size.small:  foo!(short, Size.medium, Size.small)(); break;
                        case Size.medium: foo!(short, Size.medium, Size.medium)(); break;
                        case Size.large:  foo!(short, Size.medium, Size.large)(); break;
                    }
                    break;
                case Size.large:
                    final switch (m) {
                        case Size.small:  foo!(short, Size.large, Size.small)(); break;
                        case Size.medium: foo!(short, Size.large, Size.medium)(); break;
                        case Size.large:  foo!(short, Size.large, Size.large)(); break;
                    }
                    break;
            }
            break;
    }
}



A more compressed way:

import std.metastrings: Format;
import std.stdio: writeln;

void foo(T, int N, int M)() {
    writeln(typeid(T), " ", N, " ", M);
}

enum Type { int_type, float_type, short_type }
enum Size { small = 100, medium = 200, large = 250 }

void main() {
    Type type = Type.short_type;
    Size n = Size.medium;
    Size m = Size.small;

    enum string inner = "
                        case Size.small:  foo!(%s, Size.%s, Size.small)(); break;
                        case Size.medium: foo!(%s, Size.%s, Size.medium)(); break;
                        case Size.large:  foo!(%s, Size.%s, Size.large)(); break;
    ";


    enum string medial = "
                    final switch (m) {
                        mixin(Format!(inner, ty, si, ty, si, ty, si));
                    }
                    break;
    ";

    enum string outer = `
            final switch (n) {
                case Size.small:
                    enum string si = "small";
                    mixin(medial);
                case Size.medium:
                    enum string si = "medium";
                    mixin(medial);
                case Size.large:
                    enum string si = "large";
                    mixin(medial);
            }
            break;
    `;

    final switch (type) {
        case Type.int_type:
            enum string ty = "int";
            mixin(outer);

        case Type.float_type:
            enum string ty = "float";
            mixin(outer);

        case Type.short_type:
            enum string ty = "short";
            mixin(outer);
    }
}


It's a mess still :-)

Bye,
bearophile

This is better, probably it can be improves a bit more:

import std.stdio: writeln;
import std.metastrings: Format;
import std.typetuple: TypeTuple;

template Iota(int stop) {
    static if (stop <= 0)
        alias TypeTuple!() Iota;
    else
        alias TypeTuple!(Iota!(stop-1), stop-1) Iota;
}

void foo(T, int N, int M)() {
    writeln(typeid(T), " ", N, " ", M);
}

enum string[] types = ["int", "float", "short"];
enum string[] sizes = ["small", "medium", "large"];

enum Type : string { int_type = types[0], float_type = types[1], short_type = types[2] } enum Size { small = 100, medium = 200, large = 250 }

void main() {
    Type run_t = Type.int_type;
    Size run_n = Size.medium;
    Size run_m = Size.small;

    enum string pattern = "if (run_t == Type.%s_type && run_n == Size.%s && run_m == Size.%s) { foo!(%s, Size.%s, Size.%s)(); goto END; }";

    foreach (t; Iota!(types.length))
        foreach (n; Iota!(sizes.length))
            foreach (m; Iota!(sizes.length))
                mixin(Format!(pattern, types[t], sizes[n], sizes[m], types[t], sizes[n], sizes[m]));

    END: {}
}


Bye,
bearophile

Simpler:

import std.stdio: writeln;
import std.typetuple: TypeTuple;

template Iota(int stop) {
    static if (stop <= 0)
        alias TypeTuple!() Iota;
    else
        alias TypeTuple!(Iota!(stop-1), stop-1) Iota;
}

void foo(T, int N, int M)() {
    writeln(typeid(T), " ", N, " ", M);
}

enum string[] types = ["int", "float", "short"];
alias TypeTuple!(int, float, short) types2;

enum string[] sizes = ["100", "200", "250"];
enum int[] sizes2 = [100, 200, 250];

void main() {
    string run_t = "int";
    string run_n = "200";
    string run_m = "100";

    foreach (t; Iota!(types.length))
        foreach (n; Iota!(sizes.length))
            foreach (m; Iota!(sizes.length))
                if (run_t == types[t] && run_n == sizes[n] && run_m == sizes[m]) {
                    foo!(types2[t], sizes2[n], sizes2[m])();
                    goto END;
                }

    END: {}
}

Last version for now, it can be improved further in various ways:


import std.typetuple: TypeTuple;
import std.metastrings: Format;

template Iota(int stop) {
    static if (stop <= 0)
        alias TypeTuple!() Iota;
    else
        alias TypeTuple!(Iota!(stop-1), stop-1) Iota;
}

enum string[] str_types = ["int", "float", "short"];
mixin(Format!("alias TypeTuple!(%s, %s, %s) types;", str_types[0], str_types[1], str_types[2]));
static assert(str_types.length == types.length); // safety

enum string[] str_sizes = ["100", "200", "250"];
mixin(Format!("enum int[] sizes = [%s, %s, %s];", str_sizes[0], str_sizes[1], str_sizes[2]));
static assert(str_sizes.length == sizes.length); // safety

void staticDispatch3(alias templ)(string run_t, string run_n, string run_m) {
    foreach (t; Iota!(str_types.length))
        foreach (n; Iota!(str_sizes.length))
            foreach (m; Iota!(str_sizes.length))
                if (run_t == str_types[t] && run_n == str_sizes[n] && run_m == str_sizes[m]) {
                    templ!(types[t], sizes[n], sizes[m])();
                    goto END;
                }
    assert(0); // safety

    END: {}
}

// ----------------------------

import std.stdio: writeln;

void foo(T, int N, int M)() {
    writeln(typeid(T), " ", N, " ", M);
}

void main() {
    staticDispatch3!(foo)("int", "200", "100");
}


A more generic staticDispatch3 like this doesn't work because items1 is a tuple, so it gets flattened...

void staticDispatch3(alias func, alias str_items1, alias items1,
                                 alias str_items2, alias items2,
                                 alias str_items3, alias items3)
                    (string run_i1, string run_i2, string run_i3) {
    static assert(str_items1.length == items1.length);
    static assert(str_items2.length == items2.length);
    static assert(str_items3.length == items3.length);

    foreach (i1; Iota!(items1.length))
        foreach (i2; Iota!(items2.length))
            foreach (i3; Iota!(items3.length))
                if (run_i1 == str_types[i1] &&
                    run_i2 == str_types[i2] &&
                    run_i3 == str_types[i3]) {
                    func!(items1[i1], items2[i2], items3[i3])();
                    goto END;
                }
    assert(0); // safety

    END: {}
}

Bye,
bearophile

That's COOL!

I'll have to look at these closer, but I definitely get what you're doing.  Thanks a million.

== Quote from bearophile (bearophileHUGS@lycos.com)'s article
> Simpler:
> import std.stdio: writeln;
> import std.typetuple: TypeTuple;
> template Iota(int stop) {
>     static if (stop <= 0)
>         alias TypeTuple!() Iota;
>     else
>         alias TypeTuple!(Iota!(stop-1), stop-1) Iota;
> }
> void foo(T, int N, int M)() {
>     writeln(typeid(T), " ", N, " ", M);
> }
> enum string[] types = ["int", "float", "short"];
> alias TypeTuple!(int, float, short) types2;
> enum string[] sizes = ["100", "200", "250"];
> enum int[] sizes2 = [100, 200, 250];
> void main() {
>     string run_t = "int";
>     string run_n = "200";
>     string run_m = "100";
>     foreach (t; Iota!(types.length))
>         foreach (n; Iota!(sizes.length))
>             foreach (m; Iota!(sizes.length))
>                 if (run_t == types[t] && run_n == sizes[n] && run_m
== sizes[m]) {
>                     foo!(types2[t], sizes2[n], sizes2[m])();
>                     goto END;
>                 }
>     END: {}
> }

Jason Spencer Wrote:

> If I want to avoid the copy, am I relegated back
> to pointers?

you can make a thin wrapper that will work as a reference-type static array

struct ReferenceArray!(ElementType, int columns, int rows)
{
  ElementType[columns][rows]* back;
  this(byte[] data)
  {
    assert(data.length==ElementType.sizeof*columns*rows);
    back = cast(ElementType[columns][rows]*)data.ptr;
  }
  //will it be inlined?
  ElementType opIndex(int column, int row)
  {
    return (*back)[row][column];
  }
}