I accidentally discovered Andrei wrote [1] multidimensional array implementation is needed. If it really is, I will work to revise the API and prepare my implementation [2] for review if nobody is doing it already.

Also as Kenji's "multidimensional indexing and slicing" pull [3] still not merged the only way is to use hacks like this:
---
// first two rows and three columns of the second matrix
array2d = matrices[1, R[0 .. 2], R[0 .. 3]];
---

[1] http://forum.dlang.org/post/kivkp0$csp$1@digitalmars.com
[2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimensionalarray.html
[3] https://github.com/D-Programming-Language/dmd/pull/443


--- Previous related topics ---

At least the ones I participated in:

* October 09, 2011: Kenji Hara proposes "Matrix-type-friendly syntax and more". His dmd pull #443 still isn't merged.
    http://forum.dlang.org/thread/j6sp68$2a7k$1@digitalmars.com
    https://github.com/D-Programming-Language/dmd/pull/443

* October 25, 2011: Original "Multidimensional arrays for D" post. No response from Phobos developers.
    http://forum.dlang.org/thread/j864es$2gi0$1@digitalmars.com

* June 17, 2012: A request for "template that can simulate a rectangular array".
    http://forum.dlang.org/thread

* June 30, 2012: A request for "fixed size multidimensional array at runtime".
    http://forum.dlang.org/thread/ldjzfqvnjltbbiovqdmy@forum.dlang.org

-- 
Денис В. Шеломовский
Denis V. Shelomovskij

On Tuesday, 8 October 2013 at 14:41:47 UTC, Denis Shelomovskij wrote:
> I accidentally discovered Andrei wrote [1] multidimensional array implementation is needed. If it really is, I will work to revise the API and prepare my implementation [2] for review if nobody is doing it already.
>
> Also as Kenji's "multidimensional indexing and slicing" pull [3] still not merged the only way is to use hacks like this:
> ---
> // first two rows and three columns of the second matrix
> array2d = matrices[1, R[0 .. 2], R[0 .. 3]];
> ---
>
> [1] http://forum.dlang.org/post/kivkp0$csp$1@digitalmars.com
> [2] http://denis-sh.bitbucket.org/unstandard/unstd.multidimensionalarray.html
> [3] https://github.com/D-Programming-Language/dmd/pull/443
>
>
> --- Previous related topics ---
>
> At least the ones I participated in:
>
> * October 09, 2011: Kenji Hara proposes "Matrix-type-friendly syntax and more". His dmd pull #443 still isn't merged.
>     http://forum.dlang.org/thread/j6sp68$2a7k$1@digitalmars.com
>     https://github.com/D-Programming-Language/dmd/pull/443
>
> * October 25, 2011: Original "Multidimensional arrays for D" post. No response from Phobos developers.
>     http://forum.dlang.org/thread/j864es$2gi0$1@digitalmars.com
>
> * June 17, 2012: A request for "template that can simulate a rectangular array".
>     http://forum.dlang.org/thread
>
> * June 30, 2012: A request for "fixed size multidimensional array at runtime".
>     http://forum.dlang.org/thread/ldjzfqvnjltbbiovqdmy@forum.dlang.org

I don't normally post here a lot (though I'm a regular reader), but I wanted to say I for one would really appreciate an official solution for proper rectangular arrays. A bit of background: I'm a numerical physicist focusing on the lattice Boltzmann method[1], where most physical quantities live on a (3D) lattice. Currently I'm using a Fortran code with is very feature-rich, but has grown organically over a decade or so and the features have come at the cost of maintainability and performance. As I'm very much interested in the D language (though I cannot devote much time to it at the moment) I've had plans of writing my own smaller D code which would contain the features I need. It would be nice to be able to use Phobos for my 3D array needs. Slicing will also be much valued to make it easier to communicate sections of the lattice through MPI. I would aim to undertake this project after I've finished my PhD thesis, in ~2 months. I don't assume an official Phobos version would be available at that time, but even having a good idea of the API that is being aimed for would save me a lot of time I think.

Cheers,

Stefan

[1] http://en.wikipedia.org/wiki/Lattice_Boltzmann_methods

On Tuesday, 8 October 2013 at 17:26:46 UTC, Stefan Frijters wrote:
andrei wrote:

* We need to have a battery of multidimensional array shapes along with
simple iteration and access primitives, at least for interfacing with
scientific libraries that define and expect such formats. I'm thinking
rectangular (generally hyperrectangular) matrices, triangular matrices,
sparse matrices, and band matrices.

I too are interesteed in this area as well. Dennis do you only plan to focus on multidimensional arrays only, or will you incorporate the above matrices as well  ??

What features are you proposing ?

Nick

09.10.2013 7:55, Nick B пишет:
> On Tuesday, 8 October 2013 at 17:26:46 UTC, Stefan Frijters wrote:
> andrei wrote:
>
> I too are interesteed in this area as well. Dennis do you only plan to
> focus on multidimensional arrays only, or will you incorporate the above
> matrices as well  ??
>
> What features are you proposing ?

I propose stuff for "multidimensional arrays only" as you noted. And I plan just to revise my existing API [1] without cardinal changes.

I.e. all I propose is rectangular multidimensional arrays slicing and iterating. For matrix and math specific tasks see DScience [2] and SciD [3]. The latter started as a fork of DScience but became a separate project and is in development. See its wiki [4]. Also such math oriented libraries have to be partially (and the are) wrapper around LAPACK.

Also it will be interest to see features you (Stefan and Nick) need e.g. as examples of code you want to compile with comments if needed. Write down at least basic features for now.

[1] http://denis-sh.bitbucket.org/unstandard/unstd.multidimensionalarray.html
[2] https://github.com/dscience-developers/dscience
[3] https://github.com/kyllingstad/scid
[4] https://github.com/kyllingstad/scid/wiki

-- 
Денис В. Шеломовский
Denis V. Shelomovskij

On Wednesday, 9 October 2013 at 08:30:11 UTC, Denis Shelomovskij wrote:
> 09.10.2013 7:55, Nick B пишет:
>> On Tuesday, 8 October 2013 at 17:26:46 UTC, Stefan Frijters wrote:
>> andrei wrote:
>>
>> I too are interesteed in this area as well. Dennis do you only plan to
>> focus on multidimensional arrays only, or will you incorporate the above
>> matrices as well  ??
>>
>> What features are you proposing ?
>
> I propose stuff for "multidimensional arrays only" as you noted. And I plan just to revise my existing API [1] without cardinal changes.
>
> I.e. all I propose is rectangular multidimensional arrays slicing and iterating. For matrix and math specific tasks see DScience [2] and SciD [3]. The latter started as a fork of DScience but became a separate project and is in development. See its wiki [4]. Also such math oriented libraries have to be partially (and the are) wrapper around LAPACK.
>
> Also it will be interest to see features you (Stefan and Nick) need e.g. as examples of code you want to compile with comments if needed. Write down at least basic features for now.
>
> [1] http://denis-sh.bitbucket.org/unstandard/unstd.multidimensionalarray.html
> [2] https://github.com/dscience-developers/dscience
> [3] https://github.com/kyllingstad/scid
> [4] https://github.com/kyllingstad/scid/wiki

Ok, off the top of my head, here are some of the points that would be great for me to have. I apologize in advance if any of them are trivial / irrelevant or out of scope; I have not had time to get my hands dirty on this subject. Even if they are not to be part of the generic multidimensional array (MDA) module, these are things that I would then like to build my own implementation of without having to work with instead of against the things that will be in Phobos.

- Many of my operations involve looping over the array in no particular order, so the first foreach example in your link #1 will be very useful.
- Another very common operation is accessing a lattice site and looking at its neighbours to determine the outcome of the operation. Of course this is easy for nested for-loops as I can just nest one deeper and pre-calculate the neighbour offsets in another array, but I don't know if there's a canonical way to do this in terms of a foreach loop, and if this would add requirements to the MDA. As an example, Python's numpy seems to have 'generic_filter' for tasks like this[1]. In my testing it was very slow though.
- I will have multiple MDAs, containing information like local densities and velocities. These will affect each other in calculations and thus being able to use zip and friends would be very useful. This would require the MDA to be a range I guess?
- My code will use wrapped MPI[2] and HDF5[3] calls for parallelism and parallel IO, respectively, and because of that I will need some control over the memory layout. Nothing fancy, but the usual C-style pointer arithmetic would need to work I think, unless there's a nicer mechanism.

I hope these comments can be of some help.

Cheers,

Stefan

[1] http://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.filters.generic_filter.html#scipy.ndimage.filters.generic_filter
[2] http://en.wikipedia.org/wiki/Message_Passing_Interface
[3] http://www.hdfgroup.org/HDF5/

On Tue, Oct 08, 2013 at 06:42:12PM +0400, Denis Shelomovskij wrote:
> I accidentally discovered Andrei wrote [1] multidimensional array implementation is needed. If it really is, I will work to revise the API and prepare my implementation [2] for review if nobody is doing it already.
> 
> Also as Kenji's "multidimensional indexing and slicing" pull [3] still not merged the only way is to use hacks like this:
> ---
> // first two rows and three columns of the second matrix array2d = matrices[1, R[0 .. 2], R[0 .. 3]];
> ---
[...]

What's the reason Kenji's pull isn't merged yet? As I see it, it does not introduce any problematic areas, but streamlines multidimensional indexing notation in a nice way that fits in well with the rest of the language. I, for one, would push for it to be merged.

In any case, I've seen your multidimensional array implementation before, and I think it would be a good thing to have it in Phobos. In fact, I've written my own as well, and IIRC one or two other people have done the same. Clearly, the demand is there.

See also the thread about std.linalg; I think before we can even talk about having linear algebra code in Phobos, we need a solidly-designed rectangular array API. As I said in that other thread, matrix algebra really should be built on top of a solid rectangular array API, and not be yet another separate kind of type that's similar to, but incompatible with rectangular arrays. A wrapper type can be used to make a rectangular array behave in the linear algebra sense (i.e. matrix product instead of per-element multiplication).


T

-- 
Debian GNU/Linux: Cray on your desktop.

On Tuesday, 8 October 2013 at 14:41:47 UTC, Denis Shelomovskij wrote:
> I accidentally discovered Andrei wrote [1] multidimensional array implementation is needed. If it really is, I will work to revise the API and prepare my implementation [2] for review if nobody is doing it already.
>
> Also as Kenji's "multidimensional indexing and slicing" pull [3] still not merged the only way is to use hacks like this:

+1

On Friday, 11 October 2013 at 22:41:06 UTC, H. S. Teoh wrote:
> What's the reason Kenji's pull isn't merged yet? As I see it, it does
> not introduce any problematic areas, but streamlines multidimensional
> indexing notation in a nice way that fits in well with the rest of the
> language. I, for one, would push for it to be merged.
>
> In any case, I've seen your multidimensional array implementation
> before, and I think it would be a good thing to have it in Phobos. In
> fact, I've written my own as well, and IIRC one or two other people have
> done the same. Clearly, the demand is there.
>
> See also the thread about std.linalg; I think before we can even talk
> about having linear algebra code in Phobos, we need a solidly-designed
> rectangular array API. As I said in that other thread, matrix algebra
> really should be built on top of a solid rectangular array API, and not
> be yet another separate kind of type that's similar to, but incompatible
> with rectangular arrays. A wrapper type can be used to make a
> rectangular array behave in the linear algebra sense (i.e. matrix
> product instead of per-element multiplication).


Hi.

I wondered how things were developing with the rectangular arrays (not sure who is in charge of reviewing, but I guess it is not HS Teoh).  It would be interesting to see this being available for D, and I agree with others that it is one of the key foundation blocks one would need to see in place before many other useful libraries can be built on top.

Let me know if anything I can help with (although cannot promise to have time, I will try).


Laeeth.

A gap in multi-dimensional rectangular arrays functionality in D is sure a huge blocker when trying to use it for data science tasks. Wonder what's the general consensus on this?

On Mon, Dec 22, 2014 at 11:35:17AM +0000, aldanor via Digitalmars-d wrote:
> A gap in multi-dimensional rectangular arrays functionality in D is sure a huge blocker when trying to use it for data science tasks. Wonder what's the general consensus on this?

Kenji's PR has been merged in the meantime, so now we have the tools to build a solid multi-dim array library. Somebody just needs to do the work, that's all.


T

-- 
Debian GNU/Linux: Cray on your desktop.