On Wednesday, 14 March 2018 at 22:17:49 UTC, jmh530 wrote:
> On Wednesday, 14 March 2018 at 20:21:15 UTC, Sam Potter wrote:
>>
>> Sure. The key word in my statement was "ideally". :-)
>>
>> For what it's worth, there is already an "informal spec" in the form of the high-level interface for numerical linear algebra and sci. comp. that has been developed (over three decades?) in MATLAB. This spec has been replicated (more or less) in Julia, Python, Octave, Armadillo/Eigen, and others. I'm not aware of all the subtleties involved in incorporating it into any standard library, let alone D's, but maybe this is an interesting place where D could get an edge over other competing languages. Considering that people in Python land have picked up D as a "faster Python", there might be more traction here than is readily apparent
>> [snip]
>
> libmir [1] originally started as std.experiemental.ndslice (that component is now mir-algorithm). They had removed it from the std.experimental because it wasn't stable enough yet and needed to make breaking changes. I think it's doing just fine as a standalone library, rather than part of the standard library. As this thread makes clear, there's certainly more work to be done on it, but I'm sure Ilya would appreciate any feedback or assistance.
>
> I'm sympathetic to your point about D getting an edge by having a better linear algebra experience. I came to D for faster Python/R/Matlab (and not C++), though if I need to do something quickly, I still defer to Python/R. However, if you look at the TIOBE index, R and Matlab are at 18/20. Python is quite a bit higher, but it's growth in popularity was not largely due to the Numpy/Scipy ecosystem. So while I think that D could get more traction if libmir turns itself into a premiere linear algebra library, we should be realistic that linear algebra is a relatively small segment of how people use programming languages. Maybe these firms might be willing to pay up for more support though...(if a user could replace pandas with libmir, I would imagine some financial firms might be interested).
>
> [1] https://github.com/libmir

Thanks for the information, and great points made.

I think it's worth mentioning Julia's rapid development. I'm not sure exactly how much traction Julia is getting in different sectors, but I've heard that it has a certain appeal by dint of it being open source (no need to buy expensive MATLAB seats). Julia looks like it could kill MATLAB. I think D has no chance of competing here (beyond being a backend for libraries) without a better scientific experience. OTOH, the fact that D doesn't have a REPL may kill it from the get go (hard to do exploratory data analysis). As you said, you still defer to Python/R/etc---I have a hard time seeing myself using a language like D for everything I work on.

This is getting pretty derailed from the main subject, sorry! I would like to let Ilya and others continue the original discussion now.

On 14 March 2018 at 09:16, Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 03/14/2018 01:01 AM, 9il wrote:
>>
>> On Tuesday, 13 March 2018 at 17:10:03 UTC, jmh530 wrote:
>>>
>>> "Note that using row-major ordering may require more memory and time than column-major ordering, because the routine must transpose the row-major order to the column-major order required by the underlying LAPACK routine."
>>
>>
>> Maybe we should use only column major order. --Ilya
>
>
> Has row-major fallen into disuse?

FWIW, I've never wanted row-major in my career; almost all linear
algebra for geometric use.
Columns correspond to spatial axis, and that's almost always what you
want to manipulate... so column-major it is.
Or with colour transformation, columns correspond to RGB primaries; same story.

On Wednesday, 14 March 2018 at 23:30:55 UTC, Sam Potter wrote:
>
> [snip]
> OTOH, the fact that D doesn't have a REPL may kill it from the get go (hard to do exploratory data analysis).


There is one in dlang-community (there might be others, can't recall), but it does not yet support Windows and there isn't much documentation. I agree it would be nice to have a D Jupyter kernel that was easy to use.

Also, I've been playing with run.dlang.io lately and while it's not the same thing as an REPL, it's one of the easiest ways to play around with D (and a select few libraries, including mir-algorithm).

On 03/14/2018 02:33 AM, 9il wrote:
> On Wednesday, 14 March 2018 at 05:40:42 UTC, Manu wrote:
>> I'd like to understand why implement a distinct vector type, rather
>> than just a Nx1/1xN matrix?
> 
> This is just and API quesiton of how elements of Nx1/1xN matrix should be accessed.
> E.g. should one specify one or two arguments to access an element

Armadillo (arma.sourceforge.net) has separate column and row vector types that allow using a single index.  They can both convert to the matrix type.

On Tuesday, 13 March 2018 at 17:10:03 UTC, jmh530 wrote:
> [1] mir-lapack uses Canonical slices in many of these functions. I assume this is correct, but I have a nagging feeling that I should compare the results of some of these functions with another language to really convince myself...When you increment an iterator on canonical it's still going in row order.

mir-lapack works with Canonical matrixes, but it is assumed that matrix columns are stored Slice rows .  -- Ilya

On Wednesday, 14 March 2018 at 12:45:24 UTC, jmh530 wrote:
> On Wednesday, 14 March 2018 at 05:01:38 UTC, 9il wrote:
>>
>> Maybe we should use only column major order. --Ilya
>
> In my head I had been thinking that the Mat type you want to introduce would be just an alias to a 2-dimensional Slice with a particular SliceKind and iterator. Am I right on that?
>

Nope, I mean completely new type, that will hold Contiguous 2D Slice as internal payload.

On Wednesday, 14 March 2018 at 16:16:55 UTC, Andrei Alexandrescu wrote:
> On 03/14/2018 01:01 AM, 9il wrote:
>> On Tuesday, 13 March 2018 at 17:10:03 UTC, jmh530 wrote:
>>> "Note that using row-major ordering may require more memory and time than column-major ordering, because the routine must transpose the row-major order to the column-major order required by the underlying LAPACK routine."
>> 
>> Maybe we should use only column major order. --Ilya
>
> Has row-major fallen into disuse?
>
> Generally: it would be great to have a standard collection of the typical data formats used in linear algebra and scientific coding. This would allow interoperation without having each library define its own types with identical layout but different names. I'm thinking of:
>
> * multidimensional hyperrectangular

Already done: Slice [1]

> * multidimensional jagged

Done as N-1 Slice composed of jagged rows:

a) Naive:
    Slice!(Contiguous, [1], Slice!(Contiguous, [1], double*)*)
or
b) Intermidiate:
    Slice!(kind, packs, SubSliceIterator!(Iterator, Slicable)) by [2]
or
c) Proper, with compressed indexes by [3]:
    Slice!(Contiguous, [1], SubSliceIteratorInst!I), where

    SubSliceIteratorInst = SubSliceIterator!(SlideIterator!(size_t*, 2, staticArray), I*);

The c) variant is already used by mir.graph to represent Graph indexes.

> * multidimensional hypertriangular if some libraries use it

I saw only 2D packed triangular matrixes in Lapack.
Ndslice provide it using `stairs` topology [4].

The types definitions are

Slice!(Contiguous, [1],
  StairsIterator!(T*))

and

Slice!(Contiguous, [1],
  RetroIterator!(MapIterator!(StairsIterator!(RetroIterator!(T*)), retro)))

The last one can be simplified though.

This types are used in mir-lapack [5], which is a Lapack wrapper with ndslice API created for Lubeck.

> * sparse vector (whatever formats are most common, I assume array of pairs integral/floating point number, with the integral either before or after the floating point number)

Done.

Multidimensional Dictionary of Keys (DOK) format
 is provided by Sparse (an alias for Slice) by [6].

Multidimensional Compressed Sparse Rows (CSR) forma
  is provided by CompressedTensor [7]

There is already sprase BLAS routines for CompressedTensor, [8].

> No need for a heavy interface on top of these. These structures would be low-maintenance and facilitate a common data language for libraries.

As you can see ndslice already provide common data language.

The next goal is to provide a high level common data language with memory automated management and Matlab like API.

BTW, could you please help with the following issue?!

struct S(int b, T)
{
}

alias V(T) = S!(1, T);

auto foo (T)(V!T v)
{
}

void main()
{
    V!double v;
    foo(v);
}

Error: template onlineapp.foo cannot deduce function from argument types !()(S!(1, double)), candidates are:
onlineapp.d(7):        onlineapp.foo(T)(V!T v)

I mean I need this kind of code compiles. Currently I use bad workarounds or define huge types like:

Slice!(Contiguous, [1],
  StairsIterator!(T*))
Slice!(Contiguous, [1],
  RetroIterator!(MapIterator!(StairsIterator!(RetroIterator!(T*)), retro)))

instead of StairsDown!T and StairsUp!T

Of course it is not a problem to add workaround for a standalone project. But for a open source library it is a huge pain (for example, see mir-lapack source with the two types above).

ndslice is very powerful when you need to construct a new type. But there is a problem that a type definition very often is too complex. So, an engineer should choose ether to use a complex generic API or provide a simple non generic. A simple generic API is required for Dlang success in math.

Please let me know you thought if the issue can be fixed.

> Andrei

Best regards,
Ilya

[1] http://docs.algorithm.dlang.io/latest/mir_ndslice_slice.html#.Slice
[2] http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#.mapSubSlices
[3] http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#pairwiseMapSubSlices
[4] http://docs.algorithm.dlang.io/latest/mir_ndslice_topology.html#.stairs
[5] https://github.com/libmir/mir-lapack
[6] http://docs.mir.dlang.io/latest/mir_sparse.html#.Sparse
[7] http://docs.mir.dlang.io/latest/mir_sparse.html#.CompressedTensor
[8] http://docs.mir.dlang.io/latest/mir_sparse_blas_gemm.html

On Thursday, 15 March 2018 at 05:04:42 UTC, 9il wrote:
> [snip]
> BTW, could you please help with the following issue?!
>
> struct S(int b, T)
> {
> }
>
> alias V(T) = S!(1, T);
>
> auto foo (T)(V!T v)
> {
> }
>
> void main()
> {
>     V!double v;
>     foo(v);
> }
>
> Error: template onlineapp.foo cannot deduce function from argument types !()(S!(1, double)), candidates are:
> onlineapp.d(7):        onlineapp.foo(T)(V!T v)
>

It looks like it should expand the alias earlier. No problem with auto foo (T)(S!(1, T) v) {};

On Thursday, 15 March 2018 at 12:49:22 UTC, jmh530 wrote:
> [snip]
>
> It looks like it should expand the alias earlier. No problem with auto foo (T)(S!(1, T) v) {};

Also, this issue also shows up in mir.ndslice.traits. I had to do the equivalent of isV below. It doesn't work to do the alternate version. However, given that you have the traits, then you can use them in a template constraint. So you have to repeat yourself in the trait once, rather than bunches of times in each function that calls them.

enum bool isV(T) = is(T : S!(1, U), U);
enum bool isV_alternate(T) = is(T : V!(U), U);

On Thursday, 15 March 2018 at 05:04:42 UTC, 9il wrote:
> [snip]
>
> BTW, could you please help with the following issue?!
>
> struct S(int b, T)
> {
> }
>
> alias V(T) = S!(1, T);
>
> auto foo (T)(V!T v)
> {
> }
>
> void main()
> {
>     V!double v;
>     foo(v);
> }
>
> Error: template onlineapp.foo cannot deduce function from argument types !()(S!(1, double)), candidates are:
> onlineapp.d(7):        onlineapp.foo(T)(V!T v)
>

This is issue 16486 [1], which is very similar to 16465 [1] and seems like should be closed.

[1] https://issues.dlang.org/show_bug.cgi?id=16486
[2] https://issues.dlang.org/show_bug.cgi?id=16465