Kenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar

It allows user defined collections usable with this syntax:

x[$-2, y[$-6, $-9], $-2]

See:
http://d.puremagic.com/issues/show_bug.cgi?id=3474


But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code:

m[i..j, k..w]

Similar slicing and dicing of arrays is used all the time in NumPy (array library for Python) code.

This is a comment by Don about it:

> (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
> It could be done, but I believe those language features need work. They don't
> permit multi-dimensional slicing. I think they should be removed, and the
> functionality folded into opIndex.

Isn't it better to improve opSlice() instead of deprecating it?


I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code:

m[i..j:2, k..w:3]

---------------

In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code.

In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html

Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library).

The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want.

I think that's a good tradeoff between the opposed needs of:
- Keeping Phobos of reasonable size (to not increase too much the burden of its management, to not slow down too much its development);
- Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows a better focusing of efforts and speeds up the development of a language-wide standard for such arrays;
- Offer a nD array type to the casual D programmer, even one that doesn't want or can't install other libraries. Even some 30-lines long D programs need multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of my functions that take an array of arrays I always have to test the input is not jagged and it is a rectangular matrix. Such test is not necessary for a multi-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with zero other installs.

This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos.

I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules.

Bye,
bearophile

09.10.2011 21:29, bearophile пишет:
> Kenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar
>
> It allows user defined collections usable with this syntax:
>
> x[$-2, y[$-6, $-9], $-2]
>
> See:
> http://d.puremagic.com/issues/show_bug.cgi?id=3474
>
>
> But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code:
>
> m[i..j, k..w]
>
> Similar slicing and dicing of arrays is used all the time in NumPy (array library for Python) code.
>
> This is a comment by Don about it:
>
>> (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
>> It could be done, but I believe those language features need work. They don't
>> permit multi-dimensional slicing. I think they should be removed, and the
>> functionality folded into opIndex.
>
> Isn't it better to improve opSlice() instead of deprecating it?
>
>
> I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code:
>
> m[i..j:2, k..w:3]
>
> ---------------
>
> In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code.
>
> In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html
>
> Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library).
>
> The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want.
>
> I think that's a good tradeoff between the opposed needs of:
> - Keeping Phobos of reasonable size (to not increase too much the burden of its management, to not slow down too much its development);
> - Avoiding the risk of incompatible multi-dimensional array types. Most code out there is able to build on a common foundation. This avoids duplication (like the creation of Python numarray and Numeric), allows a better focusing of efforts and speeds up the development of a language-wide standard for such arrays;
> - Offer a nD array type to the casual D programmer, even one that doesn't want or can't install other libraries. Even some 30-lines long D programs need multi-dimensional arrays, but they often don't need a complex scientific library too (example of a problem: in the preconditions of my functions that take an array of arrays I always have to test the input is not jagged and it is a rectangular matrix. Such test is not necessary for a multi-dimensional array that is never jagged). Putting the bare bones multi-dimensional array type in Phobos allows people to use them with zero other installs.
>
> This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos.
>
> I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules.
>
> Bye,
> bearophile

Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of adding it into Phobos, but just had no time for review of my code. Draft version is in attachment.

rarray.d - something like Fortran array
main.d - example of using rarray module
output.txt - main.d output

I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=6798

I think opSlice!n (n is dimension integer typed size_t) is rarely used, therefore using that name for the enhancement is almost safe for backward compatibility.

Kenji Hara

2011/10/10 Denis Shelomovskij <verylonglogin.reg@gmail.com>:
> 09.10.2011 21:29, bearophile пишет:
>>
>> Kenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar
>>
>> It allows user defined collections usable with this syntax:
>>
>> x[$-2, y[$-6, $-9], $-2]
>>
>> See:
>> http://d.puremagic.com/issues/show_bug.cgi?id=3474
>>
>>
>> But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code:
>>
>> m[i..j, k..w]
>>
>> Similar slicing and dicing of arrays is used all the time in NumPy (array
>> library for Python) code.
>>
>> This is a comment by Don about it:
>>
>>> (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
>>> It could be done, but I believe those language features need work. They
>>> don't
>>> permit multi-dimensional slicing. I think they should be removed, and the
>>> functionality folded into opIndex.
>>
>> Isn't it better to improve opSlice() instead of deprecating it?
>>
>>
>> I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code:
>>
>> m[i..j:2, k..w:3]
>>
>> ---------------
>>
>> In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code.
>>
>> In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html
>>
>> Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library).
>>
>> The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want.
>>
>> I think that's a good tradeoff between the opposed needs of:
>> - Keeping Phobos of reasonable size (to not increase too much the burden
>> of its management, to not slow down too much its development);
>> - Avoiding the risk of incompatible multi-dimensional array types. Most
>> code out there is able to build on a common foundation. This avoids
>> duplication (like the creation of Python numarray and Numeric), allows a
>> better focusing of efforts and speeds up the development of a language-wide
>> standard for such arrays;
>> - Offer a nD array type to the casual D programmer, even one that doesn't
>> want or can't install other libraries. Even some 30-lines long D programs
>> need multi-dimensional arrays, but they often don't need a complex
>> scientific library too (example of a problem: in the preconditions of my
>> functions that take an array of arrays I always have to test the input is
>> not jagged and it is a rectangular matrix. Such test is not necessary for a
>> multi-dimensional array that is never jagged). Putting the bare bones
>> multi-dimensional array type in Phobos allows people to use them with zero
>> other installs.
>>
>> This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos.
>>
>> I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules.
>>
>> Bye,
>> bearophile
>
> Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of adding it into Phobos, but just had no time for review of my code. Draft version is in attachment.
>
> rarray.d - something like Fortran array
> main.d - example of using rarray module
> output.txt - main.d output
>

kenji hara:

> I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=6798

Thank you, you are good.

So is this:
y[$-6, 0..$:2]

Translated like this?
y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))

Bye,
bearophile

2011/10/10 bearophile <bearophileHUGS@lycos.com>:
> So is this:
> y[$-6, 0..$:2]
>
> Translated like this?
> y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))

I have no thought about it.
I'm not sure that the additional stepping is really useful, but I
think adding it into syntax is not impossible -- it might not conflict
with associative array literal.

Kenji Hara

On 10.10.2011 04:41, kenji hara wrote:
> 2011/10/10 bearophile<bearophileHUGS@lycos.com>:
>> So is this:
>> y[$-6, 0..$:2]
>>
>> Translated like this?
>> y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))
>
> I have no thought about it.
> I'm not sure that the additional stepping is really useful, but I
> think adding it into syntax is not impossible -- it might not conflict
> with associative array literal.
>
> Kenji Hara

Personally, I think that since strided operations are so inefficient, they should remain ugly.

On 10.10.2011 03:18, bearophile wrote:
> kenji hara:
> 
>> I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=6798
> 
> Thank you, you are good.
> 
> So is this:
> y[$-6, 0..$:2]
> 
> Translated like this?
> y.opIndex(y.opDollar!0 - 6, y.opSlice!1(0, y.opDollar!1, 2))

Nice! This seems like the way to go.

Posted pull request. https://github.com/D-Programming-Language/dmd/pull/443

This patch does not break existing codes (at least dmd test suite and
phobos building).

I have not added stride syntax (e.g. a[0..$:2]), but I think that
adding it is not so difficult.

Kenji Hara

2011/10/10 kenji hara <k.hara.pg@gmail.com>:
> I got an idea for multidimentional indexing and slicing. http://d.puremagic.com/issues/show_bug.cgi?id=6798
>
> I think opSlice!n (n is dimension integer typed size_t) is rarely used, therefore using that name for the enhancement is almost safe for backward compatibility.
>
> Kenji Hara
>
> 2011/10/10 Denis Shelomovskij <verylonglogin.reg@gmail.com>:
>> 09.10.2011 21:29, bearophile пишет:
>>>
>>> Kenji Hara has just copied in GIT a large patch by Don: https://github.com/9rnsr/dmd/branches/opDollar
>>>
>>> It allows user defined collections usable with this syntax:
>>>
>>> x[$-2, y[$-6, $-9], $-2]
>>>
>>> See:
>>> http://d.puremagic.com/issues/show_bug.cgi?id=3474
>>>
>>>
>>> But a syntax like this too is quite important for a user-defined matrix type to be used in numerical code:
>>>
>>> m[i..j, k..w]
>>>
>>> Similar slicing and dicing of arrays is used all the time in NumPy (array
>>> library for Python) code.
>>>
>>> This is a comment by Don about it:
>>>
>>>> (4) I have NOT implemented $ inside opSlice(), opSliceAssign().
>>>> It could be done, but I believe those language features need work. They
>>>> don't
>>>> permit multi-dimensional slicing. I think they should be removed, and the
>>>> functionality folded into opIndex.
>>>
>>> Isn't it better to improve opSlice() instead of deprecating it?
>>>
>>>
>>> I don't remember if people appreciate the idea of a stride (like the third step argument of std.range.iota), this too is used in scientific array-oriented code:
>>>
>>> m[i..j:2, k..w:3]
>>>
>>> ---------------
>>>
>>> In Python code there is often the need for a good multi-dimensional array type, even in not-scientific code.
>>>
>>> In the Python standard library there is a array module, but it's almost a joke, it's 1D, and it's not used much: http://docs.python.org/library/array.html
>>>
>>> Python programmers use the multi-dimensional array of NumPy, it's widely used around the world, it's used by SciPy too (a scientific programming library).
>>>
>>> The experience of Python, with its sorely felt lack of a good multi-dimensional array type, the failure of its array module, and the success of NumPy, the problems caused by two precedent incompatible array libraries Numeric (http://people.csail.mit.edu/jrennie/python/numeric/ ) and numarray (http://www.stsci.edu/resources/software_hardware/numarray/numarray.html), tells me that it will be good to have a bare-bones, but efficient multi-dimensional array type. Plus external libraries (not present in Phobos) that use those arrays to implement all the things they want.
>>>
>>> I think that's a good tradeoff between the opposed needs of:
>>> - Keeping Phobos of reasonable size (to not increase too much the burden
>>> of its management, to not slow down too much its development);
>>> - Avoiding the risk of incompatible multi-dimensional array types. Most
>>> code out there is able to build on a common foundation. This avoids
>>> duplication (like the creation of Python numarray and Numeric), allows a
>>> better focusing of efforts and speeds up the development of a language-wide
>>> standard for such arrays;
>>> - Offer a nD array type to the casual D programmer, even one that doesn't
>>> want or can't install other libraries. Even some 30-lines long D programs
>>> need multi-dimensional arrays, but they often don't need a complex
>>> scientific library too (example of a problem: in the preconditions of my
>>> functions that take an array of arrays I always have to test the input is
>>> not jagged and it is a rectangular matrix. Such test is not necessary for a
>>> multi-dimensional array that is never jagged). Putting the bare bones
>>> multi-dimensional array type in Phobos allows people to use them with zero
>>> other installs.
>>>
>>> This multi-dimensional Phobos array type doesn't even need to contain code to invert a matrix or compute determinant, etc. It just needs basic operations like allocation, indexing, multi-dimensional slicing, change of shape, iteration... Everything else is in modules/packages external to Phobos.
>>>
>>> I am not suggesting to put a sparse multi-dimensional array type in Phobos. This need is much less common in casual short programs. This is better left to external modules.
>>>
>>> Bye,
>>> bearophile
>>
>> Suddenly, I had a talk with an Fortran guy at previous weekend and have wrote something like Fortran arrays in D last week. I had an idea of adding it into Phobos, but just had no time for review of my code. Draft version is in attachment.
>>
>> rarray.d - something like Fortran array
>> main.d - example of using rarray module
>> output.txt - main.d output
>>
>

Kenji Hara:

> Posted pull request. https://github.com/D-Programming-Language/dmd/pull/443

Things are getting interesting. Thank you.
Such syntax is the building material for a good enough user-defined dense nD array. I presume the memory representation of such arrays (possibly in Phobos) will be tiled, to help CPU caches.


> This patch does not break existing codes (at least dmd test suite and
> phobos building).

Given how important multi-dimensional slicing it, a bit of breakage is acceptable if it allows to make the situation simpler & cleaner. People that will implement a nD array years from now will thank you :-)


> I have not added stride syntax (e.g. a[0..$:2]), but I think that
> adding it is not so difficult.

OK.

Bye,
bearophile

Don:

> Personally, I think that since strided operations are so inefficient, they should remain ugly.

In Chapel (that's a high performance language full of nice ideas) the stride is present, see page 16 "Range operators": http://chapel.cray.com/tutorials/CUG2011/CUG11-2-Basics.pdf

Bye,
bearophile