On Monday, 28 December 2015 at 14:42:43 UTC, Ilya Yaroshenko wrote:
> On Monday, 28 December 2015 at 11:13:57 UTC, Martin Nowak wrote:
>> We're branching for 2.070 soon, would be nice if this can make it, but only if it's really ready.
>
> Whooohooo!
>
> Thanks!
>
> Ilya

I'm writing a blog post giving an overview of std.ndslice and comparing and contrasting to Numpy, to be released when this is merged, and I was wondering if I can have your permission to include your great image processing example in the post?

On Tuesday, 29 December 2015 at 16:28:52 UTC, Robert burner Schadek wrote:
> On Tuesday, 29 December 2015 at 16:11:00 UTC, Ilya Yaroshenko wrote:
>>
>> OK, lets discuss every function.
>>
>
> That is acceptably the problem. It is not about the documentation of the functions it is about the documentation binding the functions together and the documentation giving the idea of the library.

OK, looks like I understand you.

My style of exploration is set of examples. For some people it is much better. But I agree that it may be bad for others.

About `iteration` module:
====================
1. from documentation:
------
Operators only change strides and lengths of a slice. The range of a slice remains unmodified. All operators return slice of the same type as the type of the argument.
------
2. They are very simple functions, and their combination is simple too. A user will understand how to use them. Furthermore `Binary representation` section in `Slice` docs has examples with `transposed` and `reversed`. So user can understand what exactly going on with iteration functions.


About selection module:
====================
1. from documentation:
------
Selectors create new views and iteration patterns over the same data, without copying.

Subspace selectors
- - -
Subspace selectors serve to generalize and combine other selectors easily. For a slice of Slice!(N, Range) type slice.pack!K creates a slice of slices of Slice!(N-K, Slice!(K+1, Range)) type by packing the last K dimensions of the top dimension pack, and the type of element of slice.byElement is Slice!(K, Range). Another way to use pack is transposition of dimension packs using evertPack. Examples of use of subspace selectors are available for selectors, Slice.shape , and Slice.elementsCount .
------

2. Selectors look simple, but their combinations can be _very_ crazy. Combination of this functions is art of imagination. I am not so talented writer to create good documentation. The problem here is that I have never discuss this module with significant amount of people. And I don't think that I will have such opportunity. So what I have done with documentation for selectors? I created cross examples:

`unpack` has examples with
  - `pack`
`evertPack` has examples with
  - `pack`
  - `unpack`
`diagonal` has examples with
  - `dropOne`
  - `dropToHypercube`
  - `reversed`
  - `pack`
  - `evertPack`
  - Note about additional examples with `blocks` and `windows`
`blocks` has examples with
  - `byElement`
  - `diagonal`
  - `pack`
  - `evertPack`
  - `unpack`
`windows` has examples with
  - `byElement`
  - `diagonal`
  - `pack`
  - `evertPack`
  - `unpack`
`byElement` has examples with
  - `pack`
  - `drop`
  - `std.range.retro`
  - `allReversed`
`byElementInStandardSimplex` has examples with
  - `allReversed `
  - `transposed`
`indexSlice` has examples with
  - `byElement`

In addition, example `Image Processing` in `package` description is a real world combination of `sliced`, `pack`, `windows`, `unpack`, `byElement`, `std.algorithm.iteration.map`, `std.array.array`,  `Slice.shape`.

This is a way how I can write documentation. I don't think that I can write using style you want or find someone who can do it instead of me. I am already spend a lot of time with English translator to improve docs. And every new sentence is a real pain, because I need to ask my translator to check it.

Ilya

On Tuesday, 29 December 2015 at 17:17:05 UTC, Jack Stouffer wrote:
> On Monday, 28 December 2015 at 14:42:43 UTC, Ilya Yaroshenko wrote:
>> On Monday, 28 December 2015 at 11:13:57 UTC, Martin Nowak wrote:
>>> We're branching for 2.070 soon, would be nice if this can make it, but only if it's really ready.
>>
>> Whooohooo!
>>
>> Thanks!
>>
>> Ilya
>
> I'm writing a blog post giving an overview of std.ndslice and comparing and contrasting to Numpy, to be released when this is merged, and I was wondering if I can have your permission to include your great image processing example in the post?

Thank you for doing this!

Yes, you can get any examples! `movingWindowByChannel` is a good function, however small function `median` can be improved, but it is not easy to do it for small filter sizes. You may want to use convolution matrices such as the Sobel operator or Gaussian blur instead of `median`. They will work faster comparing with `median`.  Another option is a note that `median` can be significantly optimised.

^____^ Many thanks!

Ilya

On 12/29/2015 11:28 AM, Robert burner Schadek wrote:
> On Tuesday, 29 December 2015 at 16:11:00 UTC, Ilya Yaroshenko wrote:
>>
>> OK, lets discuss every function.
>>
>
> That is acceptably the problem. It is not about the documentation of the
> functions it is about the documentation binding the functions together
> and the documentation giving the idea of the library.

Hopefully this is something that you or someone else could help by creating pull requests. Any volunteers? -- Andrei

On Tuesday, 29 December 2015 at 18:08:52 UTC, Andrei Alexandrescu wrote:
> On 12/29/2015 11:28 AM, Robert burner Schadek wrote:
>> On Tuesday, 29 December 2015 at 16:11:00 UTC, Ilya Yaroshenko wrote:
>>>
>>> OK, lets discuss every function.
>>>
>>
>> That is acceptably the problem. It is not about the documentation of the
>> functions it is about the documentation binding the functions together
>> and the documentation giving the idea of the library.
>
> Hopefully this is something that you or someone else could help by creating pull requests. Any volunteers? -- Andrei

Collection of posts like Jack Stouffer is writing will bring an understanding of how to improve docs. Currently we can collect links of such posts for a while and store them on package description page, thought.

YouTube has 7.5K videos for "numpy" query.

Ilya

On Tuesday, 29 December 2015 at 17:38:06 UTC, Ilya Yaroshenko wrote:
> On Tuesday, 29 December 2015 at 17:17:05 UTC, Jack Stouffer wrote:
>> On Monday, 28 December 2015 at 14:42:43 UTC, Ilya Yaroshenko wrote:
>>> On Monday, 28 December 2015 at 11:13:57 UTC, Martin Nowak wrote:
>>>> We're branching for 2.070 soon, would be nice if this can make it, but only if it's really ready.
>>>
>>> Whooohooo!
>>>
>>> Thanks!
>>>
>>> Ilya
>>
>> I'm writing a blog post giving an overview of std.ndslice and comparing and contrasting to Numpy, to be released when this is merged, and I was wondering if I can have your permission to include your great image processing example in the post?
>
> Thank you for doing this!

First draft: http://jackstouffer.com/hidden/nd_slice.html

Please critique.

On Tuesday, 29 December 2015 at 21:19:19 UTC, Jack Stouffer wrote:
> On Tuesday, 29 December 2015 at 17:38:06 UTC, Ilya Yaroshenko wrote:
>> On Tuesday, 29 December 2015 at 17:17:05 UTC, Jack Stouffer wrote:
>...
> First draft: http://jackstouffer.com/hidden/nd_slice.html
>
> Please critique.

Looks solid. Good work!

On Tuesday, 29 December 2015 at 21:19:19 UTC, Jack Stouffer wrote:
> On Tuesday, 29 December 2015 at 17:38:06 UTC, Ilya Yaroshenko wrote:
>> On Tuesday, 29 December 2015 at 17:17:05 UTC, Jack Stouffer wrote:
>>> On Monday, 28 December 2015 at 14:42:43 UTC, Ilya Yaroshenko wrote:
>>>> On Monday, 28 December 2015 at 11:13:57 UTC, Martin Nowak wrote:
>>>>> We're branching for 2.070 soon, would be nice if this can make it, but only if it's really ready.
>>>>
>>>> Whooohooo!
>>>>
>>>> Thanks!
>>>>
>>>> Ilya
>>>
>>> I'm writing a blog post giving an overview of std.ndslice and comparing and contrasting to Numpy, to be released when this is merged, and I was wondering if I can have your permission to include your great image processing example in the post?
>>
>> Thank you for doing this!
>
> First draft: http://jackstouffer.com/hidden/nd_slice.html
>
> Please critique.

Awesome!

Please find my notes below.

The Basics
======

About `iota.front -> slice.font -> user accessing the data`:

1.
From the `sliced` documentation for `range` parameter: "only index operator auto `opIndex(size_t index)` is required for ranges".

See also recently added the last two examples for sliced http://dtest.thecybershadow.net/artifact/website-76234ca0eab431527327d5ce1ec0ad74c6421533-387174b023f8cb9612cfcddc76788896/web/phobos-prerelease/std_experimental_ndslice_slice.html#.sliced

The first one is for Input Range primitives, the second one is for Random Access Range primitives.

2.
Consequently `Slice` never invokes `front`, `popFront`, `empty`, and others range primitives. `Slice` invokes only `opIndex(size_t)`, and if `save` is defined, `slice.save` invokes `range.save`.

In addition, `sliced` can be used like Finite Random Access Range Constructor: user may define only `opIndex(size_t)` and `save` to make a Finite Random Access Range by calling `sliced`.

3.
If type of slice is `Slice!(3, Range)`, then `slice.front` do not invokes `range[indexToFront]`, it returns a slice type of `Slice!(2, Range)`.

In addition, `indexToFront` may not be zero, because primitives like `popFront` or operators like `reversed` may be called before.

For slices over arrays `indexToFront` is not defined because the pointer always refers to the first position: the pointer is moved when the slice changes.

For more details see also examples for Internal Binary Representation http://dtest.thecybershadow.net/artifact/website-76234ca0eab431527327d5ce1ec0ad74c6421533-387174b023f8cb9612cfcddc76788896/web/phobos-prerelease/std_experimental_ndslice_slice.html#.Slice

Getting Hands On
======
 - Please use "~>0.8.7" instead of "~>0.8.0". "~>0.8.7" has LDC 0.17.0 alpha1 support.

Examples
======
 - A Basic Example With A Benchmark is not honest (difference 76 is very large). To make it honest:
     1. Larger 100x1000 matrix can be used instead of 10x10, because numpy have a significant initialization overhead.
     2. Both `100_000.iota.sliced(100, 1000)` and `100_000.iota.array.sliced(100, 1000)` can be tested. The last one is with `std.array.array`. Memory access is expensive.
     3. Only iteration should be tested. Allocation and initilization should be separated from iteration both in D and Python.

   I expect smaller differnce in perfomance than 76 times.

 - You may want to test both DMD and LDC 0.17.0 alpha1 for bechmarks https://github.com/ldc-developers/ldc/releases/tag/v0.17.0-alpha1
   LDC 0.17.0 alpha1 works well with dip80-ndslice v0.8.7.

 - Python users love small code and they can be afraid to see large (template constraints)
   `mean` funciton.
   An example with lambda function can be added:
   ---
   auto means = 100_000.iota.sliced(100, 10000).rotated(3).map!(r => sum(r) / r.length);
   ---
   One line!

 - Nitpick: `transposed` (without params for 2D slice) is more clear than `rotated(3)`, IMO

 - `dub --build=release`can be added where you have noted dmd optimisation flags.

Numpy's Main Problem, and How D Avoids It
=====
 - D version with allocation using `std.array.array` can be added (or just `new int[1000]`). The slice over `repeat` is not mutable (users from numpy world may expect that it is an array).

Ilya

On Wednesday, 30 December 2015 at 00:24:38 UTC, Ilya Yaroshenko wrote:
> On Tuesday, 29 December 2015 at 21:19:19 UTC, Jack Stouffer wrote:
>> On Tuesday, 29 December 2015 at 17:38:06 UTC, Ilya Yaroshenko wrote:
>>> On Tuesday, 29 December 2015 at 17:17:05 UTC, Jack Stouffer wrote:
>>>> [...]
>>>
>>> Thank you for doing this!
>>
>> First draft: http://jackstouffer.com/hidden/nd_slice.html
>>
>> Please critique.
>
> Awesome!
>
> Please find my notes below.
>[...]
>  - You may want to test both DMD and LDC 0.17.0 alpha1 for bechmarks https://github.com/ldc-developers/ldc/releases/tag/v0.17.0-alpha1
>    LDC 0.17.0 alpha1 works well with dip80-ndslice v0.8.7.
> [...]
> Ilya

ldmd2/ldc2 flag -mcpu=native will optimise code for your CPU. -- Ilya

On Wednesday, 30 December 2015 at 01:03:39 UTC, Ilya Yaroshenko wrote:
> On Wednesday, 30 December 2015 at 00:24:38 UTC, Ilya Yaroshenko wrote:
>> On Tuesday, 29 December 2015 at 21:19:19 UTC, Jack Stouffer wrote:
>>> On Tuesday, 29 December 2015 at 17:38:06 UTC, Ilya Yaroshenko wrote:
>>>> On Tuesday, 29 December 2015 at 17:17:05 UTC, Jack Stouffer wrote:
>>>>> [...]
>>>>
>>>> Thank you for doing this!
>>>
>>> First draft: http://jackstouffer.com/hidden/nd_slice.html
>>>
>>> Please critique.
>>
>> Awesome!
>>
>> Please find my notes below.
>>[...]
>>  - You may want to test both DMD and LDC 0.17.0 alpha1 for bechmarks https://github.com/ldc-developers/ldc/releases/tag/v0.17.0-alpha1
>>    LDC 0.17.0 alpha1 works well with dip80-ndslice v0.8.7.
>> [...]
>> Ilya
>
> ldmd2/ldc2 flag -mcpu=native will optimise code for your CPU. -- Ilya

The paragraph

"This function does not calculate border cases in which a window overlaps the image partially. However, the function can still be used to carry out such calculations. That can be done by creating an amplified image, with the edges reflected from the original image, and then applying the given function to the new file. Note:"

ends with "Note:". --Ilya