It is a pleasure to announce the Dlang Statistical Package by John Michael Hall.

API
http://mir-stat.libmir.org/

GitHub
http://github.com/libmir/mir-stat

DUB
https://code.dlang.org/packages/mir-stat

The initial release provides descriptive statistics and algorithms for transforming data that are useful in statistical applications.

The very basic stuff like `gmean` [1] is located in the mir-algorithm package, it will be downloaded automatically.

The generation of random numbers of various distributions is provided by mir-random package [2].

---

libmir.org infrastructure supports cross-site links now.

New documentation macro set:
 GREF
 GREF1
 GREF_ALTTEXT
 GREF1_ALTTEXT

The G* macros add an argument to the first position that should refer to the mir package name.

Example:
/++
Module header
Macros:
 NDSLICEREF = $(GREF_ALTTEXT mir-algorithm, $(TT $2), $2, mir, ndslice, $1)$(NBSP)
+/

/++
See_also: $(NDSLICEREF slice, Slice)
+/
Slice!(double, 2) eye(size_t n);

---

Have a good day!

Ilya

[1] http://mir-algorithm.libmir.org/mir_math_stat.html
[2] http://mir-random.libmir.org/mir_random_variable.html

On Thursday, 8 October 2020 at 16:40:01 UTC, 9il wrote:
> It is a pleasure to announce the Dlang Statistical Package by John Michael Hall.
>
> [...]

Thanks for this great piece of software. Does Mir provides s.th. similar like Pandas DataFrame, especially the feature to give columns a name and marking as inde x columns?

Kind regards
Andre

On Thursday, 8 October 2020 at 17:53:53 UTC, Andre Pany wrote:
> [snip]
>
> Thanks for this great piece of software. Does Mir provides s.th. similar like Pandas DataFrame, especially the feature to give columns a name and marking as inde x columns?
>
> Kind regards
> Andre

Magpie [1] was an initial effort as a summer of code project. The last commit was September 2019.

There is also some basic support in mir (example at [2]). Ilya can speak more about long-term plans for enhancing that.

One limitation in mir is that Slice's only allow for the same type throughout. For instance, a Slice!(double*, 1u) is a 1-dimensional slice of doubles. Data frames in R or Pandas DataFrames allow for columns with different types, so for instance you can calculate some summary statistic based on some category (like color). So to really get the same functionality, you need to support slices with heterogeneous types.

[1] https://github.com/Kriyszig/magpie
[2] https://github.com/libmir/mir-algorithm/blob/f30ccd9f7abc63166c9179e04b2817bf656764bd/source/mir/ndslice/allocation.d#L330

On Thursday, 8 October 2020 at 18:17:30 UTC, jmh530 wrote:
> On Thursday, 8 October 2020 at 17:53:53 UTC, Andre Pany wrote:
>> [snip]
>>
>> Thanks for this great piece of software. Does Mir provides s.th. similar like Pandas DataFrame, especially the feature to give columns a name and marking as inde x columns?
>>
>> Kind regards
>> Andre
>
> Magpie [1] was an initial effort as a summer of code project. The last commit was September 2019.
>
> There is also some basic support in mir (example at [2]). Ilya can speak more about long-term plans for enhancing that.
>
> One limitation in mir is that Slice's only allow for the same type throughout. For instance, a Slice!(double*, 1u) is a 1-dimensional slice of doubles. Data frames in R or Pandas DataFrames allow for columns with different types, so for instance you can calculate some summary statistic based on some category (like color). So to really get the same functionality, you need to support slices with heterogeneous types.
>
> [1] https://github.com/Kriyszig/magpie
> [2] https://github.com/libmir/mir-algorithm/blob/f30ccd9f7abc63166c9179e04b2817bf656764bd/source/mir/ndslice/allocation.d#L330

Thanks for these info. Magpie looks huge and really useful. I will give it a try.

I am also highly interested in the long term plans of Mir, as you explained the current limitations. Still in my scenario it is always the same type. A 2d array of doubles, read from parquet files, transformed and written into a new parquet file.

Kind regards
Andre

On 10/8/20 12:40 PM, 9il wrote:
> It is a pleasure to announce the Dlang Statistical Package by John Michael Hall.
> 
> API
> http://mir-stat.libmir.org/
> 
> GitHub
> http://github.com/libmir/mir-stat
> 
> DUB
> https://code.dlang.org/packages/mir-stat
> 
> The initial release provides descriptive statistics and algorithms for transforming data that are useful in statistical applications.
> 
> The very basic stuff like `gmean` [1] is located in the mir-algorithm package, it will be downloaded automatically.
> 
> The generation of random numbers of various distributions is provided by mir-random package [2].

Outstanding work by all involved. Thank you for driving this (and all of mir) forward. We have already found use in our computational biology lab.

On Thursday, 8 October 2020 at 16:40:01 UTC, 9il wrote:
> It is a pleasure to announce the Dlang Statistical Package by John Michael Hall.
>
> [...]

Awesome! Are there any plans to add functions for inferential stats?

On Sunday, 11 October 2020 at 10:14:04 UTC, tastyminerals wrote:
> On Thursday, 8 October 2020 at 16:40:01 UTC, 9il wrote:
>> It is a pleasure to announce the Dlang Statistical Package by John Michael Hall.
>>
>> [...]
>
> Awesome! Are there any plans to add functions for inferential stats?

Next thing I want to add is histogram (influenced by Boost histogram), but I have been a bit busy lately and haven't finished it. After histogram, the next step would probably be pdfs/cdfs/icdfs, but I was thinking about just borrowing from what is in dstats (I'll need to look into the license compatibility). With those functions in there, then t-test and similar functions would be straightforward.

On Sunday, 11 October 2020 at 17:10:19 UTC, jmh530 wrote:
> On Sunday, 11 October 2020 at 10:14:04 UTC, tastyminerals wrote:
>> On Thursday, 8 October 2020 at 16:40:01 UTC, 9il wrote:
>>> It is a pleasure to announce the Dlang Statistical Package by John Michael Hall.
>>>
>>> [...]
>>
>> Awesome! Are there any plans to add functions for inferential stats?
>
> Next thing I want to add is histogram (influenced by Boost histogram), but I have been a bit busy lately and haven't finished it. After histogram, the next step would probably be pdfs/cdfs/icdfs, but I was thinking about just borrowing from what is in dstats (I'll need to look into the license compatibility). With those functions in there, then t-test and similar functions would be straightforward.

Maybe we should replace Boost with MIT for most of the Mir packages. What do you think?

On Sunday, 11 October 2020 at 17:35:26 UTC, 9il wrote:
> [snip]
>
> Maybe we should replace Boost with MIT for most of the Mir packages. What do you think?

I can't speak to the technical differences between the two. My understanding is that MIT is more permissive than Boost, but MIT always requires the user to include a copy notice and Boost has an exception.

Anyway, it looks like the dstats/distrib.d file [1] is based on MathExtra [2] that is based on Cephes [3]. The dstat file looks like it has a 3-part BSD license, while MathExtra is MIT licensed. Cephes seems to be copyrighted.




[1] https://github.com/DlangScience/dstats/blob/master/source/dstats/distrib.d
[2] http://www.dsource.org/projects/mathextra
[3] https://www.netlib.org/cephes/

On Monday, 12 October 2020 at 00:43:51 UTC, jmh530 wrote:
> On Sunday, 11 October 2020 at 17:35:26 UTC, 9il wrote:
>> [snip]

> I can't speak to the technical differences between the two. My understanding is that MIT is more permissive than Boost, ....

I make all my stuff Boost so that anyone can do whatever they want with the code. So I'm hoe its not that permissive.