On Tuesday, 4 August 2015 at 18:56:20 UTC, jmh530 wrote:
> On Tuesday, 4 August 2015 at 09:48:07 UTC, Chris wrote:
>> I agree with bachmeier. You cannot go wrong. You mentioned nested loops. D allows you to concatenate (or "pipe") loops. So instead of
>>
>> foreach
>> {
>>   foreach
>>   {
>>     foreach
>>     {
>>     }
>>   }
>> }
>>
>> you have something like
>>
>> int[] numbers = [-2, 1, 6, -3, 10];
>> foreach (ref n; numbers
>>   .map!(a => a * 5)  // multiply each value by 5
>>   .filter!(a => a > 0))  // filter values that are 0 or less
>> {
>>   //  Do something
>> }
>>
>
> I don't think I had seen an example like this before (though it is obvious in retrospect). Is there any advantage in terms of performance?

ldc and gdc can often achieve parity with explicit loops.

Dear all,

Thank you for your replies. I am now really convinced that D is a decent choice for my project (also I am really happy to see that the forum is really active and apparently many of you use D for your scientific projects). I am just looking forward to writing the code. I had a very quick look at lecture given at DConf 2015 - good talk, and I believe D has a big promise in Science. Perhaps the only problem being is the mathematical library, like numpy.

Until now I usually wrote the prototype algorithms in Python and then translated the code onto C for speed. It would be just dream to use only one language. The dominant languages in science now for production codes are Fortran or C/C++, may be D could become another option?

With kind regards,
Yury

On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:
> Dear all,
>
> Thank you for your replies. I am now really convinced that D is a decent choice for my project (also I am really happy to see that the forum is really active and apparently many of you use D for your scientific projects). I am just looking forward to writing the code. I had a very quick look at lecture given at DConf 2015 - good talk, and I believe D has a big promise in Science. Perhaps the only problem being is the mathematical library, like numpy.
>
> Until now I usually wrote the prototype algorithms in Python and then translated the code onto C for speed. It would be just dream to use only one language. The dominant languages in science now for production codes are Fortran or C/C++, may be D could become another option?
>
> With kind regards,
> Yury

I think NumPy was written in C(++) and is imported as a Python module. So if you can get your hands on the original underlying C(++) library, you can call NumPy directly from D, can't you? In case you do this, let me know how you fared with it. NumPy is usually the deadbeat argument when people have to choose between Python or other languages.

On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:

> The dominant languages in science now for production codes are Fortran or C/C++, may be D could become another option?
>
> With kind regards,
> Yury

Yes. The question is whether we can put together a group of developers to build the infrastructure, which is a lot more than just code. That means, in particular, good documentation and using it for our own projects.

Everyone these days talks about how Python is a powerhouse scientific programming language. A decade ago it was crap. I know, because I watched it for years wishing I could use it. There were some poorly documented, domain-specific, hacked-together libraries, but Python was not for the most part a suitable choice.

There is no reason we can't do the same for D. The main question is whether we are sufficiently committed to that goal. Others may consider Python, Julia, and Matlab to be good enough alternatives (I don't, but not everyone necessarily agrees with me).

On Wednesday, 5 August 2015 at 18:20:20 UTC, Chris wrote:
> On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:
>> Dear all,
>>
>> Thank you for your replies. I am now really convinced that D is a decent choice for my project (also I am really happy to see that the forum is really active and apparently many of you use D for your scientific projects). I am just looking forward to writing the code. I had a very quick look at lecture given at DConf 2015 - good talk, and I believe D has a big promise in Science. Perhaps the only problem being is the mathematical library, like numpy.
>>
>> Until now I usually wrote the prototype algorithms in Python and then translated the code onto C for speed. It would be just dream to use only one language. The dominant languages in science now for production codes are Fortran or C/C++, may be D could become another option?
>>
>> With kind regards,
>> Yury
>
> I think NumPy was written in C(++) and is imported as a Python module. So if you can get your hands on the original underlying C(++) library, you can call NumPy directly from D, can't you? In case you do this, let me know how you fared with it. NumPy is usually the deadbeat argument when people have to choose between Python or other languages.

Isn't the useful bit of NumPy more like a set of wrappers around other C/C++/Fortran libraries?  And the whole point of NumPy is that you can call it easily from python, which doesn't make for a nice calling convention from D (although you can call from PyD if you want).

Anyway, I agree that it's a big project, but not an infeasible one to implement similar functionality in D.

On Wednesday, 5 August 2015 at 18:49:21 UTC, bachmeier wrote:
> On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:
>
>> The dominant languages in science now for production codes are Fortran or C/C++, may be D could become another option?
>>
>> With kind regards,
>> Yury
>
> Yes. The question is whether we can put together a group of developers to build the infrastructure, which is a lot more than just code. That means, in particular, good documentation and using it for our own projects.
>
> Everyone these days talks about how Python is a powerhouse scientific programming language. A decade ago it was crap. I know, because I watched it for years wishing I could use it. There were some poorly documented, domain-specific, hacked-together libraries, but Python was not for the most part a suitable choice.
>
> There is no reason we can't do the same for D. The main question is whether we are sufficiently committed to that goal. Others may consider Python, Julia, and Matlab to be good enough alternatives (I don't, but not everyone necessarily agrees with me).

Yes - I fully agree with everything you say here.

John Colvin has done great work in putting together a scientific computing portal for D, and writing a wrapper so you can write D within an ipython notebook (and call it transparently from python code).  Great for rapid iteration and exploration of data, and it means you don't need to write the whole stack from scratch in D.

One place I started in a small way was implementing a limited subset of dataframe functionality.  There's not much to it, but it's something very handy to be able to slurp in data from csv or hdf5 that might not fit in a more rigid format and do spreadsheet type stuff with it.

So instead of an array of variants, you define a type for the column (every entry in the column is the same type, but different columns may be of different types).  In addition each column has a name and you can add and remove columns easily.  I've implemented a just about usable version of that, but it's not pretty, rigorous, or especially efficient.  The next stage is creating indexing by columns a la pandas.

Dataframes aren't intellectually very exciting, but they are very useful for iterative data exploration and quick prototyping since all of that starts with getting the data in from somewhere in a standard format.

The problem I have is that I have an ambitious project and too few resources for now.  So I can't at this stage put much time into making anything someone else could use.  But maybe we could work together on parts of this, if that would be interesting.

I am speaking to Vlad Lefenfeld about this a bit too.

On the pure numerical stuff, speak to John Colvin.

If you want you can email me laeeth laeeth dot com.


Laeeth.

On Wednesday, 5 August 2015 at 23:37:37 UTC, Laeeth Isharc wrote:
>
> Dataframes aren't intellectually very exciting, but they are very useful for iterative data exploration and quick prototyping since all of that starts with getting the data in from somewhere in a standard format.
>

May not be intellectually exciting, but look at how popular pandas is for python.

On Thursday, 6 August 2015 at 02:38:15 UTC, jmh530 wrote:
> On Wednesday, 5 August 2015 at 23:37:37 UTC, Laeeth Isharc wrote:
>>
>> Dataframes aren't intellectually very exciting, but they are very useful for iterative data exploration and quick prototyping since all of that starts with getting the data in from somewhere in a standard format.
>>
>
> May not be intellectually exciting, but look at how popular pandas is for python.

Yes - exactly my point.  I figured I needed to acknowledged this however, since smart people often work on what's stimulating rather than what's most useful  given the choice.  Having dataframes will open up more possibilities.  Via PyD and some  extra glue, it shouldn't be hard to make D dataframes  easily convertible to pandas dataframes and back.  Thats perfect for the Ipython notebook because you then don't need to convert all your code at once, and you still have access to Python notebooks.

On Thursday, 6 August 2015 at 06:56:11 UTC, Laeeth Isharc wrote:
> On Thursday, 6 August 2015 at 02:38:15 UTC, jmh530 wrote:
>> On Wednesday, 5 August 2015 at 23:37:37 UTC, Laeeth Isharc wrote:
>>>
>>> Dataframes aren't intellectually very exciting, but they are very useful for iterative data exploration and quick prototyping since all of that starts with getting the data in from somewhere in a standard format.
>>>
>>
>> May not be intellectually exciting, but look at how popular pandas is for python.
>
> Yes - exactly my point.  I figured I needed to acknowledged this however, since smart people often work on what's stimulating rather than what's most useful  given the choice.  Having dataframes will open up more possibilities.  Via PyD and some  extra glue, it shouldn't be hard to make D dataframes  easily convertible to pandas dataframes and back.  Thats perfect for the Ipython notebook because you then don't need to convert all your code at once, and you still have access to Python notebooks.

I mean access to Python libraries.

On Wednesday, 5 August 2015 at 18:49:21 UTC, bachmeier wrote:

> Yes. The question is whether we can put together a group of developers to build the infrastructure, which is a lot more than just code. That means, in particular, good documentation and using it for our own projects.

Right on! I would be willing to help with documentation if there were a concerted effort in this direction. There have been a number of failed individual efforts over the years. So how can a group effort be promoted?

Is the Dscience github project an adequate platform? How can other people get involved? Is a dedicated discussion group needed? Can we develop a plan of some sort rather than just a scatter of individual efforts?

> Everyone these days talks about how Python is a powerhouse scientific programming language. A decade ago it was crap. I know, because I watched it for years wishing I could use it. There were some poorly documented, domain-specific, hacked-together libraries, but Python was not for the most part a suitable choice.

I started with Python in the years of the Numeric+numarray->NumPy transition. It was messy. Personally I think a unified library like NumPy, to underpin other more specialized libraries, is of paramount importance to any success of D in science.

Ideally there would be a NumPy/ndarray usage-compatible module in D. That would make D much more attractive to potential Python converts and lower the entry barrier considerably.