I've just created a new dsource project and uploaded my collection of numerical routines. You can find it here:

  http://www.dsource.org/projects/scid

Hopefully others will find it useful as well. Most of the code is written with a "get it to work - NOW" philoshophy, and as such there is a lot of potential for improvement and polish, not least when it comes to performance. I am aware of this, so there is really no need to benchmark it against established scientific libraries quite yet. ;) More routines will be added as I write them (which is usually when I need them for work), and the existing ones will be polished whenever I have time to do so.

Huge thanks to Bill Baxter for letting me use his BLAS and LAPACK bindings!

Requirements:
  DMD 2.037
  BLAS and LAPACK libraries

-Lars

== Quote from Lars T. Kyllingstad (public@kyllingen.NOSPAMnet)'s article
> I've just created a new dsource project and uploaded my collection of
> numerical routines. You can find it here:
>    http://www.dsource.org/projects/scid
> Hopefully others will find it useful as well. Most of the code is
> written with a "get it to work - NOW" philoshophy, and as such there is
> a lot of potential for improvement and polish, not least when it comes
> to performance. I am aware of this, so there is really no need to
> benchmark it against established scientific libraries quite yet. ;) More
> routines will be added as I write them (which is usually when I need
> them for work), and the existing ones will be polished whenever I have
> time to do so.
> Huge thanks to Bill Baxter for letting me use his BLAS and LAPACK bindings!
> Requirements:
>    DMD 2.037
>    BLAS and LAPACK libraries
> -Lars

This looks absolutely terrific given that it's a work in progress and all.  I agree that we shouldn't be too obsessed with performance yet because:

1.  **Make it work**, make it right, make it fast.  We can always optimize it
later provided that there aren't any ridiculously bad design decisions.
2.  At least some of the time people (myself included) would rather a nice API and
decent performance than an ugly API (read:  a straight Fortran or C API) and
lightening quick performance.

We've got the beginnings of a good matrix/numerics lib in your work.  I believe that my dstats lib would provide just about all the statistical functionality a good scientific lib would need, and I'm getting close to declaring it beta-quality.  I'd say based on a quick look at what scipy has that we also need the following for a credible full-fledged scientific lib:

Machine learning, i.e. classification and clustering.  (I'd probably be qualified to write that and was halfway thinking of breaking ground on it over Christmas break.  However, I'm not committing to this yet, so if someone else already has a work in progress, let me know.  Also, there's a Java machine learning library called Weka that I've used before.  It's very complete but the API is painful and it's GPL, so IDK if it'd be worth it to port to D.)

Optimization.  (A few people have tried but IDK if they've actually gotten far off the ground with it.)

Basic image I/O and processing.

Plotting.  I've considered doing this a few times, but I've decided it needs to be put off until D2 is stable and the GUI toolkits for it are reasonably stable.  One layer of instability (D2 itself) is workable, but two layers (D2 and the GUI libs) is not.

dsimcha wrote:
> == Quote from Lars T. Kyllingstad (public@kyllingen.NOSPAMnet)'s article
>> I've just created a new dsource project and uploaded my collection of
>> numerical routines. You can find it here:
>>    http://www.dsource.org/projects/scid
>> Hopefully others will find it useful as well. Most of the code is
>> written with a "get it to work - NOW" philoshophy, and as such there is
>> a lot of potential for improvement and polish, not least when it comes
>> to performance. I am aware of this, so there is really no need to
>> benchmark it against established scientific libraries quite yet. ;) More
>> routines will be added as I write them (which is usually when I need
>> them for work), and the existing ones will be polished whenever I have
>> time to do so.
>> Huge thanks to Bill Baxter for letting me use his BLAS and LAPACK bindings!
>> Requirements:
>>    DMD 2.037
>>    BLAS and LAPACK libraries
>> -Lars
> 
> This looks absolutely terrific given that it's a work in progress and all.  I
> agree that we shouldn't be too obsessed with performance yet because:
> 
> 1.  **Make it work**, make it right, make it fast.  We can always optimize it
> later provided that there aren't any ridiculously bad design decisions.

A lot of the code is ported from 20-30 year old (i.e. extremely well tested) FORTRAN code, so any performance gains would be a bonus provided by the D language itself, and most likely not in the algorithms.

Also, if you spot any ridiculously bad design decisions, let me know. ;)


> 2.  At least some of the time people (myself included) would rather a nice API and
> decent performance than an ugly API (read:  a straight Fortran or C API) and
> lightening quick performance.

Yeah, that's what I think too. And, the way it is now, people who for some reason want to use the old-style api can just access the scid.ports functions directly.


> We've got the beginnings of a good matrix/numerics lib in your work.  I believe
> that my dstats lib would provide just about all the statistical functionality a
> good scientific lib would need, and I'm getting close to declaring it
> beta-quality.  I'd say based on a quick look at what scipy has that we also need
> the following for a credible full-fledged scientific lib:
> 
> Machine learning, i.e. classification and clustering.  (I'd probably be qualified
> to write that and was halfway thinking of breaking ground on it over Christmas
> break.  However, I'm not committing to this yet, so if someone else already has a
> work in progress, let me know.  Also, there's a Java machine learning library
> called Weka that I've used before.  It's very complete but the API is painful and
> it's GPL, so IDK if it'd be worth it to port to D.)
> 
> Optimization.  (A few people have tried but IDK if they've actually gotten far off
> the ground with it.)
> 
> Basic image I/O and processing.
> 
> Plotting.  I've considered doing this a few times, but I've decided it needs to be
> put off until D2 is stable and the GUI toolkits for it are reasonably stable.  One
> layer of instability (D2 itself) is workable, but two layers (D2 and the GUI libs)
> is not.

I agree with all of this. Here's a piece of advice to anyone who wants to try their hand at these and other things: There are a LOT of established and *extremely* well-tested numerical libraries lying around on the internet, and many of them are in the public domain. Instead of spending a lot of time re-inventing the wheel (I've tried it), port some old code and try to improve on it when you get it working. Here are a few starting points:

  http://www.netlib.org/
  http://gams.nist.gov/

And I forgot to say in my original announcement: I welcome contributors with open arms. I have no intention of making this a one-man show. If you want to contribute or participate in some way, let me know.

-Lars

dsimcha wrote:
> We've got the beginnings of a good matrix/numerics lib in your work.  I believe
> that my dstats lib would provide just about all the statistical functionality a
> good scientific lib would need, and I'm getting close to declaring it
> beta-quality.  I'd say based on a quick look at what scipy has that we also need
> the following for a credible full-fledged scientific lib:

dstats has actually been kind of an inspiration for me. I've never used it, because I don't do any statistics, but I've looked through the API docs and the examples, and I really like the interface.


> Machine learning, i.e. classification and clustering.  (I'd probably be qualified
> to write that and was halfway thinking of breaking ground on it over Christmas
> break.  However, I'm not committing to this yet, so if someone else already has a
> work in progress, let me know.  Also, there's a Java machine learning library
> called Weka that I've used before.  It's very complete but the API is painful and
> it's GPL, so IDK if it'd be worth it to port to D.)
> 
> Optimization.  (A few people have tried but IDK if they've actually gotten far off
> the ground with it.)
> 
> Basic image I/O and processing.
> 
> Plotting.  I've considered doing this a few times, but I've decided it needs to be
> put off until D2 is stable and the GUI toolkits for it are reasonably stable.  One
> layer of instability (D2 itself) is workable, but two layers (D2 and the GUI libs)
> is not.

A few things that can be added to the list:

- special functions
- Fourier transforms
- Monte Carlo methods
- series
- least-squares fitting
- differential and integral equations

There is a good chance that I will need the latter for work just after the holidays, in which case I'll write it myself.

I'll add the above lists to the project home page.

-Lars

Lars T. Kyllingstad wrote:
> Most of the code is written with a "get it to work - NOW" philoshophy, and as such there is a lot of potential for improvement and polish, not least when it comes to performance. I am aware of this, so there is really no need to benchmark it against established scientific libraries quite yet.

I wish more of software were written with this attitude. Too many a promising project is abandoned because it is not "proper", "comprehensive", "neat", "polished", "cool", "impressive", "performant", "unique", "impressive", "trail-blazing", ... And the only one who loses is US. WE lose.

This is tantamount to quenching creativity, the very property that differentiates us from the aboriginals. But observe: not because they'd be inferior, but simply because such a big part of their existence is consumed by the very acts of finding food and shelter, that we take for granted, while we have the time to indulge in quests of remote rewards.

The difference between "the Nike philosophy" (just do it), and doing it "the day AFTER tomorrow, after proper design and deliberation", is not the same as between "Bill Gates' get it out now, who cares how crappy", and "wait 'till it's perfect".

On 22.12.2009 1:44, Georg Wrede wrote:
> Lars T. Kyllingstad wrote:
>>  Most of the code is
>>  written with a "get it to work - NOW" philoshophy, and as such there is
>>  a lot of potential for improvement and polish, not least when it comes
>>  to performance. I am aware of this, so there is really no need to
>>  benchmark it against established scientific libraries quite yet.
>
> I wish more of software were written with this attitude. Too many a
> promising project is abandoned because it is not "proper",
> "comprehensive", "neat", "polished", "cool", "impressive", "performant",
> "unique", "impressive", "trail-blazing", ... And the only one who loses
> is US. WE lose.
>
> This is tantamount to quenching creativity, the very property that
> differentiates us from the aboriginals. But observe: not because they'd
> be inferior, but simply because such a big part of their existence is
> consumed by the very acts of finding food and shelter, that we take for
> granted, while we have the time to indulge in quests of remote rewards.
>
> The difference between "the Nike philosophy" (just do it), and doing it
> "the day AFTER tomorrow, after proper design and deliberation", is not
> the same as between "Bill Gates' get it out now, who cares how crappy",
> and "wait 'till it's perfect".

LOL WUT



Sorry, someone had to say it. :)