On Tuesday, 18 January 2022 at 17:03:33 UTC, sfp wrote:

> You must also consider that the items that bioinfornatics listed are all somewhat contingent on each other. In isolation they aren't nearly as useful. You might have a numpy/scipy clone, but if you don't also have a matplotlib clone (or some other means of doing data visualization from D) their utility is a bit limited.

To my knowledge pyd still works. There's not much to be gained from rewriting a plotting library from scratch. It's not common that you're plotting 100 million times for each run of your program.

I see too much NIH syndrome here. If you can call another language, all you need to do is write convenience wrappers on top of the many thousands of hours of work done in that language. You can replace the pieces where it makes sense to do so. The goal of the D program is whatever analysis you're doing on top of those libraries, not the libraries themselves.

We call C libraries all the time. Nobody thinks that's a problem. A bunch of effort has gone into calling C++ libraries and there's tons of support for that effort. When it comes to calling any other language, even for things that don't require performance, there's no interest. The ability to interoperate with other languages is the number one reason I started using D and the main reason I still use it.

On Tue, Jan 18, 2022 at 08:28:52PM +0000, bachmeier via Digitalmars-d wrote:
> On Tuesday, 18 January 2022 at 17:03:33 UTC, sfp wrote:
> 
> > You must also consider that the items that bioinfornatics listed are all somewhat contingent on each other. In isolation they aren't nearly as useful. You might have a numpy/scipy clone, but if you don't also have a matplotlib clone (or some other means of doing data visualization from D) their utility is a bit limited.
> 
> To my knowledge pyd still works. There's not much to be gained from rewriting a plotting library from scratch. It's not common that you're plotting 100 million times for each run of your program.
>
> I see too much NIH syndrome here. If you can call another language, all you need to do is write convenience wrappers on top of the many thousands of hours of work done in that language. You can replace the pieces where it makes sense to do so. The goal of the D program is whatever analysis you're doing on top of those libraries, not the libraries themselves.
[...]

+1.  Why do we need to reinvent numpy/scipy? One of the advantages conferred by D's metaprogramming capabilities is easier integration with other languages.  Adam Ruppe's jni.d is one prime example of how metaprogramming can abstract away the nasty amounts of boilerplate you're otherwise forced to write when interfacing with Java via JNI. D's C ABI compatibility also means you can leverage the tons of C libraries out there right now, instead of waiting for somebody to reinvent the same libraries in D years down the road.  D's capabilities makes it very amenable to being a "glue" language for interfacing with other languages.


T

-- 
Caffeine underflow. Brain dumped.

On Tuesday, 18 January 2022 at 21:22:11 UTC, H. S. Teoh wrote:
> On Tue, Jan 18, 2022 at 08:28:52PM +0000, bachmeier via Digitalmars-d wrote:
>> On Tuesday, 18 January 2022 at 17:03:33 UTC, sfp wrote:
>> 
>> > You must also consider that the items that bioinfornatics listed are all somewhat contingent on each other. In isolation they aren't nearly as useful. You might have a numpy/scipy clone, but if you don't also have a matplotlib clone (or some other means of doing data visualization from D) their utility is a bit limited.
>> 
>> To my knowledge pyd still works. There's not much to be gained from rewriting a plotting library from scratch. It's not common that you're plotting 100 million times for each run of your program.
>>
>> I see too much NIH syndrome here. If you can call another language, all you need to do is write convenience wrappers on top of the many thousands of hours of work done in that language. You can replace the pieces where it makes sense to do so. The goal of the D program is whatever analysis you're doing on top of those libraries, not the libraries themselves.
> [...]
>
> +1.  Why do we need to reinvent numpy/scipy? One of the advantages conferred by D's metaprogramming capabilities is easier integration with other languages.  Adam Ruppe's jni.d is one prime example of how metaprogramming can abstract away the nasty amounts of boilerplate you're otherwise forced to write when interfacing with Java via JNI. D's C ABI compatibility also means you can leverage the tons of C libraries out there right now, instead of waiting for somebody to reinvent the same libraries in D years down the road.  D's capabilities makes it very amenable to being a "glue" language for interfacing with other languages.

The next release of my embedr library (which I've been able to do now that my work life is finally returning to normal) will make it trivial to call D functions from R. What I mean by that is that you write a file of D functions and by the magic of metaprogramming, you don't need to write any boilerplate at all.

Example:

```
import mir.random;
import mir.random.variable;

RVector rngexample(int n) {
	auto gen = Random(unpredictableSeed);
	auto rv = uniformVar(-10, 10); // [-10, 10]
	auto result = RVector(n);
	foreach(ii; 0..n) {
		result[ii] = rv(gen);
	}
	return result;
}
mixin(createRFunction!rngexample);
```

The only way you can do better is if someone else writes the program for you. But then it doesn't make much difference which language is used.

On Tuesday, 18 January 2022 at 22:00:42 UTC, bachmeier wrote:
> On Tuesday, 18 January 2022 at 21:22:11 UTC, H. S. Teoh wrote:
>> [...]
>
> The next release of my embedr library (which I've been able to do now that my work life is finally returning to normal) will make it trivial to call D functions from R. What I mean by that is that you write a file of D functions and by the magic of metaprogramming, you don't need to write any boilerplate at all.
>
> Example:
>
> ```
> import mir.random;
> import mir.random.variable;
>
> RVector rngexample(int n) {
> 	auto gen = Random(unpredictableSeed);
> 	auto rv = uniformVar(-10, 10); // [-10, 10]
> 	auto result = RVector(n);
> 	foreach(ii; 0..n) {
> 		result[ii] = rv(gen);
> 	}
> 	return result;
> }
> mixin(createRFunction!rngexample);
> ```
>
> The only way you can do better is if someone else writes the program for you. But then it doesn't make much difference which language is used.

This is all news to me. It's a shame these libraries and their capabilities aren't advertised my prominently.

How hard would it be to automatically wrap a D library and expose it to Python, MATLAB, and Julia simultaneously? Say the library even has a simple C-style API, or a very simple single-inheritance OO hierarchy with no templates.

On Tuesday, 18 January 2022 at 21:22:11 UTC, H. S. Teoh wrote:
> [snip]
>
> +1.  Why do we need to reinvent numpy/scipy? One of the advantages conferred by D's metaprogramming capabilities is easier integration with other languages.  Adam Ruppe's jni.d is one prime example of how metaprogramming can abstract away the nasty amounts of boilerplate you're otherwise forced to write when interfacing with Java via JNI. D's C ABI compatibility also means you can leverage the tons of C libraries out there right now, instead of waiting for somebody to reinvent the same libraries in D years down the road.  D's capabilities makes it very amenable to being a "glue" language for interfacing with other languages.
>
>
> T

I'm all for leveraging C libraries in D, but if you have code that needs to be performant then you may run into limitations with python. If you're building one chart with Matplotlib, then it's probably fine. If you have some D code that takes longer to run (e.g. a simulation that deals with a lot of data and many threads), then you might be a little more careful about what python code to incorporate and how. I don't know the technical details needed to get the best performance in that situation (are there benchmarks?), but I saw some work done about using python buffer protocol when calling D functions from python.

In addition, the python code might itself be calling the same C libraries that D can (e.g. LAPACK) (though potentially with different defaults, trading off performance vs. accuracy, resulting in python being faster in some cases than D). In that case, python is also a glue language. Taking the same approach in D can simplify your code base a little bit and you don't need to worry about any additional overhead or limitations from GIL that might get introduced.

Again, not something you need to worry about when performance is not a big issue.

On Tuesday, 18 January 2022 at 22:21:40 UTC, Ola Fosheim Grøstad wrote:
> ...D's potential strength here is not so much in being able to bind to C++ in a limited fashion (like Python), but being able to port C++ to D and improve on it. To get there you need feature parity, which is what this thread is about.

Not just 'feature' parity, but 'performance' parity too:

"Broad adoption of high-level languages by the scientific community is unlikely without compiler optimizations to mitigate the performance penalties these languages abstractions impose." - https://www.cs.rice.edu/~vs3/PDF/Joyner-MainThesis.pdf

On Wednesday, 19 January 2022 at 04:45:20 UTC, forkit wrote:
> On Tuesday, 18 January 2022 at 22:21:40 UTC, Ola Fosheim Grøstad wrote:
>> ...D's potential strength here is not so much in being able to bind to C++ in a limited fashion (like Python), but being able to port C++ to D and improve on it. To get there you need feature parity, which is what this thread is about.
>
> Not just 'feature' parity, but 'performance' parity too:
>
> "Broad adoption of high-level languages by the scientific community is unlikely without compiler optimizations to mitigate the performance penalties these languages abstractions impose." - https://www.cs.rice.edu/~vs3/PDF/Joyner-MainThesis.pdf

That paper is from 2008, meanwhile in 2021,

https://www.hpcwire.com/off-the-wire/julia-joins-petaflop-club//

This is what D has to compete against, not only C++ with the existing SYSCL/CUDA tooling and their ongoing integration into ISO C++.