On Tuesday, 17 March 2015 at 21:00:11 UTC, bachmeier wrote:
> On Tuesday, 17 March 2015 at 19:00:06 UTC, jmh530 wrote:
>> In addition, further development of the ability to call D from R or Python* or Julia (or vice-versa) would also be a positive.
>
> What do you have in mind? I no longer work much with Python so my knowledge is limited, but calling D from R or Julia should be no different from calling C from those languages, as you normally compile your C code into a shared library anyway.
>
> I've done the R->D thing many times and in the process have worked with a big chunk of the R API. Things like allocating R data structures from a D function, adding assertions to your D code to allow for an easy exit when things don't work out, and calling functions in the R math library, among other things, are not difficult.

PyD is pretty nice, although one wouldn't want to call it from an inner loop.  And it's easy to go via cython also.  (I wonder how well cython works with interfacing with D via the C++ interface, because that way you could extend python with D classes and have them be faster than going through PyD).

PyD also works with numpy memoryviews I think.

For Julia, easy to call D.  It would be nice to port julia.h to D (I started, but got distracted) so D can call back Julia.

On Tuesday, 17 March 2015 at 22:04:08 UTC, jmh530 wrote:
> Nevertheless, I think that there would be a lot of value in writing up what you have done with R and D. That sounds very interesting.

I posted a link elsewhere in this thread, but it's buried in all these posts, so here it is again:

https://bitbucket.org/bachmeil/dmdinline

Unfortunately there is little documentation (though I'm working on that). I only use Linux but I would be happy if someone that knows Windows would find that it works there. I do a lot of matrix algebra, regressions, and simple rewriting of loops.

I don't personally use lme4. I looked at the source on Github. Everything has to be passed as an SEXP struct from R to C++ and you have to return a single SEXP struct. I believe all the C++ code used with Rcpp is marked extern "C". D can create and take as function arguments SEXP's, so you should be able to call into the code the same way that R does. That's not to say it's trivial but it is doable. It might make a good example of the dmdinline package. It depends on how much time it takes to get it working and how much of the R code would have to be rewritten in D.

On Wednesday, 18 March 2015 at 01:52:00 UTC, Laeeth Isharc wrote:
> PyD is pretty nice, although one wouldn't want to call it from an inner loop.

Why wouldn't you want to call it from an inner loop?

On Wednesday, 18 March 2015 at 02:02:27 UTC, bachmeier wrote:
> On Wednesday, 18 March 2015 at 01:52:00 UTC, Laeeth Isharc wrote:
>> PyD is pretty nice, although one wouldn't want to call it from an inner loop.
>
> Why wouldn't you want to call it from an inner loop?

See benchmark someone did a while back that had D relatively slow, which prob was PyD judging by comment from the developer.  But there are different kinds of demands, and I can't say I am highly experienced with using pyD in that context.  So if you found different I would love to hear more.

If I had time I would experiment, but I don't right now unfort.

On Tuesday, 17 March 2015 at 20:50:51 UTC, Bienlein wrote:
>
>> Go is only a CSP-like, it isn't CSP. cf Python-CSP and PyCSP, not to
>> mention JCSP and GPars.
>
> I'm not really sure whether this can be put exactly that way. On a machine with 4 GB RAM you can spawn about 80.000 goroutines (aka green threads). Let's say each threads calculates a large fibonacci number. If the fibonacci calculation yields the processor frequently all 80.000 run seamingly in parallel and return their result almost "at the same time".
>
> With GPars this would not work. You can only start some 2.000 Java threads on a machine with the same amount of memory. If also the fibonacci calculation in Groovy/GPars yields the processor, using GPars the first 2.000 fibonacci calculations will nevertheless crowd out the following 2.000 threads of all those 80.000 fibonacci calculations and so on. I once tried this out with both Go and Groovy/GPars.
>
> -- Bienlein

What about using a JVM with green threads support or Quasar, wouldn't it be more comparable?

--
Paulo

On Tue, 2015-03-17 at 20:50 +0000, Bienlein via Digitalmars-d wrote:
> > Go is only a CSP-like, it isn't CSP. cf Python-CSP and PyCSP,
> > not to
> > mention JCSP and GPars.
> 
> I'm not really sure whether this can be put exactly that way. On a machine with 4 GB RAM you can spawn about 80.000 goroutines (aka green threads). Let's say each threads calculates a large fibonacci number. If the fibonacci calculation yields the processor frequently all 80.000 run seamingly in parallel and return their result almost "at the same time".
> 
> With GPars this would not work. You can only start some 2.000 Java threads on a machine with the same amount of memory. If also the fibonacci calculation in Groovy/GPars yields the processor, using GPars the first 2.000 fibonacci calculations will nevertheless crowd out the following 2.000 threads of all those 80.000 fibonacci calculations and so on. I once tried this out with both Go and Groovy/GPars.

<This needs to go to the GPars mailing lists list so I have cross posted.>

A priori I do not believe the claim made here: a GPars task is submitted to a thread pool, which is exactly what the goroutines are. Thus the number of Java threads is not a bound on the number of GPars tasks. Any bounds will be provided by the Fork/Join pool.

If you have code you can send me, I will run it and do the benchmarking. If not I guess I will have to write some. Until we have the results of executing code, the above must be treated as pure conjecture, which I believe will be refuted. If the claim however turns out to be true, then it highlights a bug in GPars.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Tue, 2015-03-17 at 15:30 -0700, Andrei Alexandrescu via Digitalmars-d wrote:
> On 3/17/15 1:50 PM, Bienlein wrote:
> >
> >> Go is only a CSP-like, it isn't CSP. cf Python-CSP and PyCSP, not to mention JCSP and GPars.
> >
> > I'm not really sure whether this can be put exactly that way. On a machine with 4 GB RAM you can spawn about 80.000 goroutines (aka green threads). Let's say each threads calculates a large fibonacci number. If the fibonacci calculation yields the processor frequently all 80.000 run seamingly in parallel and return their result almost "at the same time".
> 
> How many physical threads would the 80K goroutines create? -- Andrei
> 
It doesn't, you specify the number of kernel threads in the thread pool manually. The default is 1.
> 

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Tue, 2015-03-17 at 22:49 +0000, deadalnix via Digitalmars-d wrote:
> On Tuesday, 17 March 2015 at 22:30:37 UTC, Andrei Alexandrescu wrote:
> > On 3/17/15 1:50 PM, Bienlein wrote:
> >>
> >>> Go is only a CSP-like, it isn't CSP. cf Python-CSP and PyCSP,
> >>> not to
> >>> mention JCSP and GPars.
> >>
> >> I'm not really sure whether this can be put exactly that way.
> >> On a
> >> machine with 4 GB RAM you can spawn about 80.000 goroutines
> >> (aka green
> >> threads). Let's say each threads calculates a large fibonacci
> >> number. If
> >> the fibonacci calculation yields the processor frequently all
> >> 80.000 run
> >> seamingly in parallel and return their result almost "at the
> >> same time".
> >
> > How many physical threads would the 80K goroutines create? -- Andrei
> 
> The Go runtime manages this. About as many as there are CPU.

The Go runtime manages the threadpool and scheduling of kernel threads, but the number of them remains a number specified manually unless you want the default of 1.

Benchmarking generally reveals that optimal number of kernel threads in a pool is number of CPUs ±1, it depends on the work and things like cacheline misses, hyperthreads, etc., etc.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

> What about using a JVM with green threads support or Quasar, wouldn't it be more comparable?
>
> --
> Paulo

Long text, contents of common interest in the last section :-)

Thanks for the hints, Paulo. Quasar looks interesting. The current number one actor implementation for the JVM is Akka (akka.io). It was earlier built on Hawtdispatch (https://github.com/fusesource/hawtdispatch), which is similar to Quasar but much simpler. Now they have plugged in their own scheduler. Those libraries follow the same approach that a small number of threads (small to make the overhead of context switches small) serve a large number of consumers (e.g., channels, actors, or whatever consumer abstraction). This works well as long as all tasks are short-runners. It becomes a problem once you have many long-runners. Me talking about large fibonacci numbers in my previous post was to indicate that it is all about the problem with long-runners. In Vert.x they have a separate scheduler for long-runners that can serve a certain number of tasks and rejects tasks once the max is exceeded. I will check out have they have implemented channel select in Quasar. In Go they can do this efficiently, because it is done by the built-in scheduler and not in the library.

As what D is concerned it has vibe.d, which follows on principle the same approach as described above from what I understand about it. There is an old discussion about integrating vibe.d in the D distribution. This way multi-threading in D comes with batteries included as in Go. I still think this is a good idea :-).

On Wed, 2015-03-18 at 08:58 +0000, Bienlein via Digitalmars-d wrote: […]
> Thanks for the hints, Paulo. Quasar looks interesting. The current number one actor implementation for the JVM is Akka (akka.io). It was earlier built on Hawtdispatch

Possibly, but there are others that get a lot of use, GPars for example :-)

[…]

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder