When looking at the success of Go it seems to me that it is caused to a large extend by the kind of multi-threading Go offers which is something like "spawn as many thousand threads as you like".

Being able to spawn as many thousand threads as needed without caring about it seems to be an important aspect for being an interesting offering for developing server-side software. It would be nice if D could also play in that niche. This could be some killer domain for D beyond being a better C++.

While Go uses channels and goroutines D takes some actor-style approach when spawning threads. This is also fine, but the problems remains that you cannot create as many D kernel threads just as you like. Maybe this could be something to improve in D and promote in order to give D a further boost. I don't mean to be pushy, it's just about exchanging ideas ;-). The FiberScheduler by Sean Kelly could achieve something in that direction. What do you think?

Regards, Bienlein

Am 16.04.2014 15:59, schrieb Bienlein:
> When looking at the success of Go it seems to me that it is caused to a
> large extend by the kind of multi-threading Go offers which is something
> like "spawn as many thousand threads as you like".
>
> Being able to spawn as many thousand threads as needed without caring
> about it seems to be an important aspect for being an interesting
> offering for developing server-side software. It would be nice if D
> could also play in that niche. This could be some killer domain for D
> beyond being a better C++.
>
> While Go uses channels and goroutines D takes some actor-style approach
> when spawning threads. This is also fine, but the problems remains that
> you cannot create as many D kernel threads just as you like. Maybe this
> could be something to improve in D and promote in order to give D a
> further boost. I don't mean to be pushy, it's just about exchanging
> ideas ;-). The FiberScheduler by Sean Kelly could achieve something in
> that direction. What do you think?
>
> Regards, Bienlein

I agree, but I also wonder why you still keep ignoring vibe.d. It achieves exactly that - right now! Integration with std.concurrency would be great, but at least for now it has an API compatible replacement that can be merged later when Sean's pull request is done.

On Wednesday, 16 April 2014 at 14:06:13 UTC, Sönke Ludwig wrote:

> I agree, but I also wonder why you still keep ignoring vibe.d. It achieves exactly that - right now! Integration with std.concurrency would be great, but at least for now it has an API compatible replacement that can be merged later when Sean's pull request is done.

The point is that vibe.d is a distributed solution. Nothing wrong about that. But in a single instance of some Go program you can >locally< spawn "as many threads as you like" with the number of threads easily being 50.000 and more. It seems that in the Go community people often do that without going distributed. Looks like this makes things a lot simpler when writing server-side applications.

Am 16.04.2014 16:16, schrieb Bienlein:
> On Wednesday, 16 April 2014 at 14:06:13 UTC, Sönke Ludwig wrote:
>
>> I agree, but I also wonder why you still keep ignoring vibe.d. It
>> achieves exactly that - right now! Integration with std.concurrency
>> would be great, but at least for now it has an API compatible
>> replacement that can be merged later when Sean's pull request is done.
>
> The point is that vibe.d is a distributed solution. Nothing wrong about
> that. But in a single instance of some Go program you can >locally<
> spawn "as many threads as you like" with the number of threads easily
> being 50.000 and more. It seems that in the Go community people often do
> that without going distributed. Looks like this makes things a lot
> simpler when writing server-side applications.

I still don't understand what you mean by "distributed". Spawning 50.000 tasks:

	import vibe.core.core;
	import std.stdio;
	
	void main()
	{
	    foreach (i; 0 .. 50_000)
		runTask({
			writefln("Hello, World!");
		});
	}

Alternatively, runWorkerTask will also distribute the tasks among a set of worker threads, which would be more in line with Go AFAIK.

On Wednesday, 16 April 2014 at 14:16:30 UTC, Bienlein wrote:
> On Wednesday, 16 April 2014 at 14:06:13 UTC, Sönke Ludwig wrote:
>
>> I agree, but I also wonder why you still keep ignoring vibe.d. It achieves exactly that - right now! Integration with std.concurrency would be great, but at least for now it has an API compatible replacement that can be merged later when Sean's pull request is done.
>
> The point is that vibe.d is a distributed solution. Nothing wrong about that. But in a single instance of some Go program you can >locally< spawn "as many threads as you like" with the number of threads easily being 50.000 and more. It seems that in the Go community people often do that without going distributed. Looks like this makes things a lot simpler when writing server-side applications.

Goroutines are not threads and by calling them as such you only confuse yourself. Their D counterpart is fiber and you can definitely spawn 50 000 fibers for single local thread.

It is not the first time you try to advocate some Go features without first actually exploring relevant domain. All you describe is already implemented in vibe.d

On Wednesday, 16 April 2014 at 14:21:03 UTC, Sönke Ludwig wrote:

> I still don't understand what you mean by "distributed". Spawning 50.000 tasks:
>
> 	import vibe.core.core;
> 	import std.stdio;
> 	
> 	void main()
> 	{
> 	    foreach (i; 0 .. 50_000)
> 		runTask({
> 			writefln("Hello, World!");
> 		});
> 	}
>
> Alternatively, runWorkerTask will also distribute the tasks among a set of worker threads, which would be more in line with Go AFAIK.

All right, I see. I spent some time looking at the vibe.d homepage and I never saw any other code than something like this:

shared static this()
{
	auto settings = new HTTPServerSettings;
	settings.port = 8080;

	listenHTTP(settings, &handleRequest);
}

void handleRequest(HTTPServerRequest req,
                   HTTPServerResponse res)
{
	res.writeBody("Hello, World!", "text/plain");
}

Not wanting just to be right, but things like that should still be in some base library of the language and not in some 3rd party library. The vibe.d homepage says "As soon as a running fiber calls a special yield() function, it returns control to the function that started the fiber.". Yielding in the FiberScheduler by Sean Kelly is transparent. That's an important point to be easy to use, I think. Also the use of libevent is mentioned. I don't understand what the implications of that exactly is.

What I mean is that some nice transparent solution in a base library for the "some ten thousand threads thing" would be nice.

Am 16.04.2014 16:43, schrieb Bienlein:
> On Wednesday, 16 April 2014 at 14:21:03 UTC, Sönke Ludwig wrote:
>
>> I still don't understand what you mean by "distributed". Spawning
>> 50.000 tasks:
>>
>>     import vibe.core.core;
>>     import std.stdio;
>>
>>     void main()
>>     {
>>         foreach (i; 0 .. 50_000)
>>         runTask({
>>             writefln("Hello, World!");
>>         });
>>     }
>>
>> Alternatively, runWorkerTask will also distribute the tasks among a
>> set of worker threads, which would be more in line with Go AFAIK.
>
> All right, I see. I spent some time looking at the vibe.d homepage and I
> never saw any other code than something like this:
>
> shared static this()
> {
>      auto settings = new HTTPServerSettings;
>      settings.port = 8080;
>
>      listenHTTP(settings, &handleRequest);
> }
>
> void handleRequest(HTTPServerRequest req,
>                     HTTPServerResponse res)
> {
>      res.writeBody("Hello, World!", "text/plain");
> }
>
> Not wanting just to be right, but things like that should still be in
> some base library of the language and not in some 3rd party library. The
> vibe.d homepage says "As soon as a running fiber calls a special yield()
> function, it returns control to the function that started the fiber.".
> Yielding in the FiberScheduler by Sean Kelly is transparent. That's an
> important point to be easy to use, I think. Also the use of libevent is
> mentioned. I don't understand what the implications of that exactly is.

It *is* transparent. Once a blocking operation, such as I/O or waiting for a message, is triggered, it will implicitly yield. But the text indeed doesn't make it very clear. The explicit yield() function is meant for (rare) cases where more control is needed, for example during lengthy computations.

Libevent is just an abstraction layer above the various asynchronous I/O APIs. It provides a platform independent way to get notified about finished operations. There is also a native WinAPI based implementation that enables integration with GUI applications.

>
> What I mean is that some nice transparent solution in a base library for
> the "some ten thousand threads thing" would be nice.

That would indeed be nice to have in the standard library, but it also needs to be carefully planned out, as it has a lot of implications on the existing code, such as making all blocking functions compatible with the fiber based model. Without a full work over it will most likely do more harm than good. And having it in a third party library allows for evolution until a stable state is reached before starting to introduce breaking changes in the standard library.

On Wed, 2014-04-16 at 13:59 +0000, Bienlein via Digitalmars-d wrote:
> When looking at the success of Go it seems to me that it is caused to a large extend by the kind of multi-threading Go offers which is something like "spawn as many thousand threads as you like".

A critically important thing here is the separation of goroutine and thread, i.e. the concurrency and parallelism is about abstraction of the programmers' expression and the underlying implementation — thread pool. Go is not about multi-threading, it is about using goroutines and programmers don't care about threads at all (to a third approximation).

> Being able to spawn as many thousand threads as needed without caring about it seems to be an important aspect for being an interesting offering for developing server-side software. It would be nice if D could also play in that niche. This could be some killer domain for D beyond being a better C++.

Go does not spawn thousands of threads, see above :-)

C++11, and increasingly C++17 are making C++ into a totally different language that 1980s C++ and C++99. It even has proposals for a reasonable concurrency and parallelism layer over the now standardized threads. Sadly though there are some really bad proposals being made to the standards committee. C++ is suffering from the fact that people with the right ideas are not proposing them for C++. Anthony Williams, Roger Orr, Jonathan Wakeley and others are doing as good a job as they can trying to make good stuff so there is some hope it will turn out well. It is a great shame that the same effort is not going into improving D's offerings here: D is in a far better position to do so much better that C++ and what it has.

> While Go uses channels and goroutines D takes some actor-style approach when spawning threads. This is also fine, but the problems remains that you cannot create as many D kernel threads just as you like. Maybe this could be something to improve in D and promote in order to give D a further boost. I don't mean to be pushy, it's just about exchanging ideas ;-). The FiberScheduler by Sean Kelly could achieve something in that direction. What do you think?

Go doesn't spawn threads, see above :-)

D would be significantly improved for a CSP implementation (which is what goroutines and channels realize). Also a fork-join framework would be a useful addition.

The problem is resource. Vibe.d, std.parallelism, std.concurrency provide some tools but for CSP and dataflow, no-one has scratched the itch.  I had been intending to do one project with D and GtkD, but ended up switching to Go + QML because it was easier to do that than write a CSP system for D.  For another C++ and Gtk project, it is easier to wait for early C++17 implementations than it is to port the code to D (*).


(*) There is an element of "how many programmers risk" here not just technical one. There are many more C++ programmers around who can use new C++ style and features than there are D programmers.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Wed, 2014-04-16 at 16:06 +0200, Sönke Ludwig via Digitalmars-d wrote: […]
> 
> I agree, but I also wonder why you still keep ignoring vibe.d. It achieves exactly that - right now! Integration with std.concurrency would be great, but at least for now it has an API compatible replacement that can be merged later when Sean's pull request is done.

Vibe.d is a single-thread event system, which is great (*) for the sort of problems Node.js, Vert.x, Tornado, Flask, Sinatra, Ratpack are used for. The point here is that CSP and dataflow are a concurrency and parallelism model that D has not got.

std.concurrency is a heavyweight thread system so not really useful except to build thread pools and fork-join infrastructure. (OK that is a gross oversimplification.) std.parallelism is a great beginning of data parallelism on a thread pool. It needs more work. The thread pool needs to be brought front and centre, as a separate thing usable by other modules. On this CSP, dataflow, actors, etc. can be built.

Due to other commitments, not least leading a massive update of GPars, I cannot lead on working on D things. If however someone can drive, I will certainly contribute, along the lines as I did when David Simcha wrote std.parallelism – mostly as a tester and reviewer.

This also raises the issue of the D infrastructure having an obvious and documented way for people to contribute to things like std.parallelism. Whatever the truth, the perception is that to work on something like std.parallelism, you have to fork the whole of Phobos. In fact, std.parallelism is a single file 4,500 lines long (**).


(*) It would be even better if it supported mocking for unit tests ;-)

(**) I am still not a fan of single files this big.
-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Wednesday, 16 April 2014 at 13:59:15 UTC, Bienlein wrote:
> Being able to spawn as many thousand threads as needed without caring about it seems to be an important aspect for being an interesting offering for developing server-side software. It would be nice if D could also play in that niche. This could be some killer domain for D beyond being a better C++.

I believe there was a benchmark comparing vibe.d to go with respect to processing of thousands of trivial requests, which proved that vibe.d is up to the task. And server doesn't really need local concurrency: client requests are isolated and have nothing to communicate to each other.