On Wednesday, 4 December 2013 at 19:05:12 UTC, Iain Buclaw wrote:
> Ditto with GDC too... I did a quick search on those two project and
> found that people had reported supposed bugs in GDC, but again none of
> which have been raised in GDC itself.

I think main issue is that no one was able to reduce those low enough for reporting. And yeah, that fiber switch issues is actually one for GDC : https://github.com/rejectedsoftware/vibe.d/issues/101

On 12/4/13 9:59 AM, Atila Neves wrote:
>> Interesting. Care to convert this post (only a little adjustment
>> needed) to a blog and publish with source code? Would make a great
>> article. Ask your friends to contribute with descriptions of their
>> implementations, too.
>
> If I convert this post into a blog it's going to take more than a little
> adjustment for me to be happy with it. I guess having had to write a PhD
> thesis has now scarred me for life.

Yah, I was just trying to lure you into starting on it until it's too late to change your mind :o).

> I'll see if I get around to it in the next few days and I'll ask the
> guys about their implementations as well. I guess I should go set up
> wordpress.
>
> Atila

This is terrific. Would make for a sure hit and a lot of interesting discussion on reddit/hackernews.


Thanks,

Andrei

On Wednesday, 4 December 2013 at 18:13:54 UTC, Etienne wrote:
> I can start sending bug reports in, it would be great to see this working.

Yes, please!

David

On 2013-12-04 17:20, Dicebot wrote:

> Ah this is rather sad. It makes tests results somewhat unstable because
> client load interferes with server load. Ideally client should be
> separate machine and has more powerful h/w than server. However, that
> also requires ~gigabit network in between as well as matching network
> cards to hit the limits of top server performance - this makes such
> tests rather cumbersome to execute.
>
> At the very least you should use process affinity to separate resources
> between client and server in more predictable manner.

Isn't it most important that all languages were tested in the same way?

-- 
/Jacob Carlborg

On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:
> It all started here:
>
> http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar@forum.dlang.org
>
> Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a new benchmark. My implementation didn't do as well on that one so I went back to the code and optimised it. Also, our other buddy Patrick from work decided to get in on this and use it as an excuse to learn Erlang. So now 4 different implementations in 4 different languages are duking it out.
>
> There are two benchmarks, both written in Go. The first is loadtest and it measures throughput. I varied the number of client connections to the server and measured how many messages were sent per second.
>
> The second benchmark is pingtest and it measures latency. The quantity I varied here was the number of wildcard subscribers (it'd take a while to explain what they are to anyone who isn't familiar with MQTT). It has a bunch of client connections talk to each other, and instead of trying to send as many messages as computerly possible they wait for the acknowledgement from the partner connection. The results are as follows (I tried making charts but LibreOffice wasn't helping and then I got bored), with the numbers being thousands of messages per second:
>
> loadtest
> Connections:   100            500            750            1k
> D + vibe.d:    121.7 +/- 1.5  166.9 +/- 1.5  171.1 +/- 3.3  167.9 +/- 1.3
> C (mosquitto): 106.1 +/- 0.8  122.4 +/- 0.4   95.2 +/- 1.3   74.7 +/- 0.4
> Erlang:        104.1 +/- 2.2  124.2 +/- 5.9  117.6 +/- 4.6  117.7 +/- 3.2
> Go:             90.9 +/- 11   100.1 +/- 0.1   99.3 +/- 0.2   98.8 +/- 0.3
>
> pingtest
> wsubs:         20             200            400
> D + vibe.d:    50.9 +/- 0.3   38.3 +/- 0.2   20.1 +/- 0.1
> C (mosquitto): 65.4 +/- 4.4   45.2 +/- 0.2   20.0 +/- 0.0
> Erlang:        49.1 +/- 0.8   30.9 +/- 0.3   15.6 +/- 0.1
> Go:            45.2 +/- 0.2   27.5 +/- 0.1   16.0 +/- 0.1
>
> So, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about.
>
> GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so.
>
> All in all though, so much for his "nothing is going to be faster than Go because this is what it's good at". If he'd measured it against mosquitto first and realised it was already losing to C I probably wouldn't have written an MQTT broker in D, so I guess that was a good thing. :)
>
> Atila
>
> P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdc

Nice benchmark! :)

So, a DMD-compiled D program vs GCC-compiled (?) C program fighting for latency? That might be a reason. Interesting we still win with throughput.

The variance/standard deviation of pingtest-C-20 with 4.4 stands out. What happens there?

> Isn't it most important that all languages were tested in the same way?

Maybe. I actually thought that if there might be a lesson here if it turns out that the quality of the implementation and appropriate use of data structures and algorithms mattered more than the choice of programming language, but in the end competing with each other just made all of the implementations in the same order of magnitude of performance.

This was never meant to be scientific, it's my Physics background showing up when I can't make measurements without calculating error bars.

This started as a lunch joke. I came here to gloat, then you guys go and ask intelligent questions, what's up with that? :P

Atila

> So, a DMD-compiled D program vs GCC-compiled (?) C program fighting for latency? That might be a reason. Interesting we still win with throughput.

Yeah, the C version was compiled with gcc 4.8.2. I'd've used either ldc or gdc with my version but:

. vibe.d didn't even compile with ldc.
. The gdc in the Arch Linux package couldn't compile anything that included one of the files in std, I can't remember which now, for most of the time I worked on this.
. When the gdc package got fixed, I managed to compile my app with vibe.d. It crashed as soon as I tried using it.

> The variance/standard deviation of pingtest-C-20 with 4.4 stands out. What happens there?

Can't remember.

Atila

On Thursday, 5 December 2013 at 08:21:07 UTC, Jacob Carlborg wrote:
> On 2013-12-04 17:20, Dicebot wrote:
>
>> Ah this is rather sad. It makes tests results somewhat unstable because
>> client load interferes with server load. Ideally client should be
>> separate machine and has more powerful h/w than server. However, that
>> also requires ~gigabit network in between as well as matching network
>> cards to hit the limits of top server performance - this makes such
>> tests rather cumbersome to execute.
>>
>> At the very least you should use process affinity to separate resources
>> between client and server in more predictable manner.
>
> Isn't it most important that all languages were tested in the same way?

Depends on your goals :) If you want just to say "hah, your language is nothing compared to my language" it is enough. If you want to make some observations/conclusions about specific implementation performance and how it scale for different conditions it becomes important to remove as many side impact as possible. And of course at high load/concurrency levels client competing with server for connections does make notable impact. In other words, it is good enough for comparison but not good enough for actual performance analysis.

I am the buddy who's always going on and on about Go.

Here's the blog posting I made explaining why/how I made the MQTT server in Go and what I learned:

  http://blog.nella.org/mqtt-code-golf/

I was a bit surprised and interested to see that MQTT with small transactions is not easy to optimize because kernel/user context switching dominates the work that's done under control of the programmer and which can be optimized via data structures and/or reducing GC overhead.

Something that both Atila and I verified in this exercise is that writing a fast network server that scales is so much easier with a very good networking library that takes advantage of lightweight threads, and with a runtime that provides GC. This frees up the programmer to focus on the protocol logic, and leave concurrency and bookkeeping in the capable hands of the machine.

And, as has been mentioned on other Go versus D threads, a lot of this is a matter of taste and team dynamics. I've worked on many teams in my career where there was not a critical mass of people who could reason correctly about C++ memory management, and who could use generic programming techniques reliably. And, in line with discoveries from psychological research, it's common that people who are not competent at something do not recognize that fact. Perhaps I'm above average in this regard: I KNOW I'm not smart enough to write correct code using templating techniques. :)

  -jeff

On Thursday, 5 December 2013 at 14:55:53 UTC, Jeff R. Allen wrote:
> Perhaps I'm above average in this regard: I KNOW I'm not smart enough to write correct code using templating techniques. :)
>
>   -jeff

I bet you're smart enough to do it in D. It's a different world compared to C++ in this regard.