It all started here:

http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar@forum.dlang.org

Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a new benchmark. My implementation didn't do as well on that one so I went back to the code and optimised it. Also, our other buddy Patrick from work decided to get in on this and use it as an excuse to learn Erlang. So now 4 different implementations in 4 different languages are duking it out.

There are two benchmarks, both written in Go. The first is loadtest and it measures throughput. I varied the number of client connections to the server and measured how many messages were sent per second.

The second benchmark is pingtest and it measures latency. The quantity I varied here was the number of wildcard subscribers (it'd take a while to explain what they are to anyone who isn't familiar with MQTT). It has a bunch of client connections talk to each other, and instead of trying to send as many messages as computerly possible they wait for the acknowledgement from the partner connection. The results are as follows (I tried making charts but LibreOffice wasn't helping and then I got bored), with the numbers being thousands of messages per second:

loadtest
Connections:   100            500            750            1k
D + vibe.d:    121.7 +/- 1.5  166.9 +/- 1.5  171.1 +/- 3.3  167.9 +/- 1.3
C (mosquitto): 106.1 +/- 0.8  122.4 +/- 0.4   95.2 +/- 1.3   74.7 +/- 0.4
Erlang:        104.1 +/- 2.2  124.2 +/- 5.9  117.6 +/- 4.6  117.7 +/- 3.2
Go:             90.9 +/- 11   100.1 +/- 0.1   99.3 +/- 0.2   98.8 +/- 0.3

pingtest
wsubs:         20             200            400
D + vibe.d:    50.9 +/- 0.3   38.3 +/- 0.2   20.1 +/- 0.1
C (mosquitto): 65.4 +/- 4.4   45.2 +/- 0.2   20.0 +/- 0.0
Erlang:        49.1 +/- 0.8   30.9 +/- 0.3   15.6 +/- 0.1
Go:            45.2 +/- 0.2   27.5 +/- 0.1   16.0 +/- 0.1

So, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about.

GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so.

All in all though, so much for his "nothing is going to be faster than Go because this is what it's good at". If he'd measured it against mosquitto first and realised it was already losing to C I probably wouldn't have written an MQTT broker in D, so I guess that was a good thing. :)

Atila

P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdc

On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:
> GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so.

GC should not impact general performance in such scenario but is likely to hinder latency which pretty much matches what you are observing. Can you possibly provide information about network layout, h/w and server code used for testing? There are lot of possible oversights that can make results less legible, would have been nice to verify those ;)

On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:
> It all started here:
>
> http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar@forum.dlang.org
>
> Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a new benchmark. My implementation didn't do as well on that one so I went back to the code and optimised it. Also, our other buddy Patrick from work decided to get in on this and use it as an excuse to learn Erlang. So now 4 different implementations in 4 different languages are duking it out.
>
> There are two benchmarks, both written in Go. The first is loadtest and it measures throughput. I varied the number of client connections to the server and measured how many messages were sent per second.
>
> The second benchmark is pingtest and it measures latency. The quantity I varied here was the number of wildcard subscribers (it'd take a while to explain what they are to anyone who isn't familiar with MQTT). It has a bunch of client connections talk to each other, and instead of trying to send as many messages as computerly possible they wait for the acknowledgement from the partner connection. The results are as follows (I tried making charts but LibreOffice wasn't helping and then I got bored), with the numbers being thousands of messages per second:
>
> loadtest
> Connections:   100            500            750            1k
> D + vibe.d:    121.7 +/- 1.5  166.9 +/- 1.5  171.1 +/- 3.3  167.9 +/- 1.3
> C (mosquitto): 106.1 +/- 0.8  122.4 +/- 0.4   95.2 +/- 1.3   74.7 +/- 0.4
> Erlang:        104.1 +/- 2.2  124.2 +/- 5.9  117.6 +/- 4.6  117.7 +/- 3.2
> Go:             90.9 +/- 11   100.1 +/- 0.1   99.3 +/- 0.2   98.8 +/- 0.3
>
> pingtest
> wsubs:         20             200            400
> D + vibe.d:    50.9 +/- 0.3   38.3 +/- 0.2   20.1 +/- 0.1
> C (mosquitto): 65.4 +/- 4.4   45.2 +/- 0.2   20.0 +/- 0.0
> Erlang:        49.1 +/- 0.8   30.9 +/- 0.3   15.6 +/- 0.1
> Go:            45.2 +/- 0.2   27.5 +/- 0.1   16.0 +/- 0.1
>
> So, D was faster than the other contenders by far in throughput, 2nd place losing to the C implementation on latency. I'm still not sure why that is. Profiling in this case is tricky. I'm pretty sure the profiler is still ticking away when a fiber yields - the top function is the one that reads from the network, which I can't do much about.
>
> GC was pretty much a non-issue, which I'm both surprised at and happy about. I allocated as much memory as I wanted at first, and by the time I tried to optmise memory usage by allocating less all I managed to eke out in performance terms was a paltry extra 8% or so.
>
> All in all though, so much for his "nothing is going to be faster than Go because this is what it's good at". If he'd measured it against mosquitto first and realised it was already losing to C I probably wouldn't have written an MQTT broker in D, so I guess that was a good thing. :)
>
> Atila
>
> P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdc

So either D is faster or you are a better coder than your buddies!

> GC should not impact general performance in such scenario but is likely to hinder latency which pretty much matches what you are observing. Can you possibly provide information about network layout, h/w and server code used for testing? There are lot of possible oversights that can make results less legible, would have been nice to verify those ;)

I'd have to dig in and see data to be convinced it's the GC making the latency worse than the C implementation, but from what I know you've got more experience with this so who knows :)

Doing this on a real network was waaay more work than I was willing to put up with for something a colleague said during lunch break, so it was done on localhost. The h/w is a Core i7 Lenovo laptop (W530) with 8GB of RAM running Arch Linux. My server code can be found at https://github.com/atilaneves/mqtt, the C implementation has its own website: http://mosquitto.org/.

Atila

> So either D is faster or you are a better coder than your buddies!

I'd say we're all the same skill level. One of them sits next to me and I heard about his Erlang implementation as it was happening. It's probable that certain techniques could be reused from another program but it's hard to tell. The easiest way would be to have another person write a different implementation in the same language and compare the two. It might happen, another colleague has been working on another Erlang implementation as well, but he's taking a while.

On 04/12/13 13:49, Atila Neves wrote:
> So, D was faster than the other contenders by far in throughput, 2nd place
> losing to the C implementation on latency. I'm still not sure why that is.
> Profiling in this case is tricky. I'm pretty sure the profiler is still ticking
> away when a fiber yields - the top function is the one that reads from the
> network, which I can't do much about.

What about the relative elegance/maintainability/ease of comprehension of the different solutions?  Playing devil's advocate for a moment, I can well understand if a preference for one language over another was decided on the basis of its performance being good _enough_ and the code being really easy to work with, rather than simply the best performer.

On Wednesday, 4 December 2013 at 15:02:44 UTC, Joseph Rushton Wakeling wrote:
> On 04/12/13 13:49, Atila Neves wrote:
>> So, D was faster than the other contenders by far in throughput, 2nd place
>> losing to the C implementation on latency. I'm still not sure why that is.
>> Profiling in this case is tricky. I'm pretty sure the profiler is still ticking
>> away when a fiber yields - the top function is the one that reads from the
>> network, which I can't do much about.
>
> What about the relative elegance/maintainability/ease of comprehension of the different solutions?  Playing devil's advocate for a moment, I can well understand if a preference for one language over another was decided on the basis of its performance being good _enough_ and the code being really easy to work with, rather than simply the best performer.

I can't read Erlang (yet) so I don't know about that one. The C code is... well C code so not great. I have a profound dislike for Go but I'd say the Go and D implementations are just as readable. Then again, I wrote the D code so if I didn't think that was readable... :P

Atila

On 12/4/13 4:49 AM, Atila Neves wrote:
> It all started here:
>
> http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar@forum.dlang.org
>
> Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a
> new benchmark. My implementation didn't do as well on that one so I went
> back to the code and optimised it. Also, our other buddy Patrick from
> work decided to get in on this and use it as an excuse to learn Erlang.
> So now 4 different implementations in 4 different languages are duking
> it out.
[snip]

Interesting. Care to convert this post (only a little adjustment needed) to a blog and publish with source code? Would make a great article. Ask your friends to contribute with descriptions of their implementations, too.

Andrei

On Wednesday, 4 December 2013 at 13:50:52 UTC, Atila Neves wrote:
> I'd have to dig in and see data to be convinced it's the GC making the latency worse than the C implementation, but from what I know you've got more experience with this so who knows :)

I am also not proficient enough with GC stuff to actually prove that but it fitted observations during my own tests pretty well. But I am not 100% sure.

> Doing this on a real network was waaay more work than I was willing to put up with for something a colleague said during lunch break, so it was done on localhost. The h/w is a Core i7 Lenovo laptop (W530) with 8GB of RAM running Arch Linux. My server code can be found at https://github.com/atilaneves/mqtt, the C implementation has its own website: http://mosquitto.org/.

Ah this is rather sad. It makes tests results somewhat unstable because client load interferes with server load. Ideally client should be separate machine and has more powerful h/w than server. However, that also requires ~gigabit network in between as well as matching network cards to hit the limits of top server performance - this makes such tests rather cumbersome to execute.

At the very least you should use process affinity to separate resources between client and server in more predictable manner.

On 12/4/13 7:41 AM, Atila Neves wrote:
> On Wednesday, 4 December 2013 at 15:02:44 UTC, Joseph Rushton Wakeling
> wrote:
>> On 04/12/13 13:49, Atila Neves wrote:
>>> So, D was faster than the other contenders by far in throughput, 2nd
>>> place
>>> losing to the C implementation on latency. I'm still not sure why
>>> that is.
>>> Profiling in this case is tricky. I'm pretty sure the profiler is
>>> still ticking
>>> away when a fiber yields - the top function is the one that reads
>>> from the
>>> network, which I can't do much about.
>>
>> What about the relative elegance/maintainability/ease of comprehension
>> of the different solutions?  Playing devil's advocate for a moment, I
>> can well understand if a preference for one language over another was
>> decided on the basis of its performance being good _enough_ and the
>> code being really easy to work with, rather than simply the best
>> performer.
>
> I can't read Erlang (yet) so I don't know about that one. The C code
> is... well C code so not great. I have a profound dislike for Go but I'd
> say the Go and D implementations are just as readable. Then again, I
> wrote the D code so if I didn't think that was readable... :P

How do they compare in lines of code?

Andrei