Well, I found out the other day that vibe.d compiles with gdc now so I went back to see if it made any difference to the benchmarks I had.

In throughput it made none.

In the latency one it was about 5-10% faster with gdc compared to dmd, which is good, but it still didn't change the relative positions of the languages.

So that was anti-climatic. :P

Atila

On Thursday, 6 March 2014 at 17:17:12 UTC, Atila Neves wrote:
> Well, I found out the other day that vibe.d compiles with gdc now so I went back to see if it made any difference to the benchmarks I had.
>
> In throughput it made none.
>
> In the latency one it was about 5-10% faster with gdc compared to dmd, which is good, but it still didn't change the relative positions of the languages.
>
> So that was anti-climatic. :P
>
> Atila

I'm suspecting that Vibe's performance is heavily based upon the systems state i.e. hdd. Not so much on the code generation.
I don't know where we can get more performance out of it. But something doesn't quite feel right.

It was already far above the competition in the throughput benchmark anyway. What exactly doesn't feel right to you?

On Friday, 7 March 2014 at 05:44:16 UTC, Rikki Cattermole wrote:
> On Thursday, 6 March 2014 at 17:17:12 UTC, Atila Neves wrote:
>> Well, I found out the other day that vibe.d compiles with gdc now so I went back to see if it made any difference to the benchmarks I had.
>>
>> In throughput it made none.
>>
>> In the latency one it was about 5-10% faster with gdc compared to dmd, which is good, but it still didn't change the relative positions of the languages.
>>
>> So that was anti-climatic. :P
>>
>> Atila
>
> I'm suspecting that Vibe's performance is heavily based upon the systems state i.e. hdd. Not so much on the code generation.
> I don't know where we can get more performance out of it. But something doesn't quite feel right.

On Friday, 7 March 2014 at 08:23:09 UTC, Atila Neves wrote:

>> I'm suspecting that Vibe's performance is heavily based upon the systems state i.e. hdd. Not so much on the code generation.
>> I don't know where we can get more performance out of it. But something doesn't quite feel right.

Robert Pike, the Go lead developer, some days ago published this tweet:

"Just looked at a Google-internal Go server with 139K goroutines serving over 68K active network connections. Concurrency wins."

In that way your MQTT benchmarks falls short with a maximum of 1k connections. You need to repeat it with 50k and 100k connections. Then Go and Erlang will rock and leave D behind. If you want to be fair with Erlang you need to make a benchmark run with 1.000k connections and more, see https://www.erlang-solutions.com/about/news/erlang-powered-whatsapp-exceeds-200-million-monthly-users

I don't like Go's simplistic nature, either, but Go is not about the language. It is about making concurrency much simpler and allowing for many many threads. IMHO this is what gives Go the attention. Except for Erlang no other system/language than Go can get something similar accomplished (except Rust maybe when it is finished, but it is not clear whether it will have good built times like Go or D).

If you want to give D a boost, put Go-style CSP and green threads into it as well. Then D will start to fly. Otherwise it will have to continue competing against C++ as its sole application area where it will always remain a niche player, because of the market dominance of C++.

On Thursday, 6 March 2014 at 17:17:12 UTC, Atila Neves wrote:
> Well, I found out the other day that vibe.d compiles with gdc now so I went back to see if it made any difference to the benchmarks I had.
>
> In throughput it made none.
>
> In the latency one it was about 5-10% faster with gdc compared to dmd, which is good, but it still didn't change the relative positions of the languages.
>
> So that was anti-climatic. :P
>
> Atila

Have you done any profiling of your code to really get a feel on what's taking the time? If it really is io bound then there's nothing gdc can really do to make it better.

Having said that, I've been getting similar results from gdc and dmd recently too, with ldc coming out as a very clear winner.

I profiled it. For throughput it was already IO bound, I'm not surprised gdc wasn't able to make it go faster. For the latency one the profiler logs were harder to grok but I can't really remember what was going on there anymore.

Atila

On Friday, 7 March 2014 at 09:15:15 UTC, John Colvin wrote:
> On Thursday, 6 March 2014 at 17:17:12 UTC, Atila Neves wrote:
>> Well, I found out the other day that vibe.d compiles with gdc now so I went back to see if it made any difference to the benchmarks I had.
>>
>> In throughput it made none.
>>
>> In the latency one it was about 5-10% faster with gdc compared to dmd, which is good, but it still didn't change the relative positions of the languages.
>>
>> So that was anti-climatic. :P
>>
>> Atila
>
> Have you done any profiling of your code to really get a feel on what's taking the time? If it really is io bound then there's nothing gdc can really do to make it better.
>
> Having said that, I've been getting similar results from gdc and dmd recently too, with ldc coming out as a very clear winner.

On 2014-03-07 08:45:21 +0000, Bienlein said:

> On Friday, 7 March 2014 at 08:23:09 UTC, Atila Neves wrote:
> 
>>> I'm suspecting that Vibe's performance is heavily based upon the systems state i.e. hdd. Not so much on the code generation.
>>> I don't know where we can get more performance out of it. But something doesn't quite feel right.
> 
> Robert Pike, the Go lead developer, some days ago published this tweet:
> 
> "Just looked at a Google-internal Go server with 139K goroutines serving over 68K active network connections. Concurrency wins."
> 
> In that way your MQTT benchmarks falls short with a maximum of 1k connections. You need to repeat it with 50k and 100k connections. Then Go and Erlang will rock and leave D behind. If you want to be fair with Erlang you need to make a benchmark run with 1.000k connections and more, see https://www.erlang-solutions.com/about/news/erlang-powered-whatsapp-exceeds-200-million-monthly-users 
> 
> 
> I don't like Go's simplistic nature, either, but Go is not about the language. It is about making concurrency much simpler and allowing for many many threads. IMHO this is what gives Go the attention. Except for Erlang no other system/language than Go can get something similar accomplished (except Rust maybe when it is finished, but it is not clear whether it will have good built times like Go or D).
> 
> If you want to give D a boost, put Go-style CSP and green threads into it as well. Then D will start to fly. Otherwise it will have to continue competing against C++ as its sole application area where it will always remain a niche player, because of the market dominance of C++.

Have you used vibe.d?  It already supports in-process fibers, and much of the work that Sönke is doing is being ported to phobos.    I have no trouble believing that MQTT implemented on top of vibed could compete with Go or Erlang.  If it can't do it right now, it's not because of a fundamental design problem, but because of bugs.

-S.

I initially capped the benchmarks at 1k connections because I ran out of file handles and didn't feel like modifying my system.

I don't know why you think that "Then Go and Erlang will rock and leave D behind" when:

. I don't see any new data to back that up
. Extrapolating the existing MQTT data doesn't suggest that

If the Erlang and Go implementations were slower but seemed to scale better then sure, but that's not what the data show at all.

Since there's so substitute to cold hard data, I went back to the measurements after setting my hard limit for file handles to 150k and using ulimit. I only bothered with Go, D and Erlang. Unfortunately, the most I got away with was around 7500 connections for loadtest. Any more than that and I got failures. I suspect this might be a limitation of the benchmark itself, which was written in Go. The failures happened for all 3 implementations. I managed to get up to 10k connections for pingtest. It failed a lot though.

The results? There's probably a problem with the Erlang implementation but I don't know because I didn't write it. But its performance falls off a cliff in both benchmarks as the number of connections gets up to or close to 10k.

For loadtest D beats both Go and Erlang and there's no sign of Go scaling better (the Erlang one definitely didn't). For pingtest at 10k Go seems to start doing better than D, so maybe you have a point there.

I suspect you might have missed the point of my original blog post. Yes, it shows D beating Erlang and Go, and that's something I obviously like. But that wasn't the point I was trying to make. My point was that just by writing it in Go doesn't mean magical performance benefits because of its CSP, and that vibe.d's fibers would do just fine in a direct competition. The data seem to support that.

Atila


On Friday, 7 March 2014 at 08:45:22 UTC, Bienlein wrote:
> On Friday, 7 March 2014 at 08:23:09 UTC, Atila Neves wrote:
>
>>> I'm suspecting that Vibe's performance is heavily based upon the systems state i.e. hdd. Not so much on the code generation.
>>> I don't know where we can get more performance out of it. But something doesn't quite feel right.
>
> Robert Pike, the Go lead developer, some days ago published this tweet:
>
> "Just looked at a Google-internal Go server with 139K goroutines serving over 68K active network connections. Concurrency wins."
>
> In that way your MQTT benchmarks falls short with a maximum of 1k connections. You need to repeat it with 50k and 100k connections. Then Go and Erlang will rock and leave D behind. If you want to be fair with Erlang you need to make a benchmark run with 1.000k connections and more, see https://www.erlang-solutions.com/about/news/erlang-powered-whatsapp-exceeds-200-million-monthly-users
>
> I don't like Go's simplistic nature, either, but Go is not about the language. It is about making concurrency much simpler and allowing for many many threads. IMHO this is what gives Go the attention. Except for Erlang no other system/language than Go can get something similar accomplished (except Rust maybe when it is finished, but it is not clear whether it will have good built times like Go or D).
>
> If you want to give D a boost, put Go-style CSP and green threads into it as well. Then D will start to fly. Otherwise it will have to continue competing against C++ as its sole application area where it will always remain a niche player, because of the market dominance of C++.

Erlang is likely to have advantage in both concurrent and actively allocating applications because of its specialized garbage collector which if effectively region per Erlang process discarded all at once.

That won't affect throughput though, only latency and still nothing you can't do with D if needed.

On Friday, 7 March 2014 at 08:23:09 UTC, Atila Neves wrote:
> It was already far above the competition in the throughput benchmark anyway. What exactly doesn't feel right to you?
>
> On Friday, 7 March 2014 at 05:44:16 UTC, Rikki Cattermole wrote:
>> On Thursday, 6 March 2014 at 17:17:12 UTC, Atila Neves wrote:
>>> Well, I found out the other day that vibe.d compiles with gdc now so I went back to see if it made any difference to the benchmarks I had.
>>>
>>> In throughput it made none.
>>>
>>> In the latency one it was about 5-10% faster with gdc compared to dmd, which is good, but it still didn't change the relative positions of the languages.
>>>
>>> So that was anti-climatic. :P
>>>
>>> Atila
>>
>> I'm suspecting that Vibe's performance is heavily based upon the systems state i.e. hdd. Not so much on the code generation.
>> I don't know where we can get more performance out of it. But something doesn't quite feel right.

Mostly related to how heavy of an effect a systems IO can have on performance i.e. hdd. Avast makes things a lot worse as well. Thanks to its file system shield. Could possibly get a performance gain by utilising Window's event manager. At Least for Windows.