Language performance benchmark be updated 2019/11/09

| Language | Time, s | Memory, MiB | | --------------- | ------- | ----------- | | Kotlin | 2.01 | 37.6 | | Nim Gcc | 2.17 | 0.7 | | C++ Gcc | 2.41 | 1.7 | | OCaml | 2.50 | 4.4 | | Go | 2.94 | 1.5 | | Java | 3.05 | 37.2 | | Crystal | 3.06 | 2.7 | | ML MLton | 3.22 | 0.7 | | Go Gcc | 3.30 | 19.2 | | Rust | 3.43 | 0.8 | | Nim Clang | 3.43 | 1.0 | | D Ldc | 3.57 | 1.4 | | D Gdc | 3.72 | 5.8 | | C# .NET Core | 4.30 | 24.3 | | Scala | 4.30 | 136.3 | | F# .NET Core | 4.35 | 24.5 | | D Dmd | 4.74 | 3.3 | | Haskell (MArray)| 6.88 | 3.5 | | C# Mono | 6.88 | 17.6 | | Javascript Node | 6.97 | 31.5 | | V Gcc | 7.30 | 0.8 | | V Clang | 9.06 | 1.0 | | Racket | 10.49 | 77.4 | | LuaJIT | 10.99 | 2.1 | | Python PyPy | 21.51 | 95.4 | | Chez Scheme | 24.72 | 29.2 | | Haskell | 29.14 | 3.4 | | Ruby truffle | 32.52 | 613.3 | | Ruby JRuby | 180.65 | 241.7 | | Ruby | 191.36 | 13.1 | | Lua 5.3 | 201.26 | 1.4 | | Elixir | 279.03 | 48.9 | | Python3 | 388.22 | 7.8 | | Python | 399.75 | 6.2 | | Tcl (FP) | 494.78 | 4.3 | | Perl | 769.17 | 5.2 | | Tcl (OO) | 1000.55 | 4.3 | https://github.com/kostya/benchmarks

On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote: > https://github.com/kostya/benchmarks Sadly, the benchmark entries appears to use different algorithms... despite the site claiming otherwise. As far as I can tell...

On Friday, 15 November 2019 at 08:25:26 UTC, Ola Fosheim Grøstad wrote: > On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote: >> https://github.com/kostya/benchmarks > > Sadly, the benchmark entries appears to use different algorithms... despite the site claiming otherwise. As far as I can tell... You mean for sorting one uses quick sort while another uses bubble or something to that affect? Did you check a fair amount and found them all different? (I haven't looked yet obviously, and trying to avoid it depending on how much you peaked :) )

On Friday, 15 November 2019 at 18:05:35 UTC, aliak wrote: > You mean for sorting one uses quick sort while another uses bubble or something to that affect? Did you check a fair amount and found them all different? The testset are very limited. Matrix multiplication for one... If you benchmark a language that calls into a C-implementation library... and get twice as good results as the C-benchmark... then you know something is not right! :-D

On Friday, 15 November 2019 at 18:05:35 UTC, aliak wrote: > On Friday, 15 November 2019 at 08:25:26 UTC, Ola Fosheim Grøstad wrote: >> On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote: >>> https://github.com/kostya/benchmarks >> >> Sadly, the benchmark entries appears to use different algorithms... despite the site claiming otherwise. As far as I can tell... > > You mean for sorting one uses quick sort while another uses bubble or something to that affect? Did you check a fair amount and found them all different? (I haven't looked yet obviously, and trying to avoid it depending on how much you peaked :) ) The Json test uses very different parser implementations. There are even multiple implementations for D. One of them uses the fast I/O library while the other one ises std.json, I think. I haven't checked the others, but I expect them to have a similar spread.

November 16, 2019

Re: Language performance benchmark be updated 2019/11/09

Posted by Jacob Shtokolov
in reply to zoujiaqing

Permalink

Jacob Shtokolov

Posted in reply to zoujiaqing

Permalink

On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote:
> https://github.com/kostya/benchmarks

Sorry, but have you tried it by yourself?

I'm running this benchmark outside of Docker, and the numbers I see are very interesting. First of all, the time measurement script is quite questionable. It's a Ruby script, and I found that the running time depends on the shell and environment. For example, when I run it under VSCode's console window, I get 20% worse results in time for all binaries than when I run it under the regular terminal emulator window.

Second, the numbers are, hmmm, how to say... bullshit? Pardon my French!

Here is what I get for the Brainfuck2 mandelbrot benchmark (a simple Brainfuck interpreter implemented in different languages):

C++ gcc version 7.4.0 (g++ -flto -O3 -o bin_cpp bf.cpp):
```
$ ../xtime.rb ./bin_cpp mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
18.05s, 3.6Mb
```

D LDC2 1.18.0 (ldc2 -ofbin_d_ldc -O5 -release -boundscheck=off bf.d)
```
$ ../xtime.rb ./bin_d_ldc mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
19.53s, 3.6Mb
```

Nim 1.0.2 (nim c -o:bin_nim_gcc -d:danger --cc:gcc --verbosity:0 bf.nim)
```
$ ../xtime.rb ./bin_nim_gcc mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
25.07s, 2.2Mb
```

Kotlin kotlinc-jvm 1.3.50 (JRE 1.8.0_201-b09) (kotlinc bf2.kt -include-runtime -d bf2-kt.jar)
```
$ ../xtime.rb java -jar bf2-kt.jar mandel.b
JIT warming up
time: 1.25s
run
# ... here is the Mandelbrot set ASCII art printed ...
26.81s, 36.6Mb
```

Golang go1.13.4 linux/amd64 (go build -o bin_go bf.go)
```
$ ../xtime.rb ./bin_go mandel.b
# ... here is the Mandelbrot set ASCII art printed ...
38.73s, 2.9Mb
```

============================================

So the results for Braunfuck2 mandel.b are:

```
C++ gcc: 18.05s, 3.6Mb
D LDC2:  19.53s, 3.6Mb
Nim:     25.07s, 2.2Mb
Kotlin:  26.81s, 36.6Mb
```

Please note that I've added `boundscheck=off` to LDC2 command line there.
Also, Kotlin is always printing `JIT warming up` and takes about 1 to 2 seconds to warm up, so the results for Brainfuck2 `bench.b` are VERY different. Kotlin is not the first one obviously.

I'm running everything on my laptop with Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
As I mentioned, no Docker containers, just using the included scripts to build the binaries.

Haven't tried other tests, but I feel like I'll get very interesting results for them also.

Would be great if someone else is able to confirm these results because these benchmarks look very manipulative.

On Saturday, 16 November 2019 at 16:07:29 UTC, Jacob Shtokolov wrote: > Haven't tried other tests Just tried to compile and run Base64 The results are: ``` C: 1.46s, 1.9Mb Rust: 1.49s, 2.4Mb D LDC2: 1.98s, 4.2Mb Golang: 2.89s, 10.9Mb C++: 3.18s, 4.8Mb ``` This test is closer to the author's numbers, but Rust implementation isn't faster than the C implementation on my machine. Golang here is faster than C++. The D version was built with bounds checks this time.

On Saturday, 16 November 2019 at 16:34:58 UTC, Jacob Shtokolov wrote: > Just tried to compile and run Base64 The Havlak test is closer to reality: ``` Nim: 12.24s, 477.8Mb C++: 17.33s, 179.3Mb Golang: 21.58s, 358.0Mb D LDC2: 23.55s, 460.4Mb D DMD: 29.04s, 461.9Mb ``` Nim is the winner. But here I would look into the code: what makes LDC produce such poorly optimized binary.

On Saturday, 16 November 2019 at 16:34:58 UTC, Jacob Shtokolov wrote: > On Saturday, 16 November 2019 at 16:07:29 UTC, Jacob Shtokolov wrote: >> Haven't tried other tests > > Just tried to compile and run Base64 > The results are: > > ``` > C: 1.46s, 1.9Mb > Rust: 1.49s, 2.4Mb > D LDC2: 1.98s, 4.2Mb > Golang: 2.89s, 10.9Mb > C++: 3.18s, 4.8Mb > ``` > > This test is closer to the author's numbers, but Rust implementation isn't faster than the C implementation on my machine. > > Golang here is faster than C++. The D version was built with bounds checks this time. Why is C++ doing so badly? Is it because of inefficient usage of buffers and that it doesn't natively support slices?

On Saturday, 16 November 2019 at 16:47:40 UTC, IGotD- wrote: > Why is C++ doing so badly? Looks like that's because they're using some libcrypto APIs (like the BIO). Also, my C++ compiler is not the latest one - 7.4.0. In the benchmark, it's claimed as GCC 9.2.1. But the rest compilers are up to date and actually the same versions as in the benchmark README file

Forums