Humble benchmark (fisher's exact test)

On 8/13/21 7:19 PM, Ki Rill wrote: > It's a simple benchmark examining: > * execution time (sec) > * memory consumption (kb) > * binary size (kb) > * conciseness of a programming language (lines of code) > > [Link](https://github.com/rillki/humble-benchmarks/tree/main/fishers-exact-test) > The most obvious improvement I can see is removing the dynamic array creations in a loop. Since that array seems to be very short, using a static array would improve performance. (Ok, now I see that you already do that for the betterC and nogc versios.) Also, I wonder how disabling GC collections would affect execution time and memory consumption: https://dlang.org/spec/garbage.html#gc_config Ali

August 14, 2021

Re: Humble benchmark (fisher's exact test)

Posted by bachmeier
in reply to Ki Rill

Permalink

bachmeier

Posted in reply to Ki Rill

Permalink

On Saturday, 14 August 2021 at 02:19:02 UTC, Ki Rill wrote:

It's a simple benchmark examining:

execution time (sec)
memory consumption (kb)
binary size (kb)
conciseness of a programming language (lines of code)

Link

I'm skeptical that you're measuring what you think you're measuring. I say this because the R version shouldn't be that much slower than the C version. All that happens when you call fisher.test is that it checks which case it's handling and then calls the builtin C function. For example, this line.

More likely a chunk of your C code is being eliminated by the optimizer. Another thing is that printing to the screen is much slower in R than in C. You shouldn't benchmark printing to the screen since that is not something you would ever do in practice. If you really want performance, you can determine which case applies to your code and then make the underlying .Call yourself. If you don't do that, you're comparing Fisher's exact test against a routine that does a lot more than Fisher's exact test. In any event, you're not comparing against an R implementation of this test.

On Saturday, 14 August 2021 at 12:48:05 UTC, bachmeier wrote:

On Saturday, 14 August 2021 at 02:19:02 UTC, Ki Rill wrote:
More likely a chunk of your C code is being eliminated by the optimizer. Another thing is that printing to the screen is much slower in R than in C. You shouldn't benchmark printing to the screen since that is not something you would ever do in practice.

It happens at the end of the program only once and takes a fraction of a second. I consider it to be irrelevant here.

If you really want performance, you can determine which case applies to your code and then make the underlying .Call yourself. If you don't do that, you're comparing Fisher's exact test against a routine that does a lot more than Fisher's exact test. In any event, you're not comparing against an R implementation of this test.

That is the point of this benchmark, to test it against Python/R implementation irrespective of what it does additionally. And to test compiled languages in general.

Forums