Robert Clipsham wrote:
> The Anh Tran wrote:
>> 3. Do you use multithread or single thread?
> 
> I'm not sure what you mean here. All the current benchmarks are single threaded as the multi threaded benchmarks use std.thread, and my knowledge of phobos is not good enough to port them. If you mean the machine itself, it does support multithreading, so tests could benefit from that.

Sorry, my english is bad.
Does your bench split into single/multi thread categories like Alioth's:
http://shootout.alioth.debian.org/u64q/
http://shootout.alioth.debian.org/u64/

They use affinity to emulate single core bench. But i think we can add a number to command line for that. Ie:
fankuch 10 4 // run fankuch bench with 4 threads, array size is 10

The Anh Tran wrote:
> Could you provide D compiler version that you're using?
> Download-able compiler packages if you don't mind :D. I'm spoiled by EasyD.
All compiler versions are given on the page. Gdc is a tango package, dmd is dmd 1.041 and a tango package, ldc is from hg/tango from svn.

The Anh Tran wrote:
> Robert Clipsham wrote:
>> The Anh Tran wrote:
>>> 3. Do you use multithread or single thread?
>>
>> I'm not sure what you mean here. All the current benchmarks are single threaded as the multi threaded benchmarks use std.thread, and my knowledge of phobos is not good enough to port them. If you mean the machine itself, it does support multithreading, so tests could benefit from that.
> 
> Sorry, my english is bad.
> Does your bench split into single/multi thread categories like Alioth's:
> http://shootout.alioth.debian.org/u64q/
> http://shootout.alioth.debian.org/u64/
> 
> They use affinity to emulate single core bench. But i think we can add a number to command line for that. Ie:
> fankuch 10 4 // run fankuch bench with 4 threads, array size is 10

No, I don't do that, tests just run as they come.

Daniel Keep wrote:
> 
> From the alioth FAQ:
> 
>> How did you measure memory use?
>>
>> By sampling GTop proc_mem for the program and it's child processes
>> every 0.2 seconds. Obviously those measurements are unlikely to be
>> reliable for programs that run for less than 0.2 seconds.

I had read this, but that's as far as I got with it!

> 
> Probably best to ensure this sampling thread is running on a different
> hardware thread to the tested program...
> 
>   -- Daniel

It should be running in an entirely different process, but that depends on how tango.sys.Process deals with processes.

Daniel Keep wrote:
> Incidentally, this might be of assistance:
> 
> http://shootout.alioth.debian.org/u32q/faq.php#measurementscripts
> 
>   -- Daniel

Thanks! I've actually already downloaded these, but being me completely overlooked them. If I remember correctly they were python scripts, and my current testing app is in D - http://hg.octarineparrot.com/dbench/file/tip/run.d

Robert Clipsham wrote:
> Hi all,
> 
> I have set up some benchmarks for dmd, ldc and gdc at http://dbench.octarineparrot.com/.
> 
> There are currently only 6 tests all from http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is not great enough to port the others to tango (I've chosen tango as ldc does not support phobos currently, so it make sense to choose tango as all compilers support it). If you would like to contribute new tests or improve on the current ones let me know and I'll include them next time I run them.
> 
> All source code can be found at http://hg.octarineparrot.com/dbench/file/tip.
> 
> Let me know if you have any ideas for how I can improve the benchmarks, I currently plan to add compile times, size of the final executable and memory usage (if anyone knows an easy way to get the memory usage of a process in D, let me know :D).

The first run should not be included in the average.

Georg Wrede wrote:
> Robert Clipsham wrote:
>> Hi all,
>>
>> I have set up some benchmarks for dmd, ldc and gdc at http://dbench.octarineparrot.com/.
>>
>> There are currently only 6 tests all from http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is not great enough to port the others to tango (I've chosen tango as ldc does not support phobos currently, so it make sense to choose tango as all compilers support it). If you would like to contribute new tests or improve on the current ones let me know and I'll include them next time I run them.
>>
>> All source code can be found at http://hg.octarineparrot.com/dbench/file/tip.
>>
>> Let me know if you have any ideas for how I can improve the benchmarks, I currently plan to add compile times, size of the final executable and memory usage (if anyone knows an easy way to get the memory usage of a process in D, let me know :D).
> 
> The first run should not be included in the average.
> 

Could you explain your reasoning for this? I can't see why it shouldn't be included personally.

Robert Clipsham schrieb:
> Georg Wrede wrote:
>> Robert Clipsham wrote:
>>> Hi all,
>>>
>>> I have set up some benchmarks for dmd, ldc and gdc at http://dbench.octarineparrot.com/.
>>>
>>> There are currently only 6 tests all from http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is not great enough to port the others to tango (I've chosen tango as ldc does not support phobos currently, so it make sense to choose tango as all compilers support it). If you would like to contribute new tests or improve on the current ones let me know and I'll include them next time I run them.
>>>
>>> All source code can be found at http://hg.octarineparrot.com/dbench/file/tip.
>>>
>>> Let me know if you have any ideas for how I can improve the benchmarks, I currently plan to add compile times, size of the final executable and memory usage (if anyone knows an easy way to get the memory usage of a process in D, let me know :D).
>>
>> The first run should not be included in the average.
>>
> 
> Could you explain your reasoning for this? I can't see why it shouldn't be included personally.

fill up of disk and memory caches. That is why the first run has a different timing to the other runs.

Robert Clipsham wrote:
> Georg Wrede wrote:
>> Robert Clipsham wrote:
>>> Hi all,
>>>
>>> I have set up some benchmarks for dmd, ldc and gdc at http://dbench.octarineparrot.com/.
>>>
>>> There are currently only 6 tests all from http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is not great enough to port the others to tango (I've chosen tango as ldc does not support phobos currently, so it make sense to choose tango as all compilers support it). If you would like to contribute new tests or improve on the current ones let me know and I'll include them next time I run them.
>>>
>>> All source code can be found at http://hg.octarineparrot.com/dbench/file/tip.
>>>
>>> Let me know if you have any ideas for how I can improve the benchmarks, I currently plan to add compile times, size of the final executable and memory usage (if anyone knows an easy way to get the memory usage of a process in D, let me know :D).
>>
>> The first run should not be included in the average.
> 
> Could you explain your reasoning for this? I can't see why it shouldn't be included personally.

Suppose you have run the same program very recently before the test. Then the executable will be in memory already, any other files it may want to access are in memory too.

This makes execution much faster than if it were the first time ever this program is run.

If things were deterministic, then you wouldn't run several times and average the results, right?

On Mon, Mar 9, 2009 at 3:15 AM, Georg Wrede <georg.wrede@iki.fi> wrote:
> Robert Clipsham wrote:
>>
>> Georg Wrede wrote:
>>>
>>> Robert Clipsham wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I have set up some benchmarks for dmd, ldc and gdc at http://dbench.octarineparrot.com/.
>>>>
>>>> There are currently only 6 tests all from http://shootout.alioth.debian.org/gp4/d.php. My knowledge of phobos is not great enough to port the others to tango (I've chosen tango as ldc does not support phobos currently, so it make sense to choose tango as all compilers support it). If you would like to contribute new tests or improve on the current ones let me know and I'll include them next time I run them.
>>>>
>>>> All source code can be found at http://hg.octarineparrot.com/dbench/file/tip.
>>>>
>>>> Let me know if you have any ideas for how I can improve the benchmarks, I currently plan to add compile times, size of the final executable and memory usage (if anyone knows an easy way to get the memory usage of a process in D, let me know :D).
>>>
>>> The first run should not be included in the average.
>>
>> Could you explain your reasoning for this? I can't see why it shouldn't be included personally.
>
> Suppose you have run the same program very recently before the test. Then the executable will be in memory already, any other files it may want to access are in memory too.
>
> This makes execution much faster than if it were the first time ever this program is run.
>
> If things were deterministic, then you wouldn't run several times and average the results, right?

Also I think standard practice for benchmarks is not to average but to
take the minimum time.
To the extent that things are not deterministic it is generally
because of factors outside of your program's control -- virtual memory
page fault kicking in, some other process stealing cycles, etc.  Or
put another way, there is no way for the measured run time of your
program to come out artificially too low, but there are lots of ways
it could come out too high.   The reason you average measurements in
other scenarios is because of an expectation that the measurements
form a normal distribution around the true value.  That is not the
case for measurements of computer program running times.  Measurements
will basically always be higher than the true intrinsic run-time for
your program.

--bb