> extends if possible.
>
> http://dpaste.dzfl.pl/f86e5db6
>
> import core.stdc.stdlib;
> void main()
> {
>   auto p1 = malloc(14);
>   auto p2 = realloc(p1, 15);
>   assert(p1 is p2);
> }

Cool, nice to know.

Currently I have this benchmark:

Without memory reserve:

------------------------
Total time (ms): 6779.16
Repetitions    : 200
Sample mode    : 33 (73 ocurrences)
Median time    : 33.689
Avg time       : 33.8958
Std dev.       : 1.86754
Minimum        : 31.414
Maximum        : 52.237
95% conf.int.  : [30.2355, 37.5561]  e = 3.66031
99% conf.int.  : [29.0853, 38.7063]  e = 4.81047
EstimatedAvg95%: [33.637, 34.1546]  e = 0.258823
EstimatedAvg99%: [33.5556, 34.2359]  e = 0.340151

With memory reserve:

------------------------
Total time (ms): 5945.1
Repetitions    : 200
Sample mode    : 29 (68 ocurrences)
Median time    : 29.3215
Avg time       : 29.7255
Std dev.       : 2.083
Minimum        : 27.458
Maximum        : 45.32
95% conf.int.  : [25.6429, 33.8081]  e = 4.08261
99% conf.int.  : [24.36, 35.0909]  e = 5.36546
EstimatedAvg95%: [29.4368, 30.0142]  e = 0.288684
EstimatedAvg99%: [29.3461, 30.1049]  e = 0.379396

But I does not gain that much:
------------------------
Total time (ms): 10833
Repetitions    : 200
Sample mode    : 53 (105 ocurrences)
Median time    : 53.848
Avg time       : 54.1652
Std dev.       : 1.72681
Minimum        : 52.118
Maximum        : 69.703
95% conf.int.  : [50.7807, 57.5497]  e = 3.38448
99% conf.int.  : [49.7172, 58.6131]  e = 4.44796
EstimatedAvg95%: [53.9259, 54.4045]  e = 0.239319
EstimatedAvg99%: [53.8507, 54.4797]  e = 0.314518

Seems to be the wrong point of optimizations.

On Monday, 11 March 2013 at 16:37:00 UTC, Namespace wrote:
> Seems to be the wrong point of optimizations.

Profile?

Also, note that appender is a tool for appending stuff *into an array*. If you don't actually need an array at the end, then you could give Array a roll?

I don't think it will necessarily do better, but it doesn't hurt to give it a roll.

> Profile?
>
> Also, note that appender is a tool for appending stuff *into an array*. If you don't actually need an array at the end, then you could give Array a roll?
>
> I don't think it will necessarily do better, but it doesn't hurt to give it a roll.

My profile output looks like this: http://dpaste.1azy.net/0409f0bc

Since I have already discussed most of the stuff in this thread, I do not know what I could still improve on the Lexer.
But I have not yet thought of Array. Because my Appender use already malloc and free, I could learn something by the implementation of 'Array'. Thanks for the suggestion.

Array has an horrible code...

I decided to start from scratch.
So I took a close look at the lexer of dmd and took over the basic functionality.
Result for std.datetime (without profiling so far):

------------------------
Total time (ms): 5971.66
Repetitions    : 200
Sample mode    : 28 (101 ocurrences)
Median time    : 28.89
Avg time       : 29.8583
Std dev.       : 3.37222
Minimum        : 27.958
Maximum        : 70.583
95% conf.int.  : [23.2489, 36.4677]  e = 6.60943
99% conf.int.  : [21.172, 38.5446]  e = 8.68627
EstimatedAvg95%: [29.3909, 30.3257]  e = 0.467357
EstimatedAvg99%: [29.2441, 30.4725]  e = 0.614212

14-Mar-2013 01:19, Namespace пишет:
> Array has an horrible code...
>
> I decided to start from scratch.
> So I took a close look at the lexer of dmd and took over the basic
> functionality.
> Result for std.datetime (without profiling so far):
>

It's getting better!

> ------------------------
> Total time (ms): 5971.66
> Repetitions    : 200
> Sample mode    : 28 (101 ocurrences)
> Median time    : 28.89
> Avg time       : 29.8583
> Std dev.       : 3.37222
> Minimum        : 27.958
> Maximum        : 70.583
> 95% conf.int.  : [23.2489, 36.4677]  e = 6.60943
> 99% conf.int.  : [21.172, 38.5446]  e = 8.68627
> EstimatedAvg95%: [29.3909, 30.3257]  e = 0.467357
> EstimatedAvg99%: [29.2441, 30.4725]  e = 0.614212



-- 
Dmitry Olshansky

Yes, and it's really fun. :)
By the way, the code can be found here: https://github.com/Dgame/Puzzle/blob/master/DLexer.d
Maybe some of you have an idea how I could improve anything.
Last trace.log is also there: https://github.com/Dgame/Puzzle/blob/master/trace.log
My goal is about 20 msecs. :D

14-Mar-2013 21:20, Namespace пишет:
> Yes, and it's really fun. :)
> By the way, the code can be found here:
> https://github.com/Dgame/Puzzle/blob/master/DLexer.d
> Maybe some of you have an idea how I could improve anything.
> Last trace.log is also there:
> https://github.com/Dgame/Puzzle/blob/master/trace.log
> My goal is about 20 msecs. :D

And my is 15 with Dscanner. But it's stuck at around 21-22.
But both of us can just cheat and buy a newer PC :)

So what we need to count is the number of syscalls + cycles spent in user mode to have a meaningful metric.

-- 
Dmitry Olshansky

14-Mar-2013 21:20, Namespace пишет:
> Yes, and it's really fun. :)
> By the way, the code can be found here:
> https://github.com/Dgame/Puzzle/blob/master/DLexer.d
> Maybe some of you have an idea how I could improve anything.
> Last trace.log is also there:
> https://github.com/Dgame/Puzzle/blob/master/trace.log
> My goal is about 20 msecs. :D

So you got rid of array creation. About time ;)

-- 
Dmitry Olshansky

On 03/14/13 18:24, Dmitry Olshansky wrote:
> 14-Mar-2013 21:20, Namespace пишет:
>> Yes, and it's really fun. :)
>> By the way, the code can be found here:
>> https://github.com/Dgame/Puzzle/blob/master/DLexer.d
>> Maybe some of you have an idea how I could improve anything.
>> Last trace.log is also there:
>> https://github.com/Dgame/Puzzle/blob/master/trace.log
>> My goal is about 20 msecs. :D
> 
> And my is 15 with Dscanner. But it's stuck at around 21-22. But both of us can just cheat and buy a newer PC :)

What are you both actually measuring? IOW what happens with the resulting tokens?

artur

15-Mar-2013 01:17, Artur Skawina пишет:
> On 03/14/13 18:24, Dmitry Olshansky wrote:
>> 14-Mar-2013 21:20, Namespace пишет:
>>> Yes, and it's really fun. :)
>>> By the way, the code can be found here:
>>> https://github.com/Dgame/Puzzle/blob/master/DLexer.d
>>> Maybe some of you have an idea how I could improve anything.
>>> Last trace.log is also there:
>>> https://github.com/Dgame/Puzzle/blob/master/trace.log
>>> My goal is about 20 msecs. :D
>>
>> And my is 15 with Dscanner. But it's stuck at around 21-22.
>> But both of us can just cheat and buy a newer PC :)
>
> What are you both actually measuring? IOW what happens with the
> resulting tokens?
>
> artur
>

I'm counting and/or filtering out specific tokens. The latter is a tiny bit slower.

The other options of Dscanner are to generate e.g. HTML highlighting for code or measure token frequencies. Haven't timed those as their speed doesn't entirely depend on lexer as is.

-- 
Dmitry Olshansky