Am 09.04.2015 um 15:20 schrieb Martin Nowak:
> On 04/09/2015 10:59 AM, Sönke Ludwig wrote:
>> As far as the profiler results can be trusted, a good chunk of the time
>> gets spent for reading individual bytes from memory, but there must be
>> something else low-level going on that make things this bad. However,
>> there is nothing fundamental in the structure/design that would cause
>> this, so I think spending more time with the profiler is the only
>> logical step now. Unfortunately my VTune license has expired and perf on
>> Linux makes the task quite a bit more involved.
>
> I didn't found too many issues.
>
> Most of the time is spent in parseJSONValue (looks like there are some
> expansive struct copies)
> https://github.com/s-ludwig/std_data_json/blob/1da3f828ae6c4fd7cac7f7e13ae9e51ec93e6f02/source/stdx/data/json/parser.d#L148

I'll have a look at the DOM part later. At the moment I'm looking at low level issues of the lexer. There is a lot of potential there, but it's a lot of detail work.

>
> and skipWhitespace. This function could really take some optimization,
> e.g. avoid UTF decoding.

All of the lexing work happens on str.representation instead of the string directly, so no decoding should happen there.

>
> https://github.com/s-ludwig/std_data_json/blob/1da3f828ae6c4fd7cac7f7e13ae9e51ec93e6f02/source/stdx/data/json/lexer.d#L345
>
> Libdparse has some optimized ASM function, might be useful.
> https://github.com/Hackerpilot/libdparse/blob/51b7d9d321aac0fcc4a9be99bbbed5db3158326c/src/std/d/lexer.d#L2233
>

On 4/9/15 5:10 AM, John Colvin wrote:
> On Thursday, 9 April 2015 at 11:49:00 UTC, Martin Nowak wrote:
>> On 04/08/2015 08:32 PM, tcha wrote:
>>
>> Now with release numbers.
>>
>>> D new - debug - 14.98s, 1782.0Mb
>> 8.53s, 1786.8Mb
>>> D new Gdc - debug - 29.08s, 1663.9Mb
>> GDC still misses @nogc support.
>>> D new Ldc - 16.99s, 1663.0Mb
>> 18.76s, 1664.1Mb
>>> D new lazy - debug - 11.50s, 213.2Mb
>> 4.57s, 206Mb
>>> D new lazy Gdc - 13.66s, 206.1Mb
>> Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet support
>> @nogc.
>>> D new lazy Ldc - 3.59s, 205.4Mb
>> 4.0s, 205.4Mb
>>
>> LDC doesn't yet have the GC improvements, therefor is much slower for
>> the DOM parsing benchmarks.
>
> Still getting trounced across the board by rapidjson.

Agreed. Rapidjson is the most important baseline. -- Andrei

On Thursday, 9 April 2015 at 11:49:00 UTC, Martin Nowak wrote:
> On 04/08/2015 08:32 PM, tcha wrote:
>
> Now with release numbers.
>
>> D new - debug - 14.98s, 1782.0Mb
> 8.53s, 1786.8Mb
>> D new Gdc - debug - 29.08s, 1663.9Mb
> GDC still misses @nogc support.
>> D new Ldc - 16.99s, 1663.0Mb
> 18.76s, 1664.1Mb
>> D new lazy - debug - 11.50s, 213.2Mb
> 4.57s, 206Mb
>> D new lazy Gdc - 13.66s, 206.1Mb
> Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet support @nogc.
>> D new lazy Ldc - 3.59s, 205.4Mb
> 4.0s, 205.4Mb
>
> LDC doesn't yet have the GC improvements, therefor is much slower for
> the DOM parsing benchmarks.

ldc -singleobj flag speeds it up dramatically

ldc2 -ofjson_d_new_lazy_ldc -O5 -release -enable-inlining -w -oq -Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d test_new_lazy.d

./json_d_new_lazy_ldc  2.79s user 0.08s system 99% cpu 2.868 total

ldc2 -singleobj -ofjson_d_new_lazy_ldc -O5 -release -enable-inlining -w -oq -Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d test_new_lazy.d

./json_d_new_lazy_ldc  1.78s user 0.08s system 99% cpu 1.869 total


for comparison:
./json_rapid_cpp  0.80s user 0.30s system 99% cpu 1.106 total


Bye.

Am 09.04.2015 um 21:06 schrieb weaselcat:
> On Thursday, 9 April 2015 at 11:49:00 UTC, Martin Nowak wrote:
>> On 04/08/2015 08:32 PM, tcha wrote:
>>
>> Now with release numbers.
>>
>>> D new - debug - 14.98s, 1782.0Mb
>> 8.53s, 1786.8Mb
>>> D new Gdc - debug - 29.08s, 1663.9Mb
>> GDC still misses @nogc support.
>>> D new Ldc - 16.99s, 1663.0Mb
>> 18.76s, 1664.1Mb
>>> D new lazy - debug - 11.50s, 213.2Mb
>> 4.57s, 206Mb
>>> D new lazy Gdc - 13.66s, 206.1Mb
>> Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet support
>> @nogc.
>>> D new lazy Ldc - 3.59s, 205.4Mb
>> 4.0s, 205.4Mb
>>
>> LDC doesn't yet have the GC improvements, therefor is much slower for
>> the DOM parsing benchmarks.
>
> ldc -singleobj flag speeds it up dramatically
>
> ldc2 -ofjson_d_new_lazy_ldc -O5 -release -enable-inlining -w -oq
> -Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d
> test_new_lazy.d
>
> ./json_d_new_lazy_ldc  2.79s user 0.08s system 99% cpu 2.868 total
>
> ldc2 -singleobj -ofjson_d_new_lazy_ldc -O5 -release -enable-inlining -w
> -oq -Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d
> test_new_lazy.d
>
> ./json_d_new_lazy_ldc  1.78s user 0.08s system 99% cpu 1.869 total
>
>
> for comparison:
> ./json_rapid_cpp  0.80s user 0.30s system 99% cpu 1.106 total
>
>
> Bye.

Not sure, but that may also have been my recent optimizations.

On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:

> Not sure, but that may also have been my recent optimizations.

Just tried it with your recent optimizations and it doesn't build with LDC 0.15.1.

Am 08.04.2015 um 20:32 schrieb tcha:
> On Wednesday, 8 April 2015 at 13:56:55 UTC, Sönke Ludwig wrote:
>> The problem is that even the pull parser alone is relatively slow.
>> Also, for some reason the linker reports unresolved symbols as soon as
>> I build without the -debug flag...
>
> Unfortunatelly I overlooked that I tested it with std_data_json v 0.12.0
> yesterday.
> So tested it again today on master:
> dmd and gdc failed to build in release
> ldc2 works
>
> I tried it also with ParseJSONStream and made fork from original
> benchmark with added tests (https://github.com/chalucha/benchmarks) so
> anyone can look (and PR)
>
> Some results:
> Ruby - 6.74s, 2106.2Mb
> Go - 5.92s, 420.9Mb
> D - 10.26s, 1322.5Mb
> D Gdc - 24.76s, 926.1Mb
> D Ldc - 26.31s, 925.0Mb
> D new - debug - 14.98s, 1782.0Mb
> D new Gdc - debug - 29.08s, 1663.9Mb
> D new Ldc - 16.99s, 1663.0Mb
> D new lazy - debug - 11.50s, 213.2Mb
> D new lazy Gdc - debug - 13.66s, 206.1Mb
> D new lazy Ldc - 3.59s, 205.4Mb
> Python - 5.89s, 1063.4Mb
> C++ Boost - 14.60s, 2884.8Mb
> C++ Rapid - 0.80s, 678.9Mb
>
>
> Also tried to dustmite the minimal failing version and here is a result:
> http://pastebin.com/YjdvT3G4
>
> It's my first use of it so I hope it can help to solve this problem. It
> outputs less errors, but also compiles fine in debug and fails to link
> in release.

I got the latest version using the pull parser down to the Crystal Pull parser speed*. This is about twice as slow as Rapid JSON.

* Using DMD to compile. LDC is for some reason much slower, but I didn't try the -singleobj switch yet.

Am 09.04.2015 um 21:26 schrieb weaselcat:
> On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:
>
>> Not sure, but that may also have been my recent optimizations.
>
> Just tried it with your recent optimizations and it doesn't build with
> LDC 0.15.1.

Should work now. I just tested LDC with -singleobj and it's faster than RapidJSON! Now the DOM parser gets interesting.

On Thursday, 9 April 2015 at 19:35:24 UTC, Sönke Ludwig wrote:
> Am 09.04.2015 um 21:26 schrieb weaselcat:
>> On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:
>>
>>> Not sure, but that may also have been my recent optimizations.
>>
>> Just tried it with your recent optimizations and it doesn't build with
>> LDC 0.15.1.
>
> Should work now. I just tested LDC with -singleobj and it's faster than RapidJSON! Now the DOM parser gets interesting.

What flags are you using? RapidJSON is still a lot faster for me.
also, I just realized the build script has the wrong inline flag for LDC(it's using -enable-inlining instead of -inline)

Am 09.04.2015 um 21:35 schrieb Sönke Ludwig:
> Am 09.04.2015 um 21:26 schrieb weaselcat:
>> On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:
>>
>>> Not sure, but that may also have been my recent optimizations.
>>
>> Just tried it with your recent optimizations and it doesn't build with
>> LDC 0.15.1.
>
> Should work now. I just tested LDC with -singleobj and it's faster than
> RapidJSON! Now the DOM parser gets interesting.

Okay let's say about the same speed, the numbers fluctuate quite a bit.

Am 09.04.2015 um 21:37 schrieb weaselcat:
> On Thursday, 9 April 2015 at 19:35:24 UTC, Sönke Ludwig wrote:
>> Am 09.04.2015 um 21:26 schrieb weaselcat:
>>> On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:
>>>
>>>> Not sure, but that may also have been my recent optimizations.
>>>
>>> Just tried it with your recent optimizations and it doesn't build with
>>> LDC 0.15.1.
>>
>> Should work now. I just tested LDC with -singleobj and it's faster
>> than RapidJSON! Now the DOM parser gets interesting.
>
> What flags are you using? RapidJSON is still a lot faster for me.
> also, I just realized the build script has the wrong inline flag for
> LDC(it's using -enable-inlining instead of -inline)

I'll have to test with that exact version, because I've used my own main() function for the test. The flags were: -singleobj -release -O5 -enable-inlining -disable-bondscheck (-disable-boundscheck is only a little faster).