August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Fri, 07 Aug 2015 08:13:01 +0000 "yawniek" <dlang@srtnwz.com> wrote: > On Friday, 7 August 2015 at 08:05:01 UTC, Daniel Kozák wrote: > > import > > std.zlib, > > std.file, > > std.stdio, > > std.conv; > > > > void main(string[] args) > > { > > auto f = File(args[1], "rb"); > > auto uncompressor = new UnCompress(HeaderFormat.gzip); > > > > foreach (buffer; f.byChunk(4096)) > > { > > auto uncompressed = > > cast(char[])(uncompressor.uncompress(buffer.idup)); > > write(uncompressed); } > > write(cast(char[])uncompressor.flush); > > } > > > > this is faster for me than zcat > > not here on os x: > d version: 3.06s user 1.17s system 82% cpu 5.156 total > gzcat 1.79s user 0.11s system 99% cpu 1.899 total Maybe stil some IO issues. On Linux it is OK. I remmeber a few days ago there has been some discussion about IO improvments fo osx. http://forum.dlang.org/post/mailman.184.1437841312.16005.digitalmars-d@puremagic.com |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek |
On Fri, 07 Aug 2015 08:13:01 +0000
"yawniek" <dlang@srtnwz.com> wrote:
> On Friday, 7 August 2015 at 08:05:01 UTC, Daniel Kozák wrote:
> > import
> > std.zlib,
> > std.file,
> > std.stdio,
> > std.conv;
> >
> > void main(string[] args)
> > {
> > auto f = File(args[1], "rb");
> > auto uncompressor = new UnCompress(HeaderFormat.gzip);
> >
> > foreach (buffer; f.byChunk(4096))
> > {
> > auto uncompressed =
> > cast(char[])(uncompressor.uncompress(buffer.idup));
> > write(uncompressed); }
> > write(cast(char[])uncompressor.flush);
> > }
> >
> > this is faster for me than zcat
>
> not here on os x:
> d version: 3.06s user 1.17s system 82% cpu 5.156 total
> gzcat 1.79s user 0.11s system 99% cpu 1.899 total
can you try it with ldc?
ldc[2] -O -release -boundscheck=off -singleobj app.d
|
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to Daniel Kozák | On Friday, 7 August 2015 at 08:24:11 UTC, Daniel Kozák wrote:
> can you try it with ldc?
>
> ldc[2] -O -release -boundscheck=off -singleobj app.d
ldc 0.15.2 beta2
2.86s user 0.55s system 77% cpu 4.392 total
v2.068-devel-8f81ffc
2.86s user 0.67s system 78% cpu 4.476 total
v2.067
2.88s user 0.67s system 78% cpu 4.529 total
(different file, half the size of the one above:)
archlinux, virtualbox vm, DMD64 D Compiler v2.067
real 0m2.079s
user 0m1.193s
sys 0m0.637s
zcat:
real 0m3.023s
user 0m0.320s
sys 0m2.440s
is there a way to get rid of the flush in the end so everything happens in one loop? its a bit inconvenient when i have another subloop that does work
|
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Fri, 07 Aug 2015 08:42:45 +0000 "yawniek" <dlang@srtnwz.com> wrote: > On Friday, 7 August 2015 at 08:24:11 UTC, Daniel Kozák wrote: > > > can you try it with ldc? > > > > ldc[2] -O -release -boundscheck=off -singleobj app.d > > > ldc 0.15.2 beta2 > 2.86s user 0.55s system 77% cpu 4.392 total > > v2.068-devel-8f81ffc > 2.86s user 0.67s system 78% cpu 4.476 total > > v2.067 > 2.88s user 0.67s system 78% cpu 4.529 total > > (different file, half the size of the one above:) > archlinux, virtualbox vm, DMD64 D Compiler v2.067 > real 0m2.079s > user 0m1.193s > sys 0m0.637s > > zcat: > real 0m3.023s > user 0m0.320s > sys 0m2.440s > > > is there a way to get rid of the flush in the end so everything happens in one loop? its a bit inconvenient when i have another subloop that does work > I am not sure but I do not tkink so, it is currently possible. Btw. If you want to remove [i]dup. you can use this code: http://dpaste.dzfl.pl/f52c82935bb5 |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to Daniel Kozák | On Friday, 7 August 2015 at 08:50:11 UTC, Daniel Kozák wrote:
>> > ldc[2] -O -release -boundscheck=off -singleobj app.d
>>
>>
>> ldc 0.15.2 beta2
>> 2.86s user 0.55s system 77% cpu 4.392 total
>>
>> v2.068-devel-8f81ffc
>> 2.86s user 0.67s system 78% cpu 4.476 total
>>
>> v2.067
>> 2.88s user 0.67s system 78% cpu 4.529 total
>>
i can now reproduce the results and indeed, its faster than zcat:
on a c4.xlarge aws instance running archlinux and dmd v2.067
same file as above on my macbook.
best run: 2.72s user 0.39s system 99% cpu 3.134 total
worst run: 3.47s user 0.46s system 99% cpu 3.970 total
zcat:
best: 4.45s user 0.28s system 99% cpu 4.764 total
worst: 4.99s user 0.57s system 99% cpu 5.568 total
so i guess on os x there is still something to be optimized
|
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Friday, 7 August 2015 at 09:12:32 UTC, yawniek wrote: > On Friday, 7 August 2015 at 08:50:11 UTC, Daniel Kozák wrote: >>> > ldc[2] -O -release -boundscheck=off -singleobj app.d >>> >>> >>> ldc 0.15.2 beta2 >>> 2.86s user 0.55s system 77% cpu 4.392 total >>> >>> v2.068-devel-8f81ffc >>> 2.86s user 0.67s system 78% cpu 4.476 total >>> >>> v2.067 >>> 2.88s user 0.67s system 78% cpu 4.529 total >>> > > > i can now reproduce the results and indeed, its faster than zcat: > on a c4.xlarge aws instance running archlinux and dmd v2.067 > same file as above on my macbook. > > best run: 2.72s user 0.39s system 99% cpu 3.134 total > worst run: 3.47s user 0.46s system 99% cpu 3.970 total > > zcat: > best: 4.45s user 0.28s system 99% cpu 4.764 total > worst: 4.99s user 0.57s system 99% cpu 5.568 total > > > so i guess on os x there is still something to be optimized Can you try it without write operation (comment out all write)? And than try it without uncompression? // without compression: void main(string[] args) { auto f = File(args[1], "r"); foreach (buffer; f.byChunk(4096)) { write(cast(char[])buffer); } } // without write: void main(string[] args) { auto f = File(args[1], "r"); auto uncompressor = new UnCompress(HeaderFormat.gzip); foreach (buffer; f.byChunk(4096)) { auto uncompressed = cast(char[])(uncompressor.uncompress(buffer)); } uncompressor.flush; } |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to Daniel Kozak | On Friday, 7 August 2015 at 11:45:00 UTC, Daniel Kozak wrote: > On Friday, 7 August 2015 at 09:12:32 UTC, yawniek wrote: >> [...] > > Can you try it without write operation (comment out all write)? And than try it without uncompression? > > > // without compression: > > void main(string[] args) > { > auto f = File(args[1], "r"); > foreach (buffer; f.byChunk(4096)) > { > write(cast(char[])buffer); > } > } 0.03s user 0.09s system 11% cpu 1.046 total > // without write: > > void main(string[] args) > { > auto f = File(args[1], "r"); > auto uncompressor = new UnCompress(HeaderFormat.gzip); > > foreach (buffer; f.byChunk(4096)) > { > auto uncompressed = cast(char[])(uncompressor.uncompress(buffer)); > } > uncompressor.flush; > } 2.82s user 0.05s system 99% cpu 2.873 total |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Fri, 07 Aug 2015 12:29:26 +0000 "yawniek" <dlang@srtnwz.com> wrote: > On Friday, 7 August 2015 at 11:45:00 UTC, Daniel Kozak wrote: > > On Friday, 7 August 2015 at 09:12:32 UTC, yawniek wrote: > >> [...] > > > > Can you try it without write operation (comment out all write)? And than try it without uncompression? > > > > > > // without compression: > > > > void main(string[] args) > > { > > auto f = File(args[1], "r"); > > foreach (buffer; f.byChunk(4096)) > > { > > write(cast(char[])buffer); > > } > > } > > 0.03s user 0.09s system 11% cpu 1.046 total > So I/O seems to be OK > > > // without write: > > > > void main(string[] args) > > { > > auto f = File(args[1], "r"); > > auto uncompressor = new UnCompress(HeaderFormat.gzip); > > > > foreach (buffer; f.byChunk(4096)) > > { > > auto uncompressed = > > cast(char[])(uncompressor.uncompress(buffer)); > > } > > uncompressor.flush; > > } > > 2.82s user 0.05s system 99% cpu 2.873 total > So maybe it is a zlib problem on osx? |
Copyright © 1999-2021 by the D Language Foundation