Thread overview | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
August 07, 2015 zlib performance | ||||
---|---|---|---|---|
| ||||
hi, unpacking files is kinda slow, probably i'm doing something wrong. below code is about half the speed of gnu zcat on my os x machine. why? why do i need to .dup the buffer? can i get rid of the casts? the chunk size has only a marginal influence. https://github.com/yannick/zcatd import std.zlib, std.file, std.stdio; void main(string[] args) { auto f = File(args[1], "r"); auto uncompressor = new UnCompress(HeaderFormat.gzip); foreach (ubyte[] buffer; f.byChunk(4096)) { auto uncompressed = cast(immutable(string)) uncompressor.uncompress(buffer.dup); write(uncompressed); } } |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Fri, 07 Aug 2015 07:19:43 +0000 yawniek via Digitalmars-d-learn <digitalmars-d-learn@puremagic.com> wrote: > hi, > > unpacking files is kinda slow, probably i'm doing something wrong. > > below code is about half the speed of gnu zcat on my os x machine. why? > > why do i need to .dup the buffer? > can i get rid of the casts? > > > the chunk size has only a marginal influence. https://github.com/yannick/zcatd > > import > std.zlib, > std.file, > std.stdio; > > void main(string[] args) > { > auto f = File(args[1], "r"); > auto uncompressor = new UnCompress(HeaderFormat.gzip); > > foreach (ubyte[] buffer; f.byChunk(4096)) > { > auto uncompressed = cast(immutable(string)) > uncompressor.uncompress(buffer.dup); > write(uncompressed); > } > } Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to Daniel Kozák | On Friday, 7 August 2015 at 07:29:15 UTC, Daniel Kozák wrote:
> Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release
i'm on master. v2.068-devel-8f81ffc
also changed file read mode to "rb".
i don't understand why the program crashes when i do not do the .dup
|
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Fri, 07 Aug 2015 07:19:43 +0000 "yawniek" <dlang@srtnwz.com> wrote: > hi, > > unpacking files is kinda slow, probably i'm doing something wrong. > > below code is about half the speed of gnu zcat on my os x machine. why? > > why do i need to .dup the buffer? It depends. In your case you don't need to. byChunk() reuse buffer which means, after each call same buffer is use, so all previous data are gone. > can i get rid of the casts? > Yes, you can use std.conv.to import std.zlib, std.file, std.stdio, std.conv; void main(string[] args) { auto f = File(args[1], "rb"); auto uncompressor = new UnCompress(HeaderFormat.gzip); foreach (buffer; f.byChunk(4096)) { auto uncompressed = to!(char[])(uncompressor.uncompress(buffer)); write(uncompressed); } } |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek |
On Fri, 07 Aug 2015 07:36:39 +0000
"yawniek" <dlang@srtnwz.com> wrote:
> On Friday, 7 August 2015 at 07:29:15 UTC, Daniel Kozák wrote:
> > Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release
>
> i'm on master. v2.068-devel-8f81ffc
> also changed file read mode to "rb".
>
> i don't understand why the program crashes when i do not do the .dup
This is weird. I would say it should not crash
|
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to Daniel Kozák | On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote: >> i don't understand why the program crashes when i do not do the .dup > This is weird. I would say it should not crash exactely. but try it yourself. the fastest version i could come up so far is below. std.conv slows it down. going from a 4kb to a 4mb buffer helped. now i'm within 30% of gzcat's performance. import std.zlib, std.file, std.stdio; void main(string[] args) { auto f = File(args[1], "rb"); auto uncompressor = new UnCompress(HeaderFormat.gzip); foreach (ubyte[] buffer; f.byChunk(1024*1024*4)) { auto uncompressed = cast(immutable(string)) uncompressor.uncompress(buffer.dup); write(uncompressed); } } |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Friday, 7 August 2015 at 07:48:25 UTC, yawniek wrote:
> On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote:
> the fastest version i could come up so far is below.
> std.conv slows it down.
> going from a 4kb to a 4mb buffer helped. now i'm within 30% of gzcat's performance.
ok maybe not, there is another problem, not everything seems to get flushed, i'm missing output
|
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to yawniek | On Fri, 07 Aug 2015 08:01:27 +0000 "yawniek" <dlang@srtnwz.com> wrote: > On Friday, 7 August 2015 at 07:48:25 UTC, yawniek wrote: > > On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote: > > the fastest version i could come up so far is below. > > std.conv slows it down. > > going from a 4kb to a 4mb buffer helped. now i'm within 30% of > > gzcat's performance. > > ok maybe not, there is another problem, not everything seems to get flushed, i'm missing output > > > import std.zlib, std.file, std.stdio, std.conv; void main(string[] args) { auto f = File(args[1], "rb"); auto uncompressor = new UnCompress(HeaderFormat.gzip); foreach (buffer; f.byChunk(4096)) { auto uncompressed = cast(char[])(uncompressor.uncompress(buffer.idup)); write(uncompressed); } write(cast(char[])uncompressor.flush); } this is faster for me than zcat |
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to Daniel Kozák |
On Fri, 7 Aug 2015 09:43:25 +0200
Daniel Kozák <kozzi@dlang.cz> wrote:
>
> On Fri, 07 Aug 2015 07:36:39 +0000
> "yawniek" <dlang@srtnwz.com> wrote:
>
> > On Friday, 7 August 2015 at 07:29:15 UTC, Daniel Kozák wrote:
> > > Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release
> >
> > i'm on master. v2.068-devel-8f81ffc
> > also changed file read mode to "rb".
> >
> > i don't understand why the program crashes when i do not do the .dup
> This is weird. I would say it should not crash
>
Ok I see it is not weird because Uncompressor class probably has slice to buffer
|
August 07, 2015 Re: zlib performance | ||||
---|---|---|---|---|
| ||||
Posted in reply to Daniel Kozák | On Friday, 7 August 2015 at 08:05:01 UTC, Daniel Kozák wrote:
> import
> std.zlib,
> std.file,
> std.stdio,
> std.conv;
>
> void main(string[] args)
> {
> auto f = File(args[1], "rb");
> auto uncompressor = new UnCompress(HeaderFormat.gzip);
>
> foreach (buffer; f.byChunk(4096))
> {
> auto uncompressed =
> cast(char[])(uncompressor.uncompress(buffer.idup));
> write(uncompressed); }
> write(cast(char[])uncompressor.flush);
> }
>
> this is faster for me than zcat
not here on os x:
d version: 3.06s user 1.17s system 82% cpu 5.156 total
gzcat 1.79s user 0.11s system 99% cpu 1.899 total
|
Copyright © 1999-2021 by the D Language Foundation