hi,

unpacking files is kinda slow, probably i'm doing something wrong.

below code is about half the speed of gnu zcat on my os x machine.
why?

why do i need to .dup the buffer?
can i get rid of the casts?


the chunk size has only a marginal influence.
https://github.com/yannick/zcatd

import
  std.zlib,
  std.file,
  std.stdio;

void main(string[] args)
{
  auto f = File(args[1], "r");
  auto uncompressor = new UnCompress(HeaderFormat.gzip);

  foreach (ubyte[] buffer; f.byChunk(4096))
  {
          auto uncompressed = cast(immutable(string)) uncompressor.uncompress(buffer.dup);
          write(uncompressed);
  }
}

On Fri, 07 Aug 2015 07:19:43 +0000
yawniek via Digitalmars-d-learn <digitalmars-d-learn@puremagic.com>
wrote:

> hi,
> 
> unpacking files is kinda slow, probably i'm doing something wrong.
> 
> below code is about half the speed of gnu zcat on my os x machine. why?
> 
> why do i need to .dup the buffer?
> can i get rid of the casts?
> 
> 
> the chunk size has only a marginal influence. https://github.com/yannick/zcatd
> 
> import
>    std.zlib,
>    std.file,
>    std.stdio;
> 
> void main(string[] args)
> {
>    auto f = File(args[1], "r");
>    auto uncompressor = new UnCompress(HeaderFormat.gzip);
> 
>    foreach (ubyte[] buffer; f.byChunk(4096))
>    {
>            auto uncompressed = cast(immutable(string))
> uncompressor.uncompress(buffer.dup);
>            write(uncompressed);
>    }
> }
Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release

On Friday, 7 August 2015 at 07:29:15 UTC, Daniel Kozák wrote:
> Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release

i'm on master. v2.068-devel-8f81ffc
also changed file read mode to "rb".

i don't understand why the program crashes when i do not do the .dup

On Fri, 07 Aug 2015 07:19:43 +0000
"yawniek" <dlang@srtnwz.com> wrote:

> hi,
> 
> unpacking files is kinda slow, probably i'm doing something wrong.
> 
> below code is about half the speed of gnu zcat on my os x machine. why?
> 
> why do i need to .dup the buffer?

It depends. In your case you don't need to.

byChunk() reuse buffer which means, after each call same buffer is use, so all previous data are gone.


> can i get rid of the casts?
> 

Yes, you can use std.conv.to

import
  std.zlib,
  std.file,
  std.stdio,
  std.conv;

void main(string[] args)
{
  auto f = File(args[1], "rb");
  auto uncompressor = new UnCompress(HeaderFormat.gzip);

  foreach (buffer; f.byChunk(4096))
  {
          auto uncompressed = to!(char[])(uncompressor.uncompress(buffer));
          write(uncompressed);
  }
}

On Fri, 07 Aug 2015 07:36:39 +0000
"yawniek" <dlang@srtnwz.com> wrote:

> On Friday, 7 August 2015 at 07:29:15 UTC, Daniel Kozák wrote:
> > Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release
> 
> i'm on master. v2.068-devel-8f81ffc
> also changed file read mode to "rb".
> 
> i don't understand why the program crashes when i do not do the .dup
This is weird. I would say it should not crash

On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote:

>> i don't understand why the program crashes when i do not do the .dup
> This is weird. I would say it should not crash
exactely. but try it yourself.

the fastest version i could come up so far is below.
std.conv slows it down.
going from a 4kb to a 4mb buffer helped. now i'm within 30% of gzcat's performance.

import
  std.zlib,
  std.file,
  std.stdio;

void main(string[] args)
{
  auto f = File(args[1], "rb");
  auto uncompressor = new UnCompress(HeaderFormat.gzip);

  foreach (ubyte[] buffer; f.byChunk(1024*1024*4))
  {
          auto uncompressed = cast(immutable(string)) uncompressor.uncompress(buffer.dup);
          write(uncompressed);
  }
}

On Friday, 7 August 2015 at 07:48:25 UTC, yawniek wrote:
> On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote:
> the fastest version i could come up so far is below.
> std.conv slows it down.
> going from a 4kb to a 4mb buffer helped. now i'm within 30% of gzcat's performance.

ok maybe not, there is another problem, not everything seems to get flushed, i'm missing output

On Fri, 07 Aug 2015 08:01:27 +0000
"yawniek" <dlang@srtnwz.com> wrote:

> On Friday, 7 August 2015 at 07:48:25 UTC, yawniek wrote:
> > On Friday, 7 August 2015 at 07:43:25 UTC, Daniel Kozák wrote:
> > the fastest version i could come up so far is below.
> > std.conv slows it down.
> > going from a 4kb to a 4mb buffer helped. now i'm within 30% of
> > gzcat's performance.
> 
> ok maybe not, there is another problem, not everything seems to get flushed, i'm missing output
> 
> 
> 

import
  std.zlib,
  std.file,
  std.stdio,
  std.conv;

void main(string[] args)
{
  auto f = File(args[1], "rb");
  auto uncompressor = new UnCompress(HeaderFormat.gzip);

  foreach (buffer; f.byChunk(4096))
  {
          auto uncompressed =
  cast(char[])(uncompressor.uncompress(buffer.idup));
  write(uncompressed); }
  write(cast(char[])uncompressor.flush);
}

this is faster for me than zcat

On Fri, 7 Aug 2015 09:43:25 +0200
Daniel Kozák <kozzi@dlang.cz> wrote:

> 
> On Fri, 07 Aug 2015 07:36:39 +0000
> "yawniek" <dlang@srtnwz.com> wrote:
> 
> > On Friday, 7 August 2015 at 07:29:15 UTC, Daniel Kozák wrote:
> > > Which compiler and version. There has been some performance problem with IO on OSX, it should be fixed in 2.068 release
> > 
> > i'm on master. v2.068-devel-8f81ffc
> > also changed file read mode to "rb".
> > 
> > i don't understand why the program crashes when i do not do the .dup
> This is weird. I would say it should not crash
> 

Ok I see it is not weird because Uncompressor class probably has slice to buffer

On Friday, 7 August 2015 at 08:05:01 UTC, Daniel Kozák wrote:
> import
>   std.zlib,
>   std.file,
>   std.stdio,
>   std.conv;
>
> void main(string[] args)
> {
>   auto f = File(args[1], "rb");
>   auto uncompressor = new UnCompress(HeaderFormat.gzip);
>
>   foreach (buffer; f.byChunk(4096))
>   {
>           auto uncompressed =
>   cast(char[])(uncompressor.uncompress(buffer.idup));
>   write(uncompressed); }
>   write(cast(char[])uncompressor.flush);
> }
>
> this is faster for me than zcat

not here on os x:
d version:  3.06s user 1.17s system 82% cpu 5.156 total
gzcat   1.79s user 0.11s system 99% cpu 1.899 total