On Fri, 07 Aug 2015 08:13:01 +0000
"yawniek" <dlang@srtnwz.com> wrote:

> On Friday, 7 August 2015 at 08:05:01 UTC, Daniel Kozák wrote:
> > import
> >   std.zlib,
> >   std.file,
> >   std.stdio,
> >   std.conv;
> >
> > void main(string[] args)
> > {
> >   auto f = File(args[1], "rb");
> >   auto uncompressor = new UnCompress(HeaderFormat.gzip);
> >
> >   foreach (buffer; f.byChunk(4096))
> >   {
> >           auto uncompressed =
> >   cast(char[])(uncompressor.uncompress(buffer.idup));
> >   write(uncompressed); }
> >   write(cast(char[])uncompressor.flush);
> > }
> >
> > this is faster for me than zcat
> 
> not here on os x:
> d version:  3.06s user 1.17s system 82% cpu 5.156 total
> gzcat   1.79s user 0.11s system 99% cpu 1.899 total

Maybe stil some IO issues. On Linux it is OK. I remmeber a few days ago there has been some discussion about IO improvments fo osx.

http://forum.dlang.org/post/mailman.184.1437841312.16005.digitalmars-d@puremagic.com

On Fri, 07 Aug 2015 08:13:01 +0000
"yawniek" <dlang@srtnwz.com> wrote:

> On Friday, 7 August 2015 at 08:05:01 UTC, Daniel Kozák wrote:
> > import
> >   std.zlib,
> >   std.file,
> >   std.stdio,
> >   std.conv;
> >
> > void main(string[] args)
> > {
> >   auto f = File(args[1], "rb");
> >   auto uncompressor = new UnCompress(HeaderFormat.gzip);
> >
> >   foreach (buffer; f.byChunk(4096))
> >   {
> >           auto uncompressed =
> >   cast(char[])(uncompressor.uncompress(buffer.idup));
> >   write(uncompressed); }
> >   write(cast(char[])uncompressor.flush);
> > }
> >
> > this is faster for me than zcat
> 
> not here on os x:
> d version:  3.06s user 1.17s system 82% cpu 5.156 total
> gzcat   1.79s user 0.11s system 99% cpu 1.899 total

can you try it with ldc?

ldc[2] -O -release -boundscheck=off -singleobj  app.d

On Friday, 7 August 2015 at 08:24:11 UTC, Daniel Kozák wrote:

> can you try it with ldc?
>
> ldc[2] -O -release -boundscheck=off -singleobj  app.d


ldc 0.15.2 beta2
2.86s user 0.55s system 77% cpu 4.392 total

v2.068-devel-8f81ffc
2.86s user 0.67s system 78% cpu 4.476 total

v2.067
2.88s user 0.67s system 78% cpu 4.529 total

(different file, half the size of the one above:)
archlinux, virtualbox vm, DMD64 D Compiler v2.067
real	0m2.079s
user	0m1.193s
sys	0m0.637s

zcat:
real	0m3.023s
user	0m0.320s
sys	0m2.440s


is there a way to get rid of the flush in the end so everything happens in one loop? its a bit inconvenient when i have another subloop that does work

On Fri, 07 Aug 2015 08:42:45 +0000
"yawniek" <dlang@srtnwz.com> wrote:

> On Friday, 7 August 2015 at 08:24:11 UTC, Daniel Kozák wrote:
> 
> > can you try it with ldc?
> >
> > ldc[2] -O -release -boundscheck=off -singleobj  app.d
> 
> 
> ldc 0.15.2 beta2
> 2.86s user 0.55s system 77% cpu 4.392 total
> 
> v2.068-devel-8f81ffc
> 2.86s user 0.67s system 78% cpu 4.476 total
> 
> v2.067
> 2.88s user 0.67s system 78% cpu 4.529 total
> 
> (different file, half the size of the one above:)
> archlinux, virtualbox vm, DMD64 D Compiler v2.067
> real	0m2.079s
> user	0m1.193s
> sys	0m0.637s
> 
> zcat:
> real	0m3.023s
> user	0m0.320s
> sys	0m2.440s
> 
> 
> is there a way to get rid of the flush in the end so everything happens in one loop? its a bit inconvenient when i have another subloop that does work
> 

I am not sure but I do not tkink so, it is currently possible.

Btw. If you want to remove [i]dup. you can use this code:

http://dpaste.dzfl.pl/f52c82935bb5

On Friday, 7 August 2015 at 08:50:11 UTC, Daniel Kozák wrote:
>> > ldc[2] -O -release -boundscheck=off -singleobj  app.d
>> 
>> 
>> ldc 0.15.2 beta2
>> 2.86s user 0.55s system 77% cpu 4.392 total
>> 
>> v2.068-devel-8f81ffc
>> 2.86s user 0.67s system 78% cpu 4.476 total
>> 
>> v2.067
>> 2.88s user 0.67s system 78% cpu 4.529 total
>> 


i can now reproduce the results and indeed, its faster than zcat:
on a c4.xlarge aws instance running archlinux and dmd v2.067
same file as above on my macbook.

best run: 2.72s user 0.39s system 99% cpu 3.134 total
worst run: 3.47s user 0.46s system 99% cpu 3.970 total

zcat:
best: 4.45s user 0.28s system 99% cpu 4.764 total
worst: 4.99s user 0.57s system 99% cpu 5.568 total


so i guess on os x there is still something to be optimized

On Friday, 7 August 2015 at 09:12:32 UTC, yawniek wrote:
> On Friday, 7 August 2015 at 08:50:11 UTC, Daniel Kozák wrote:
>>> > ldc[2] -O -release -boundscheck=off -singleobj  app.d
>>> 
>>> 
>>> ldc 0.15.2 beta2
>>> 2.86s user 0.55s system 77% cpu 4.392 total
>>> 
>>> v2.068-devel-8f81ffc
>>> 2.86s user 0.67s system 78% cpu 4.476 total
>>> 
>>> v2.067
>>> 2.88s user 0.67s system 78% cpu 4.529 total
>>> 
>
>
> i can now reproduce the results and indeed, its faster than zcat:
> on a c4.xlarge aws instance running archlinux and dmd v2.067
> same file as above on my macbook.
>
> best run: 2.72s user 0.39s system 99% cpu 3.134 total
> worst run: 3.47s user 0.46s system 99% cpu 3.970 total
>
> zcat:
> best: 4.45s user 0.28s system 99% cpu 4.764 total
> worst: 4.99s user 0.57s system 99% cpu 5.568 total
>
>
> so i guess on os x there is still something to be optimized

Can you try it without write operation (comment out all write)? And than try it without uncompression?


// without compression:

void main(string[] args)
{
  auto f = File(args[1], "r");
  foreach (buffer; f.byChunk(4096))
  {
          write(cast(char[])buffer);
  }
}

// without write:

void main(string[] args)
{
  auto f = File(args[1], "r");
  auto uncompressor = new UnCompress(HeaderFormat.gzip);

  foreach (buffer; f.byChunk(4096))
  {
          auto uncompressed = cast(char[])(uncompressor.uncompress(buffer));
  }
  uncompressor.flush;
}

On Friday, 7 August 2015 at 11:45:00 UTC, Daniel Kozak wrote:
> On Friday, 7 August 2015 at 09:12:32 UTC, yawniek wrote:
>> [...]
>
> Can you try it without write operation (comment out all write)? And than try it without uncompression?
>
>
> // without compression:
>
> void main(string[] args)
> {
>   auto f = File(args[1], "r");
>   foreach (buffer; f.byChunk(4096))
>   {
>           write(cast(char[])buffer);
>   }
> }

 0.03s user 0.09s system 11% cpu 1.046 total


> // without write:
>
> void main(string[] args)
> {
>   auto f = File(args[1], "r");
>   auto uncompressor = new UnCompress(HeaderFormat.gzip);
>
>   foreach (buffer; f.byChunk(4096))
>   {
>           auto uncompressed = cast(char[])(uncompressor.uncompress(buffer));
>   }
>   uncompressor.flush;
> }

2.82s user 0.05s system 99% cpu 2.873 total

On Fri, 07 Aug 2015 12:29:26 +0000
"yawniek" <dlang@srtnwz.com> wrote:

> On Friday, 7 August 2015 at 11:45:00 UTC, Daniel Kozak wrote:
> > On Friday, 7 August 2015 at 09:12:32 UTC, yawniek wrote:
> >> [...]
> >
> > Can you try it without write operation (comment out all write)? And than try it without uncompression?
> >
> >
> > // without compression:
> >
> > void main(string[] args)
> > {
> >   auto f = File(args[1], "r");
> >   foreach (buffer; f.byChunk(4096))
> >   {
> >           write(cast(char[])buffer);
> >   }
> > }
> 
>   0.03s user 0.09s system 11% cpu 1.046 total
>

So I/O seems to be OK

> 
> > // without write:
> >
> > void main(string[] args)
> > {
> >   auto f = File(args[1], "r");
> >   auto uncompressor = new UnCompress(HeaderFormat.gzip);
> >
> >   foreach (buffer; f.byChunk(4096))
> >   {
> >           auto uncompressed =
> > cast(char[])(uncompressor.uncompress(buffer));
> >   }
> >   uncompressor.flush;
> > }
> 
> 2.82s user 0.05s system 99% cpu 2.873 total
> 

So maybe it is a zlib problem on osx?