Have anybody cooked up any range adaptors for on the fly decoding of bzipped files? Preferable compatible with phobos standard interfaces for file io.

Should probably be built on top of

http://code.dlang.org/packages/bzip2

On Saturday, 2 May 2015 at 13:50:10 UTC, Per Nordlöw wrote:
> Have anybody cooked up any range adaptors for on the fly decoding of bzipped files? Preferable compatible with phobos standard interfaces for file io.
>
> Should probably be built on top of
>
> http://code.dlang.org/packages/bzip2

i use Stephan Schiffels code from

http://forum.dlang.org/thread/djhteyhpcnaskpabxijj@forum.dlang.org?page=2

On Saturday, 2 May 2015 at 20:37:44 UTC, tom wrote:
> i use Stephan Schiffels code from
> http://forum.dlang.org/thread/djhteyhpcnaskpabxijj@forum.dlang.org?page=2

See polished version at:

https://github.com/nordlow/justd/blob/master/zio.d

On Sunday, 3 May 2015 at 14:37:32 UTC, Per Nordlöw wrote:
> On Saturday, 2 May 2015 at 20:37:44 UTC, tom wrote:
>> i use Stephan Schiffels code from
>> http://forum.dlang.org/thread/djhteyhpcnaskpabxijj@forum.dlang.org?page=2
>
> See polished version at:
>
> https://github.com/nordlow/justd/blob/master/zio.d

cool. btw (at least GzipByLine) its sloooow if you compare it to gzcat and pipe it into stdin and use ByLineFast.
i think its mainly the buffer appending that  suboptimal.

On Monday, 4 May 2015 at 20:53:27 UTC, monty wrote:
> cool. btw (at least GzipByLine) its sloooow if you compare it to gzcat and pipe it into stdin and use ByLineFast.
> i think its mainly the buffer appending that  suboptimal.

Could anyone please point out if this is possible to fix somehow? Is the solution to avoid the GC and do buffer reuse somehow?

On Tuesday, 5 May 2015 at 06:48:36 UTC, Per Nordlöw wrote:
> Could anyone please point out if this is possible to fix somehow? Is the solution to avoid the GC and do buffer reuse somehow?

I *really* need this for reading Gigabytes of DBpedia data...

On Saturday, 2 May 2015 at 13:50:10 UTC, Per Nordlöw wrote:
> Should probably be built on top of
>
> http://code.dlang.org/packages/bzip2

Docs here:

http://www.bzip.org/1.0.3/html/hl-interface.html

On Tuesday, 5 May 2015 at 06:50:12 UTC, Per Nordlöw wrote:
> On Tuesday, 5 May 2015 at 06:48:36 UTC, Per Nordlöw wrote:
>> Could anyone please point out if this is possible to fix somehow? Is the solution to avoid the GC and do buffer reuse somehow?
>
> I *really* need this for reading Gigabytes of DBpedia data...

my approach would be to more or less port the ByLineFast approach.
i just haven't had any time to do this yet.

maybe this could work too though:
https://github.com/biod/BioD/tree/master/bio/core/bgzf


also check out:
https://github.com/agordon/fileslurp
https://github.com/biod/BioD/blob/master/bio/core/utils/bylinefast.d

On Tuesday, 5 May 2015 at 07:11:36 UTC, monty wrote:
> also check out:
> https://github.com/agordon/fileslurp
> https://github.com/biod/BioD/blob/master/bio/core/utils/bylinefast.d

Thanks!