Hi,

This code to count lines in a gzipped file exits with "Program exited with code -9" when run with the latest version of the library, I guess because I am doing unsafe things. Could someone tell me how to change it to make it work? The actual program I'm writing processes a file line by line, so ideally I'd like to keep the structure of open a file, then foreach over it.

Thanks very much

Andrew

/+ dub.sdl:
	name "hello"
	dependency "iopipe" version="~>0.2.0"
+/

import std.stdio;
import std.typecons;
import iopipe.textpipe;
import iopipe.zip;
import iopipe.bufpipe;
import std.io : File = File;

void main()
{
  auto counter = 0;
  auto fileToRead = File("file.gz").refCounted.bufd.unzip(CompressionFormat.gzip);

  foreach (line; fileToRead.assumeText.byLineRange!false)
  {
    counter++;
  }

  writeln(counter);
}

On 8/7/20 9:40 AM, Andrew wrote:
> Hi,
> 
> This code to count lines in a gzipped file exits with "Program exited with code -9" when run with the latest version of the library, I guess because I am doing unsafe things. Could someone tell me how to change it to make it work? The actual program I'm writing processes a file line by line, so ideally I'd like to keep the structure of open a file, then foreach over it.
> 
> Thanks very much

As of iopipe v0.2.0, io is no longer a required dependency, it's optional. So you must also add a dependency for io.

I tried adding

dependency "io" version="~>0.3.0"

But it fails with:

Got no configuration for dependency io ~>0.3.1 of hello ~master!?

If I add

dependency "io" version="*"

it works.

I think this is an issue with dub when using an inline recipe file, but I don't know?

Note that in this simple example, the line count is stored in the line pipe, you can retreive the number of lines by accessing the `segments` member of the pipe (undocumented, I have to fix that). So my code looks like:

---
/+ dub.sdl:
    name "hello"
    dependency "iopipe" version="~>0.2.0"
    dependency "io" version="*"
+/

import std.stdio;
//import std.typecons; // refCounted not @safe
import iopipe.textpipe;
import iopipe.zip;
import iopipe.bufpipe;
import iopipe.refc; // refCounted that is @safe
import std.io : File = File; // just a note, I don't know why you are renaming here...

void main() @safe // yay @safe!
{
  auto counter = 0;
  auto fileToRead = File("file.gz").refCounted.bufd
      .unzip(CompressionFormat.gzip)
      .assumeText
      .byLine;

  fileToRead.process();
  writeln(fileToRead.segments);
}
---

FYI, I noticed that in my simple test, this outputs one less than the actual lines. I'll have to look into *that* too.

That dependency on writeln also irks me ;) I need to get working on that iopipe replacement for it...

-Steve

On 8/7/20 9:40 AM, Andrew wrote:
> Hi,
> 
> This code to count lines in a gzipped file exits with "Program exited with code -9" when run with the latest version of the library, I guess because I am doing unsafe things.

BTW the safety improvements only change whether it compiles as @safe or not.

If it's building but running is exiting with a code then it's possible there's a bug somewhere.

I did have to change a lot of code to get it to build properly.

If you have it building, but it's exiting with an error, then please file an issue with an example file and sample source that causes the issue.

-Steve

On 8/7/20 8:57 PM, Steven Schveighoffer wrote:
> I tried adding
> 
> dependency "io" version="~>0.3.0"
> 
> But it fails with:
> 
> Got no configuration for dependency io ~>0.3.1 of hello ~master!?
> 
> If I add
> 
> dependency "io" version="*"
> 
> it works.
> 
> I think this is an issue with dub when using an inline recipe file, but I don't know?

ugh. This is an issue with iopipe specifying io version 0.2.x. I will fix this.

-Steve

On 8/7/20 9:31 PM, Steven Schveighoffer wrote:
> On 8/7/20 8:57 PM, Steven Schveighoffer wrote:
>> I think this is an issue with dub when using an inline recipe file, but I don't know?
> 
> ugh. This is an issue with iopipe specifying io version 0.2.x. I will fix this.

OK, iopipe 0.2.2 is released, with no specific io dependency.

It should work with io 0.3.1 as a separate listed dependency.

-Steve

On Saturday, 8 August 2020 at 02:06:36 UTC, Steven Schveighoffer wrote:
> On 8/7/20 9:31 PM, Steven Schveighoffer wrote:
>> On 8/7/20 8:57 PM, Steven Schveighoffer wrote:
>>> I think this is an issue with dub when using an inline recipe file, but I don't know?
>> 
>> ugh. This is an issue with iopipe specifying io version 0.2.x. I will fix this.
>
> OK, iopipe 0.2.2 is released, with no specific io dependency.
>
> It should work with io 0.3.1 as a separate listed dependency.
>
> -Steve

Thank you for looking into this. I assumed the problem was at my end, as you have actually saved my bacon before, processing exactly the same file, the problem being that it was a blocked gzip file (ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr22.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz) I will raise an issue, as I think io is being imported and linked ok.