My executable throws as

    core.exception.InvalidMemoryOperationError@(0)

when compiled with DMD git master.

I get no stack trace in GDB.

What to do?

On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
> My executable throws as
>
>     core.exception.InvalidMemoryOperationError@(0)

I've tracked it down to being caused by

    foreach (line; File(path).byLine) {}

when path contains a very large text file (392 MB, 1658080 lines).

Do I have any alternatives to this that doesn't trigger exception?

On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
> On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
>> My executable throws as
>>
>>    core.exception.InvalidMemoryOperationError@(0)
>
> I've tracked it down to being caused by
>
>     foreach (line; File(path).byLine) {}
>
> when path contains a very large text file (392 MB, 1658080 lines).
>
> Do I have any alternatives to this that doesn't trigger exception?

I get the exact same behaviour when I try using MmFile instead:

    import std.mmfile: MmFile;
    auto mmf = new MmFile(path, MmFile.Mode.read, 0, null, pageSize);
    const data = cast(char[])mmf[];
    foreach (line; data.splitter('\n') {}

Does byLine use splitter? If so, maybe splitter is the problem...

On 01/21/2015 04:10 AM, "Nordlöw" wrote:
> On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
>> My executable throws as
>>
>>     core.exception.InvalidMemoryOperationError@(0)
>
> I've tracked it down to being caused by
>
>      foreach (line; File(path).byLine) {}
>
> when path contains a very large text file (392 MB, 1658080 lines).
>
> Do I have any alternatives to this that doesn't trigger exception?

Known bug with a pull request:

  https://issues.dlang.org/show_bug.cgi?id=13856

Here is the duplicate of it, which I've opened recently:

  https://issues.dlang.org/show_bug.cgi?id=14005

Ali

On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli wrote:
> Known bug with a pull request:
>
>   https://issues.dlang.org/show_bug.cgi?id=13856
>
> Here is the duplicate of it, which I've opened recently:
>
>   https://issues.dlang.org/show_bug.cgi?id=14005
>
> Ali

How can this affect Mmfile plus splitter() aswell?

On Wednesday, 21 January 2015 at 15:34:35 UTC, Per Nordlöw wrote:
> On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli wrote:
>> Known bug with a pull request:
>>
>>  https://issues.dlang.org/show_bug.cgi?id=13856
>>
>> Here is the duplicate of it, which I've opened recently:
>>
>>  https://issues.dlang.org/show_bug.cgi?id=14005
>>
>> Ali
>
> How can this affect Mmfile plus splitter() aswell?

I think it can't. The root of 13856/14005 is in std.stdio.readln. Unless MmFile uses readln somehow, this looks like a different issue to me.

On Wednesday, 21 January 2015 at 13:07:11 UTC, Nordlöw wrote:
> On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
>> On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
>>> My executable throws as
>>>
>>>   core.exception.InvalidMemoryOperationError@(0)
>>
>> I've tracked it down to being caused by
>>
>>    foreach (line; File(path).byLine) {}
>>
>> when path contains a very large text file (392 MB, 1658080 lines).
>>
>> Do I have any alternatives to this that doesn't trigger exception?
>
> I get the exact same behaviour when I try using MmFile instead:
>
>     import std.mmfile: MmFile;
>     auto mmf = new MmFile(path, MmFile.Mode.read, 0, null, pageSize);
>     const data = cast(char[])mmf[];
>     foreach (line; data.splitter('\n') {}
>
> Does byLine use splitter? If so, maybe splitter is the problem...

I tested it with a generated large file, and the (rough) file size and line count are apparently not enough to trigger the bug.

Could you share the text file? Maybe it can be compressed to a manageable size? Or maybe you can reduce it to a manageable size?

A 'binary reduction' might work: Take the first half of the file, and test with that. Error? Proceed with that half. No error? Try the other half. When both halves produce no error, give up.

Or maybe dustmite can help here?

If the file contains sensitive information, and you cannot reduce it to a reasonable size, you may be able to programmatically replace classes of characters with one character. E.g. replace all alphanumeric characters with 'x'. Be cautious of replacing multibyte characters with single bytes. And newlines should probably be left intact.

On Wednesday, 21 January 2015 at 20:50:30 UTC, anonymous wrote:
> Or maybe dustmite can help here?
>
> If the file contains sensitive information, and you cannot reduce it to a reasonable size, you may be able to programmatically replace classes of characters with one character. E.g. replace all alphanumeric characters with 'x'. Be cautious of replacing multibyte characters with single bytes. And newlines should probably be left intact.

Breaking out that single code calling File.byLine on the same file didn't trigger the error, unfortunately...

I guess dustmite is my only friend here.

I'm guessing some other code prior to the call together with the call is causing the bug.

I'll do some more testing and see if I can reduce my code...

On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
> My executable throws as
>
>     core.exception.InvalidMemoryOperationError@(0)
>
> when compiled with DMD git master.
>
> I get no stack trace in GDB.
>
> What to do?

InvalidMemoryOperationError generally means that you are performing certain disallowed memory operations during a full garbage collection, such as allocating while the gc is running.  This usually happens when you call a function that allocates in a destructor, which will trigger this error as the destructor is run by the gc.

It appears that the gc issues mentioned above can also trigger it.  If you're running off git head, maybe you can apply that PR 2794 and see if it helps.  Otherwise, maybe you've hit some other gc issue somewhere.

On Friday, 23 January 2015 at 10:13:43 UTC, Joakim wrote:
> InvalidMemoryOperationError generally means that you are performing certain disallowed memory operations during a full garbage collection, such as allocating while the gc is running.

If my app is single-threaded this cannot happen then.

>  This usually happens when you call a function that allocates in a destructor, which will trigger this error as the destructor is run by the gc.
>
> It appears that the gc issues mentioned above can also trigger it.  If you're running off git head, maybe you can apply that PR 2794 and see if it helps.  Otherwise, maybe you've hit some other gc issue somewhere.

So if GC.disable prevents the error from occurring I will have a clue, right?